New Bedford Health Department Partners with PS@EDC to Evaluate Overdose Response Efforts
Spotlights |

Program evaluation is a critical component of any prevention effort. It provides communities with the information they need to assess whether new interventions are being delivered as planned and whether they are achieving the goals they were designed to reach. A good evaluation can help communities become more skillful and exact in describing what they plan to do, monitor what they are doing, and improve.

So, in 2019, when the City of New Bedford Health Department was awarded a grant of more than $1.8 million from the Substance Abuse and Mental Health Services Administration to continue its work to fight opioids, evaluation was built into their plan from the start. Funded through the First Responders-Comprehensive Addiction and Recovery Act, the 4-year grant would support the health department’s TRAIN (Teach, Reach, and Initiate Naloxone) Project—a program designed to prepare first responders and key personnel in New Bedford and two neighboring communities to administer the life-saving overdose reversal medication naloxone and provide post-overdose follow-up to affected individuals and their families.

To support their evaluation work, the health department turned to Prevention Solutions@EDC (PS@EDC).

Getting on the Same Page

“Because the project included multiple, moving parts and the involvement of multiple players—the New Bedford Health Department, the police department, and the Fishing Partnership Support Services—it was critical that we started with a clear evaluation plan so that everyone knew what we were doing and where we were headed,” said Jessica Goldberg, one of two PS@EDC technical assistance specialists who worked with the health department. “Everyone needed to be on the same page.”

To better understand the needs and culture of the three communities that would be served by the project, Goldberg and colleague Shai Fuxman conducted 15 key informant interviews with various community stakeholders, including representatives from law enforcement, treatment and recovery agencies, organizations serving the fishing community, and organizations serving the Spanish- and Portuguese-speaking communities.

The team also worked with health department leadership to develop a logic model—a graphic roadmap that laid out what the project hoped to accomplish and the concrete steps they would take to get there. For each of the project’s key goals (training, naloxone distribution, and post-overdose follow-up), the logic model presented a statement of need, core objectives, a detailed list of corresponding activities, and clear process and outcome measures.

“The logic model not only helped the department define a clear path forward but also helped it communicate this path to their partners. And by seeing it written down, the partners had something concrete to respond to. They could provide the input needed to ensure that the plan worked for everyone,” said Goldberg.

Developing Data Collection Tools

Once the logic model was in place, the health department turned its attention to developing the set of data collection tools it would use to track program delivery. For their training sessions, they would need to collect information on attendance (number of people trained, demographics, and roles) and effectiveness. For naloxone distribution, information on the number and types of requests, and on when and with whom distributed kits were being used. And for the outreach visits, they’d need data on numbers of visits, when they occurred, and types of support services provided (e.g., outpatient counseling, medication-assisted treatment). They also hoped to collect data on community knowledge and awareness of overdose and treatment information, as well as outcome data related to treatment admissions and overdose reversals. To date, the group has created three separate tools to support data collection, with more underway.

PS@EDC worked with the department to think through the types of information they wanted to collect, such as information required by the grant. But they also worked together, and in collaboration with project partners, to develop a practical data collection approach that wouldn’t present an undue burden. For example, they re-organized some of the forms to differentiate between “must-have” data (page 1) and “would be nice to have” data (page 2). Wherever possible, they streamlined the collection process, recognizing that theirs was just one of several grants for which partners were collecting data. They also vetted individual survey questions to make sure they were being asked in culturally sensitive ways.

“It was a very collaborative and iterative design process,” says Goldberg. “New Bedford is both ethnically and linguistically diverse, and much of the information we need to collect touches on difficult and sensitive topics. So, we needed to make sure that our data collection approaches, as well as the specific ways we asked certain questions, were respectful and appropriate. And the only way we could do that was with the close involvement of our partners.”

Developing a Centralized Data Repository

PS@EDC also worked with New Bedford to develop a centralized data repository to store the collected data. Built-in Microsoft Teams, the repository includes separate areas for each of the project components: a training tracker, naloxone distribution tracker, and post-overdose outreach tracker. Designated staff will be trained to input the data, thus helping to ensure consistency across components. The repository will not only facilitate project reporting but also support data analysis over time. As with all aspects of the project, it was designed to reflect the needs of its users—in this case, people who are really busy.

“New Bedford is committed to creating an evaluation process that is both respectful and practical. This is reflected in the design of their project, as well as the design and implementation of their evaluation. As data collection commences, it will be exciting to see our work in action, and collectively identify ways to refine our processes,” says Goldberg.

To learn more about PS@EDC’s work to prevent opioid overdose in New Bedford and other communities around the U.S., contact Jessica Goldberg at jgoldberg@edc.org.

Opioids Epidemiology & Data Collection Collaborations & Partnerships

Program evaluation is a critical component of any prevention effort. It provides communities with the information they need to assess whether new interventions are being delivered as planned and whether they are achieving the goals they were designed to reach. A good evaluation can help communities become more skillful and exact in describing what they plan to do, monitor what they are doing, and improve.

So, in 2019, when the City of New Bedford Health Department was awarded a grant of more than $1.8 million from the Substance Abuse and Mental Health Services Administration to continue its work to fight opioids, evaluation was built into their plan from the start. Funded through the First Responders-Comprehensive Addiction and Recovery Act, the 4-year grant would support the health department’s TRAIN (Teach, Reach, and Initiate Naloxone) Project—a program designed to prepare first responders and key personnel in New Bedford and two neighboring communities to administer the life-saving overdose reversal medication naloxone and provide post-overdose follow-up to affected individuals and their families.

To support their evaluation work, the health department turned to Prevention Solutions@EDC (PS@EDC).

Getting on the Same Page

“Because the project included multiple, moving parts and the involvement of multiple players—the New Bedford Health Department, the police department, and the Fishing Partnership Support Services—it was critical that we started with a clear evaluation plan so that everyone knew what we were doing and where we were headed,” said Jessica Goldberg, one of two PS@EDC technical assistance specialists who worked with the health department. “Everyone needed to be on the same page.”

To better understand the needs and culture of the three communities that would be served by the project, Goldberg and colleague Shai Fuxman conducted 15 key informant interviews with various community stakeholders, including representatives from law enforcement, treatment and recovery agencies, organizations serving the fishing community, and organizations serving the Spanish- and Portuguese-speaking communities.

The team also worked with health department leadership to develop a logic model—a graphic roadmap that laid out what the project hoped to accomplish and the concrete steps they would take to get there. For each of the project’s key goals (training, naloxone distribution, and post-overdose follow-up), the logic model presented a statement of need, core objectives, a detailed list of corresponding activities, and clear process and outcome measures.

“The logic model not only helped the department define a clear path forward but also helped it communicate this path to their partners. And by seeing it written down, the partners had something concrete to respond to. They could provide the input needed to ensure that the plan worked for everyone,” said Goldberg.

Developing Data Collection Tools

Once the logic model was in place, the health department turned its attention to developing the set of data collection tools it would use to track program delivery. For their training sessions, they would need to collect information on attendance (number of people trained, demographics, and roles) and effectiveness. For naloxone distribution, information on the number and types of requests, and on when and with whom distributed kits were being used. And for the outreach visits, they’d need data on numbers of visits, when they occurred, and types of support services provided (e.g., outpatient counseling, medication-assisted treatment). They also hoped to collect data on community knowledge and awareness of overdose and treatment information, as well as outcome data related to treatment admissions and overdose reversals. To date, the group has created three separate tools to support data collection, with more underway.

PS@EDC worked with the department to think through the types of information they wanted to collect, such as information required by the grant. But they also worked together, and in collaboration with project partners, to develop a practical data collection approach that wouldn’t present an undue burden. For example, they re-organized some of the forms to differentiate between “must-have” data (page 1) and “would be nice to have” data (page 2). Wherever possible, they streamlined the collection process, recognizing that theirs was just one of several grants for which partners were collecting data. They also vetted individual survey questions to make sure they were being asked in culturally sensitive ways.

“It was a very collaborative and iterative design process,” says Goldberg. “New Bedford is both ethnically and linguistically diverse, and much of the information we need to collect touches on difficult and sensitive topics. So, we needed to make sure that our data collection approaches, as well as the specific ways we asked certain questions, were respectful and appropriate. And the only way we could do that was with the close involvement of our partners.”

Developing a Centralized Data Repository

PS@EDC also worked with New Bedford to develop a centralized data repository to store the collected data. Built-in Microsoft Teams, the repository includes separate areas for each of the project components: a training tracker, naloxone distribution tracker, and post-overdose outreach tracker. Designated staff will be trained to input the data, thus helping to ensure consistency across components. The repository will not only facilitate project reporting but also support data analysis over time. As with all aspects of the project, it was designed to reflect the needs of its users—in this case, people who are really busy.

“New Bedford is committed to creating an evaluation process that is both respectful and practical. This is reflected in the design of their project, as well as the design and implementation of their evaluation. As data collection commences, it will be exciting to see our work in action, and collectively identify ways to refine our processes,” says Goldberg.

To learn more about PS@EDC’s work to prevent opioid overdose in New Bedford and other communities around the U.S., contact Jessica Goldberg at jgoldberg@edc.org.

Date
Resource Type
Spotlights
Topics
Opioids
Practice
Epidemiology & Data Collection
Collaborations & Partnerships