Using Project Extension for Community Healthcare Outcomes to Enhance Substance Use Disorder Care in Primary Care: Mixed Methods Study

Background: Substance use and overdose deaths make up a substantial portion of injury-related deaths in the United States, with the state of Ohio leading the nation in rates of diagnosed substance use disorder (SUD). Ohio’s growing epidemic has indicated a need to improve SUD care in a primary care setting through the engagement of multidisciplinary providers and the use of a comprehensive approach to care. Objective: The purpose of this study was to assess the ability of the Weitzman Extension for Community Healthcare Outcomes (ECHO): Comprehensive Substance Use Disorder Care program to both address and meet 7 series learning objectives and address substances by analyzing (1) the frequency of exposure to the learning objective topics and substance types during case discussions and (2) participants’ change in knowledge, self-efficacy, attitudes, and skills related to the treatment of SUDs pre-to postseries. The 7 series learning objective themes included harm reduction, team-based care, behavioral techniques, medication-assisted treatment, trauma-informed care, co-occurring conditions, and social determinants of health. Methods: We used a mixed methods approach using a conceptual content analysis based on series learning objectives and substances and a 2-tailed paired-samples t test of participants’ self-reported learner outcomes. The content analysis gauged the frequency and dose of learning objective themes and illicit and nonillicit substances mentioned in participant case presentations and discussions, and the paired-samples t test compared participants’ knowledge, self-efficacy, attitudes, and skills associated with learning objectives and medication management of substances from pre-to postseries. Results: The results of the content analysis indicated that 3 learning objective themes—team-based care, harm reduction, and social determinants


Background
In the United States, overdose deaths continue to be a major cause of injury-related deaths.Since the onset of the COVID-19 pandemic, numbers have only accelerated, and the state of Ohio has led the nation in high substance use disorder (SUD) rates, including drug use and prescription drug use.The Centers for Disease Control and Prevention ranks the state among the top 5 across the United States with the highest rates of opioid overdose deaths [1].While research has shown an increase in the number of people enrolled in substance use treatment in Ohio between 2015 and 2019 there was still a notable high increase in the annual average prevalence of past-year illicit drug use disorder in Ohio (3.6%) compared to the regional average (3%) and the national average (2.9%) [2].In addition, past-month alcohol use disorder (9.3%), cannabis use disorder (5.8%), and tobacco use disorder (35.2%) were higher than the national average among young adults aged 18-25 years [2].Ohio's growing epidemic has highlighted the need to improve SUD care in a primary care setting by training providers to better address differences in care and social determinants of health through the use of behavioral techniques, harm-reduction philosophy of care, medication management, and a team-based care approach.

Weitzman Extension for Community Healthcare Outcomes: Comprehensive Substance Use Disorder Care Program
Beginning in 2021, Buckeye Health Plan and Ohio University Heritage College of Osteopathic Medicine have partnered with the Weitzman Institute (WI), a national primary care research, policy, and education institute, to provide targeted support and education to Ohio primary care medical and behavioral health providers working with underserved patients, especially those in the rural, southeastern Appalachian region, using the evidence-based Project Extension for Community Healthcare Outcomes (ECHO) learning model.Project ECHO uses frequent videoconference sessions to connect a target audience of learners with subject matter experts for didactic and case-based instruction and engaged discussion [3].Through regular attendance at these sessions, Project ECHO aims to equip learners with the knowledge, confidence, and skills to better manage complex cases.WI has over 11 years of experience in developing and delivering Project ECHO programs to meet the needs of providers working in resource-limited settings.As an early adopter of the model in 2012, Weitzman ECHO programs have been offered in 22 topic areas to over 8000 health care professionals across all 50 states, Washington D.C., and Puerto Rico.Working in collaboration, Buckeye Health Plan and Ohio University aimed to leverage this expertise and offer multiple Project ECHO programs each year for providers in topics of the greatest need and interest.
As described, one of Ohio's most dire population health needs is to improve outcomes for patients experiencing addiction.Thus, SUD was selected as the second ECHO program developed through this partnership.More specifically, opioids are a heightened concern throughout both Ohio and the United States, and the opioid epidemic has spurred significant funding allocations, such as the Biden Administration's US $1.5 billion award to states and territories to end the epidemic [4].However, there are many additional substances of concern, both illicit and nonillicit, such as alcohol, tobacco, cannabis, methamphetamine, and cocaine [5], which may receive less attention given the directed funding for opioids.For this reason, it was decided that the ECHO would address not only opioids, or any one substance, but rather be designed to provide techniques to help providers address SUD overall through a comprehensive, team-based lens and a harm reduction philosophy of care.Reflecting this broad topical approach, the program was titled the Weitzman ECHO: Comprehensive Substance Use Disorder Care (CSUDC ECHO) program.
CSUDC ECHO consisted of 24 twice-monthly sessions held between July 2021 and July 2022.Each 1-hour session included a 20-to 25-minute didactic presentation followed by 1 patient case submitted by a participant before the session and discussed live for the remaining 35-40 minutes.Textbox 1 outlines the didactic presentation topics for each session.A multidisciplinary core faculty facilitated each session and was comprised of 1 physician with dual board certification in family medicine and addiction medicine and experienced in treating SUDs at federally qualified health centers; 1 nurse practitioner who developed and leads a federally qualified health center medication-assisted treatment (MAT) program; 1 supervisory licensed counselor; and 1 population health expert.Together, the faculty built a 12-month curriculum covering diverse topics such as medication management, team-based care, trauma-informed care, stages of change and motivational interviewing, polysubstance use and co-occurring conditions, and coordinating levels of care.
Participants were recruited by email blasts targeted to each partner's network of Ohio primary care providers and other members of the care team.A total of 109 participants attended at least one session, 16 participants attended between 7 and 11 sessions, and 23 participants attended over 12 (half) of the sessions.On average, there were 32 attendees at each session.Continuing education credits were offered to medical providers, behavioral health providers, and nurses.

Purpose of Study
The purpose of this study was to assess the ability of CSUDC ECHO to both address and meet 7 learning objectives (Textbox 2) and address multiple substances by analyzing (1) the frequency of exposure to the learning objective topics and substance types during case discussions and (2) participants' knowledge, self-efficacy, skills, and attitudes related to the treatment of SUDs pre-to postprogram.

•
Project a harm reduction philosophy of care into your treatment of patients experiencing substance use disorders and explain this concept to peers.
• Use the care team more effectively to improve the management of patients experiencing substance use disorders.
• Use motivational interviewing and other behavioral techniques to improve patient outcomes related to substance use disorders.
• Better differentiate and implement medication management strategies for patients experiencing substance use disorders.
• Illustrate trauma-informed practices in the screening, assessment, and treatment of patients experiencing substance use disorders.

•
Describe and manage common co-occurring conditions and polysubstance use more effectively in patients experiencing substance use disorders.
• Distinguish and address factors related to social determinants of health faced by specific populations experiencing substance use disorders.

Study Design and Data Collection
This study used a mixed methods design, using a conceptual content analysis [6]

Ethical Considerations
This study was approved by the Community Health Center, Inc, Institutional Review Board (IRB; 1190) on January 6, 2022.
Informed consent was accounted for by the authors through the administration of a consent form on the postseries survey gathering participant consent to use their deidentified survey data for the paired-samples t test analysis.The deductive content analysis was considered a secondary analysis and was given exempt status.All data used in this study were deidentified, accounting for privacy and confidentially.No compensation for participation in this study was deemed necessary by the IRB.

Survey Tools
The preseries and postseries surveys were internally created and based on the Consolidated Framework for Implementation Research (CFIR) [7] and Moore's Model of Outcomes Assessment Framework [8].The specific CFIR domains assessed for include intervention characteristics, outer setting, inner setting, characteristics of individuals, and process [7].
Additionally, the levels of Moore's Model of Outcomes Assessment Framework assessed for include level 2 (satisfaction), level 3a (declarative knowledge), level 3b (procedural knowledge), level 4 (competence), level 5 (performance), and level 6 (patient health) [8].The surveys assessed changes in participants' self-reported knowledge, attitudes, self-efficacy, and skills through statements centered on the series' learning objectives.The preseries survey also collected participant characteristics including provider type and years of experience working with patients diagnosed with SUDs, as well as team-based care practices.Additionally, the postseries survey collected information on engagement and practice changes.The preseries survey instrument is presented in Multimedia Appendix 1 and the postseries survey instrument is presented in Multimedia Appendix 2.
While the preseries survey and postseries survey tools were based on CFIR [7] and Moore's Model of Outcomes Assessment Framework [8], both surveys were internally designed.The internal research and evaluation and CSUDC ECHO programmatic teams created the survey tools through several iterations of the internal review, which also consisted of selecting the appropriate domain (ie, knowledge, attitudes, self-efficacy, and skills) to assess each series' learning objective.Each domain used a 5-point Likert scale to assess responses.
The surveys were then presented to the CSUDC ECHO series stakeholders and faculty for review and approval before administering the surveys to the ECHO attendees.See Multimedia Appendices 1 and 2 for the domain placement of learning objectives and the 5-point Likert scales.

Conceptual Content Analysis
To further evaluate Weitzman ECHO CSUDC aims, researchers conducted a conceptual content analysis [6] using a set of a priori themes extracted from the series' learning objectives.Series' learning objectives are detailed in Textbox 2. To establish a priori themes, researchers met before the launch of the ECHO to examine the series' 7 learning objectives and extracted 7 themes for the content analysis.The themes were: harm reduction, team-based care, behavioral techniques, MAT, trauma-informed care, co-occurring conditions, and social determinants of health.To assess the frequency to which multiple substances were discussed, the themes also included 5 illicit and nonillicit substances of concern: alcohol, stimulants, opioids, cannabis, tobacco, or nicotine, plus polysubstance use when any 2 or more of these substances were identified.A conceptual analysis approach was used to gauge the dose and frequency of all learning objective themes and selected illicit and nonillicit substances.The content analysis aimed to confirm the discussion of the series' learning objectives during case presentations and to determine to what extent multiple substances were able to be addressed.
Researchers evaluated all 22 participant-led ECHO case presentations and discussions for the presence of the selected themes in the prepared participant cases, faculty recommendations, and participant recommendations.Case presentations and discussions consisted of participants independently preparing a patient case to present and receive participant and faculty guidance for a patient treatment plan.Case presentations were recorded and transcribed using Zoom videoconferencing software (Zoom Video Communications, Inc).The transcriptions were then used for the conceptual content analysis.
To ensure coding accuracy, 4 researchers independently coded 27% (n=6) of the case presentations and met to reconcile discrepancies and better establish coding parameters.After reconciling discrepancies, 1 researcher coded the remaining 16 case presentations and discussion transcripts.The content analysis themes and descriptions are presented in Table 1.
Table 1.Conceptual content analysis themes and learning objectives.

Learning objective Theme
Project a person-centered philosophy of care into your treatment of patients experiencing substance use disorders and explain this concept to peers.

Harm reduction
Use the care team more effectively to improve the management of patients experiencing substance use disorders.Team-based care Use motivational interviewing and other behavioral techniques to improve patient outcomes related to substance use disorders.

Behavioral techniques
Differentiate and implement medication management strategies for patients experiencing substance use disorders.Medication-assisted treatment Illustrate trauma-informed practices in the screening, assessment, and treatment of patients experiencing substance use disorders.

Trauma-informed care
Describe and manage common co-occurring conditions and polysubstance use more effectively in patients experiencing substance use disorders.

Co-occurring conditions
Distinguish and address factors related to social determinants of health faced by specific populations experiencing substance use disorders.

Paired-Samples t Test
To determine if Project ECHO CSUDC affected participant learner outcomes, researchers calculated mean scores reported on a Likert scale of 1 to 5 and conducted a paired-samples t test to compare pre-and postseries scores at a .05significance level.The surveys consisted of matching statements assessing knowledge, self-efficacy, attitudes, and skills associated with the series' learning objectives.The data were assessed for normality and homogeneity of variance and the assumptions were met.The data analysis was conducted using SPSS Statistics for Windows (version 26.0;IBM Corp).

Conceptual Content Analysis
The conceptual content analysis indicated that all of the a priori themes relating to the learning objectives resulted in high frequencies and doses, appearing in a majority of case presentations and discussions.Further, 3 themes appeared in 100% (n=22) of case presentations and discussions, including team-based care at a frequency of 156, followed by harm reduction at a frequency of 152, and social determinants of health at a frequency of 135.In total, 4 themes appeared in less than 100% (n=22) of case presentations and discussions, but above 81% (n=18), including co-occurring conditions with a frequency of 118 and appearing in 95% (n=21) of case presentations and discussions, followed by behavioral techniques at a frequency of 108 and appearing in 91% (n=20) of case presentations and discussions, MAT at a frequency of 89 and appearing in 86% (n=19) of case presentations and discussions, and trauma-informed care at a frequency of 79 and appearing in 82% (n=18) case presentations and discussions.Additionally, multiple substances were represented but at differing frequencies.The substance that resulted in the highest frequency and dose was alcohol at a frequency of 64 and appeared in 81% (n=18) of case presentations and discussions, followed by stimulants at a frequency of 55 and 77% (n=17) of case presentations and discussions, opioids at a frequency of 49 and 59% (n=13) of case presentations and discussions.Cannabis resulted with a frequency of 38 but appeared in 64% (n=14) of case presentations and discussions.Finally, tobacco and nicotine resulted in the lowest frequency at 11 and dose appearing in 27% (n=6) of case presentations and discussions.When evaluating polysubstance use, which was limited to the use of two or more of the listed substances, we found a dose of 95% (n=21) of case presentations and discussions.The frequency of polysubstance use was not included in the conceptual content analysis since it was not a learning objective theme and the emphasis of the conceptual content analysis was focused on the specific illicit and nonillicit substance types.The results of the conceptual content analysis are presented in Table 3.

Knowledge
In total, 4 knowledge domain statements resulted in statistically significant increases: understanding polysubstance use in patients experiencing SUD (P=.02), understanding the approach colleagues in other disciplines use to address SUD (P=.02), knowledge of medication management strategies for nicotine use disorder (P=.03), and knowledge of medication management strategies for opioid use disorder (OUD; P=.003).Additionally, all knowledge domain statements resulted in an increased change in mean score from preseries to postseries.The results of the knowledge domain preseries and postseries scores are presented in Table 4.

Attitudes
No attitudes domain statements resulted as statistically significant.All attitudes domain statements resulted in an increased change in mean score from preseries to postseries except the statement about a treatment plan for a patient experiencing an illicit SUD only being successful if abstinence is maintained, which resulted in a negative change in mean score.The negative change in mean score from preseries to postseries was the appropriate direction of change for alignment with promoting a harm reduction philosophy.The results of the attitudes domain preseries and postseries scores are presented in Table 5.

Self-Efficacy
In total, 2 self-efficacy statements resulted in statistically significant increases: choosing a medication management strategy for nicotine use disorder (P=.002) and alcohol use disorder (P=.02).Additionally, all self-efficacy domain statements resulted in an increased change in mean score from preseries to postseries.The results of the self-efficacy domain preseries and postseries scores are presented in Table 6.

Skill
In total, 1 skill domain statement resulted in a statistically significant increase: using the stages of change theory to provide stage-based interventions to patients experiencing SUDs (P=.03).Additionally, all skill domain statements resulted in an increased change in mean score from preseries to postseries.The results of the skill domain preseries and postseries scores are presented in Table 7.

Principal Findings
Ohio's annual average prevalence of tobacco use, heroin use, use of prescription pain relievers, OUDs, illicit drug use disorder, and SUD have been higher compared to both regional and national averages [2].Considering the need to address this public health concern, CSUDC ECHO was implemented to train Ohio providers and care team members in substance use care.CSUDC ECHO enhanced the Project ECHO work in this field by focusing content and learning objectives on a comprehensive, team-based lens and a harm reduction philosophy of care to address multiple illicit and nonillicit substances including opioids, alcohol, nicotine, cannabis, and stimulants.To assess the ability of the CSUDC ECHO program to meet its 7 program learning objectives (Textbox 2) and address multiple substances, this study analyzed (1) the frequency of exposure to learning objective themes and substance types during case presentations and discussions and (2) participating providers' change in knowledge, attitudes, self-efficacy, and skills related to the treatment of SUDs.
Study results demonstrate that all 7 learning objectives were frequently addressed in the content of case presentations and discussions throughout the program, with team-based care being the most frequently mentioned, 3 objectives appearing in 100% (n=22) of case discussions (eg, team-based care, harm reduction, and co-occurring conditions), and all 7 objectives appearing in >81% (n=18) of all cases discussed.This may have resulted in the learner outcome improvement pre-to postprogram for multiple learner domains (eg, knowledge, self-efficacy, and skill) for the following themes: team-based care, MAT, polysubstance use, and behavioral techniques.No pattern emerged among the participants included in the paired-samples t test analysis exposure to didactic topics and changes in learner outcomes.
Alcohol, stimulants, opioids, cannabis, and nicotine were addressed in the content of case presentations and discussions throughout CSUDC ECHO with alcohol being the most frequently mentioned and most common substance appearing in cases, 4 substances appearing in >59% (n=13) of case discussions (eg, alcohol, stimulants, opioid, and cannabis), and all coded substances appearing in at least a quarter of cases.The dialogue about these substances during case discussions likely resulted in improvements to the following learner outcomes related to medication management: alcohol use disorder, OUD, and nicotine use disorder.Medication management of cannabis use disorder was not assessed in the pre-to postsurveys.Additionally, the didactic presentation topics that centered on alcohol, opioid, and nicotine use disorder resulted in a higher attendance rate with about 40% (n=6) to 50% (n=8) of the participants included in the paired-samples t test analysis attending the sessions, as compared to only 20% (n=3) of the aforementioned participant sample having attended the session centered on stimulant use disorder.
These findings indicate that the ECHO program's content aligned with its stated learning objectives; met its learning objectives for the 3 themes where significant improvements were measured; and met its intent to address multiple substances in case presentations and discussions.While case presentations and discussions comprise from half to the majority of content in the sessions (30-35 minutes of a 60-minute session), content during sessions also includes faculty didactic presentations (20-25 minutes), which also addresses these 7 learning objectives and various substances but was not a part of the content analysis.Therefore, learner outcome improvements may also be a result of content addressed in didactic presentations.
While the Project ECHO model has been shown to be effective in training the primary care workforce [9], specifically on OUD [10,11] and addiction medicine [12,13], there has been no documentation, to our knowledge, of the ability of a team-based, comprehensive SUD and polysubstance-focused Project ECHO designed to improve learner outcomes (eg, knowledge, self-efficacy, and skills).Although Komaromy and colleagues [14] investigated the frequency of cases presented based on substance type in a comprehensive SUD-focused ECHO, a content analysis of the case presentation and discussion transcripts was not analyzed to either assess the frequency of substances or learning objectives.Furthermore, to our knowledge, this process has not been combined in a mixed method approach to compare learner outcomes with a content analysis to gauge the ability of an SUD-focused Project ECHO program to meet its stated learning objectives.Our results reported here align with this literature and expand to demonstrate that Project ECHO is a potential tool to effectively educate XSL • FO RenderX multidisciplinary providers in a comprehensive approach to SUD care.

Strengths
This study has several strengths which promote the ability of the Project ECHO model in enhancing health care providers' knowledge, self-efficacy, and skill associated with comprehensive SUD care.The focus of this study is unique as there is minimal research exploring the benefits and training ability of Project ECHO with a comprehensive SUD care focus.This study's noteworthy strength is the use of a mixed methods design that presents a comprehensive evaluation correlating the content addressed in the case presentations and discussions to statistically significant learner outcomes to demonstrate how this telementoring continuing education series improved provider's knowledge, skills, and self-efficacy to benefit participating providers and their practices.

Limitations
This study faced several limitations during data collection and analysis.The first limitation of this study was the limited sample size and low response rate.There was a decline between the number of participants who completed the preseries survey and postseries survey, resulting in a low comparative sample, which restricted the options for statistical analysis.Another limitation was generalizability; the results of this Project ECHO are limited to the target audience of medical providers, behavioral health providers, and care team members from the state of Ohio, which is not a representative sample of broader populations nationally.Additionally, participants self-selected to take part in the Project ECHO series, which presents the potential for self-selection bias.Another limitation this study faced was the lack of available or reliable data on Project ECHO and its ability to meet learning objectives and address multiple substances through providers' knowledge, self-efficacy, skill, and attitudes.Furthermore, self-reported data to assess knowledge and skills, and self-reported data in general, could present participant biases and is difficult to corroborate with outcomes.The use of internally designed survey instruments instead of using validated instruments presents as a limitation.In light of these limitations, future studies in this subject matter should include a larger data set.Additionally, future studies using a nested analysis approach might provide more insight into how the learning objective themes coincide with the various illicit and nonillicit substance types and would be a useful analysis to contribute to the knowledge base.Another recommendation for future studies in this subject matter should include a deeper analysis of attendance dose and exposure to didactic topics to better understand the impact on changes in learner outcomes.Future research with greater validity will contribute to the significant gaps in literature regarding this subject.

Conclusions
The purpose of this research study was to assess the ability of CSUDC ECHO to both address and meet 7 learning objectives (Textbox 2) and address multiple substances by analyzing (1) the frequency of exposure to the learning objective topics and substance types during case presentations and discussions and (2) participants' knowledge, self-efficacy, skills, and attitudes related to the treatment of SUDs from pre-to postprogram.The results of this study indicate that CSUDC ECHO was able to both address and meet its learning objectives while addressing multiple substances, as demonstrated by improvements in learner knowledge, self-efficacy, and skills.All learning objective themes resulted in high frequencies and doses, appearing in a majority of case presentations throughout the series.These promising results suggest that Project ECHO is a potential tool to educate primary care providers, behavioral health providers, and care team members in a comprehensive approach to SUD assessment and treatment through complex case discussions combined with didactic learning for certain settings.As Project ECHO programs continue to be established globally and existing programs strengthen, further research examining the model's ability to achieve positive learning outcomes and factors that may contribute to these outcomes (eg, frequency of topic dose) is needed to confirm the outcomes in larger population samples, additional topics of focus, and other geographical settings.

Textbox 2 .
Weitzman Extension for Community Healthcare Outcomes: Comprehensive Substance Use Disorder Care learning objectives.

Table 2 .
Participant characteristics of all ECHO a participants and paired-samples t test analysis sample.
a ECHO: Extension for Community Healthcare Outcomes.b CSUDC: Comprehensive Substance Use Disorder Care.c SUD: substance use disorder.

Table 3 .
The results of frequency and percentage of case appearances (dose) of conceptual content analysis themes.
a MAT: medication-assisted treatment.b -: not available.

Table 4 .
The results of the paired-samples t test for the knowledge domain.

Table 5 .
The results of the paired-samples t test for the attitudes domain.

Table 6 .
The results of the paired-samples t test for the self-efficacy domain.

Table 7 .
The results of the paired-samples t test for the skill domain.