Naloxone Coprescribing and the Prevention of Opioid Overdoses: Quasi-Experimental Metacognitive Assessment of a Novel Education Initiative

Introduction

In 2020, of the 91,799 drug-related overdoses in the United States, 75% involved an opioid [1]. Naloxone is an invaluable tool to prevent opioid overdose [2], and coprescription initiatives (or programs to encourage providers to prescribe naloxone to patients receiving opioids) are a common, important intervention to reduce fatal overdoses. This is especially true in primary care settings, since eligible patients who meet the Centers for Disease Control and Prevention coprescription guidelines are underprescribed naloxone to take home [3].

Academic detailing programs are educational outreach approaches used to improve clinician decision-making, and they have consistently shown a positive effect on altering prescriber behavior and clinical decision-making [4]. Health systems have implemented naloxone coprescription academic detailing programs with demonstrated positive effects on the number of providers prescribing and patients receiving naloxone [2,5]. Critical evaluations of such programs have shown acceptability and feasibility of such interventions [3], including positive impact on increasing the number of prescriptions despite hesitancy around the logistics of implementation [6] and increasing the number of prescriptions after brief interventions [7].

Learners in the health professions are important allies for combatting overdose; interventions have been developed for health profession students to be trained in overdose education and naloxone distribution [8]. Results from student-focused overdose education and naloxone distribution interventions indicate increases in average participant knowledge around identifying and responding to suspected overdoses [8]. To extend knowledge on changes in participant thinking during educational interventions, one area to consider is the effect on participant metacognition. Here, metacognition refers to the beliefs, attitudes, and confidence related to influencing a particular cognitive task, colloquially summarized as thinking about thinking. The measurement of participant metacognitive processes is especially important in health education because of the importance of the desired outcomes and the need for a life span approach to learning in the health professions. Improvements in metacognition in health education interventions have been linked to improved content knowledge acquisition, improved clinical reasoning, and decreased avoidable errors [9]. However, assessing metacognition is not often a focus of medical education evaluation, and those who wish to evaluate metacognition are often met with a lack of clarity on how to effectively measure it [10].

The dual purposes of this brief report are to (1) present a metacognitive evaluation of a naloxone coprescription academic detailing intervention for health professions students and practitioners and (2) describe the application of a metacognitive evaluation for future medical education interventions.

Methods

Data

Participants completed a self-paced, web-based academic detailing naloxone coprescription intervention implemented by Rowan University School of Osteopathic Medicine (RUSOM). This brief continuing medical education (CME)–eligible course provided a standardized, evidence-based curriculum to train RUSOM–affiliated health care providers, administrators, students, and executives across a variety of health care settings on how to implement and sustain naloxone coprescribing programs. Participants were recruited via email, and the only incentive was providing the CME credit at no cost. Consent was provided by agreeing to a question prompt to continue each survey after reading the informed consent documentation.

Data for this analysis came from a 20-item knowledge assessment based on the Centers for Disease Control and Prevention naloxone coprescription guidelines [11], previously validated overdose knowledge assessment instruments [12], as well as guidance based on best practices in implementation science [13]. It was delivered in a pre-post design, where participants completed the knowledge assessment before accessing the educational intervention and after course completion. The course was designed as a single module to be completed in 1 session. Participants were eligible for CME credits after completion if they achieved a passing score; multiple individuals had more than 1 posttest score as they attempted to meet that minimum score. To avoid a bias in results, knowledge scores that came chronologically first were used as the posttest score in all analyses.

To assess metacognition, items were designed with confidence-weighted true-false (CTF) scoring, which combines traditional true-false questions with each learner’s rated confidence for each item (I am confident this is true; I believe this is true, but I am unsure; I believe this is false, but I am unsure; and I am confident this is false). CTF is a useful and simple means to measure both cognitive and metacognitive achievements [14].

Study Sample

The sample includes any individual who completed both pre- and posttest assessments for the naloxone coprescription educational intervention between dates April 2020 and July 2021. To access the intervention, participants had to register via a university web application (from which voluntary demographic data were derived) and then log in to their learning management system. While the intervention provided an opportunity for CME credits, any individual was able to register for and take the course, including nonprescribers and students.

Analysis

Descriptive statistics were calculated for individuals in the study sample. For both the pre- and posttest, 3 content knowledge scores were calculated: the summed CTF score (where confidently incorrect scores equal 0 points and confidently correct scores equal 4 points), the percent correct CTF score (based on maximum of 80), and the binary percent correct score (true/false [T/F]) (number correct regardless of confidence or number of items). In addition, metacognitive scores were calculated using the methods described by Dutke and Barenberg [14] and included absolute accuracy of confidence judgments (AC), bias of confidence judgments (BS), confidence correct probability (CCP), confidence incorrect probability (CIP), and discrimination between correct and incorrect decisions (DIS). AC reflects the overall match between participant confidence and the outcome of their choice. An increase in AC suggests that individuals are better able to gauge both when they are confident in correct answers and unconfident in incorrect answers. BS is similar to AC but gives an indication of the direction and severity of participant ability to correctly asses their level of confidence in an answer. Values close to 0 indicate an exact match between confidence or nonconfidence and correctness or incorrectness, positive values suggest overconfidence (more confident but less correct), and negative values indicate underconfidence (less confident but more correct). However, the BS does not indicate the relative contribution of confidence to correct or incorrect answers, and the CCP and CIP are used to discern the respective probabilities of being confident that the answer is correct (CCP) or confident that the answer is incorrect (CIP). A higher CCP score indicates higher confidence when the answer is correct. A lower CIP score indicates less confidence when the answer is incorrect. A high CCP and low CIP suggests improvement in metacognition. Finally, the DIS is the difference between the CCP and CIP probabilities and is used to indicate how reliably a participant discriminates between correct and incorrect answers, with higher values indicating appropriate participant metacognitive monitoring and the ability to discriminate between concepts that are known and those that need reinforcement [14]. To correct for a left-skewed distribution of assessment values, Wilcoxon signed rank analyses were applied to assess changes in individual scores between pre- and posttest assessments. Finally, Rosenthal correlations were calculated to determine the effect size of the intervention on each metacognitive score. Item-level examinations of CTF distribution were completed to add context to the metacognitive outcomes and identify concepts in the naloxone coprescription framework that may need reinforcement. McNemar tests were used to determine whether there was a significant change in correctness from pre- to posttest for each item. Statistical analyses were completed using Stata 17 (StataCorp LLC).

Ethical Considerations

This study was approved by the Rutgers University Institutional Review Board (ID2019000275). Participants were provided informed consent at pretest and posttest, and data were deidentified prior to analysis. The course and CME credit were provided at no cost to participants, and no additional compensation was provided.

Results

Sample descriptive statistics are shown in Table 1; 307 individuals completed both pre- and posttests. As shown in Table 2, analysis of overall test scores showed a statistically significant improvement in content knowledge after completing the educational intervention, both in CTF score and binary correct-incorrect score. For both, the effect size of the intervention was moderate.

Significant differences in metacognitive scores suggest potential improvements in metacognitive monitoring occurred during the intervention. There is a statistically significant increase in absolute AC with a moderate effect size, suggesting that after intervention individuals are better able to gauge when they are confident in correct answers and unconfident in incorrect answers. For BS, median response values changed from negative to positive with a strong effect size, suggesting an overall change from being underconfident (negative values) in answer choices to appropriately confident (null or positive values) after intervention. Both CCP and CIP had a significant, positive change after intervention with strong effects. There was a significant decrease in DIS score after intervention with a very low effect size, which likely reflects an underlying increase in confidence in incorrect answers after academic detailing.

Table 3 shows the CTF and binary T/F frequencies for each item and an indication of significant change from pre- to posttest using McNemar test. Most items saw their binary correct answers increase at posttest; only 1 item (item 15) saw a significant decline in correct answers (t₃₀₆=−4.41; P=.04). This item was part of a conceptual group of questions (items 7, 12, and 15) on determining individual risk of overdose using the Risk Index for Overdose or Serious Opioid-Induced Respiratory Depression (RIOSORD) tool. From a metacognitive perspective, this group of questions also saw the frequency of confident incorrect answers increase at posttest between 117% and 350%.

Discussion

In summary, findings suggest that the naloxone coprescription academic detailing intervention was effective at delivering content area knowledge and stimulating metacognition about coprescription practices. From a knowledge gain perspective, the intervention saw increases in participant knowledge along the key objectives of a naloxone coprescription program. In addition, metacognitively, results suggest that individuals were more likely to be confident in their answer choices after the intervention. While the confidence gain was seen mostly among participants who chose correct answers, a small number of participants also became overconfident in their incorrect answers. This finding could support the development of refresher courses as a tactic to reexpose those who were overconfident to the material to correct any misunderstanding of course content [15], and for naloxone-prescribing programs, refreshers would be needed to account for the changing nature of the naloxone marketplace or clinical guidelines for overdose risk. The absolute AC significantly improved after intervention. Participants were better able to confidently discern correct and incorrect answers at posttest.

Across medical education settings, metacognitive evaluations have been implemented successfully, which has resulted in improvements in metacognition itself [16], the learning and retrieval of basic science information [17], and moderation of performance test anxiety in observed clinical examinations [18]. Even withstanding the complexity of metacognitive measurement concepts [10], CTF presents itself as a simple mechanism for metacognitive evaluation available to medical educators and evaluators, allowing them to assess potential areas of weakness in content delivery and specific areas where students may struggle with concepts [14]. Academic detailing programs applying metacognitive evaluative processes may be best served by developing feedback loops for learners and curriculum designers driven by the results CTF tests. In our results, learners were the most confident in incorrect answers for questions detailing the specifics of assessing individual risk of overdose. Feedback to learners could provide clarification on application of the RIOSORD tool through follow-up emails, refresher courses, or the development of learning communities to support implementation and adoption. Feedback to curriculum designers may prompt an evaluation of course content to identify what course updates were needed to ensure key concept delivery.

This specific intervention was self-paced and web-based, a common format available for CME. Electronic interventions have been shown to be no different for metacognition than in-person interventions, despite having no formal educator to guide the process [19]. This is important evidence to bolster the benefit of web-based continuing education [9], especially given the proliferation of web-based education that occurred during the COVID-19 pandemic [20]. Evidence suggests that if learners are going to engage in a self-paced curriculum, adding a metacognitive layer forces learners to critically think about their content knowledge acquisition [21]. The identified potential overconfidence observed in this study after receiving education is consistent with other metacognitive evaluations [22].

This study is not without limitations. The evaluation used a 1-group pretest, posttest design, which limits generalizability of the findings. While the course on best practices for coprescribing was brief and designed to be completed in 1 session, it is unknown how or whether other naloxone initiatives may have influenced participants. The academic detailing program’s enrollment was open to RUSOM and its affiliates; it is not possible to rule out selection bias as 1 factor influencing score improvements.

While metacognitive processing was shown to be important for behavior change, we do not have a long-term measure to determine whether the intervention resulted in increased naloxone prescription or even whether learners went on to implement coprescription initiatives in their practice settings. As referenced earlier, there are multiple ways to assess metacognition, of which CTF is one, and the validity of one accepted measure of metacognition has yet to be established. However, this particular method of assessing metacognition with multiple conceptual domains allows evaluators to use several diagnostic measures to understand the conditions under which knowledge gain is occurring in educational interventions. Future research could measure long-term changes in these particular scores, tracking metacognitive monitoring as skills are applied, and potentially correlate both cognitive and metacognitive changes with on-the-ground prescribing and implementation behaviors.