In 2020, of the 91,799 drug-related overdoses in the United States, 75% involved an opioid [1]. Naloxone is an invaluable tool to prevent opioid overdose [2], and coprescription initiatives (or programs to encourage providers to prescribe naloxone to patients receiving opioids) are a common, important intervention to reduce fatal overdoses. This is especially true in primary care settings, since eligible patients who meet the Centers for Disease Control and Prevention coprescription guidelines are underprescribed naloxone to take home [3].

Academic detailing programs are educational outreach approaches used to improve clinician decision-making, and they have consistently shown a positive effect on altering prescriber behavior and clinical decision-making [4]. Health systems have implemented naloxone coprescription academic detailing programs with demonstrated positive effects on the number of providers prescribing and patients receiving naloxone [2,5]. Critical evaluations of such programs have shown acceptability and feasibility of such interventions [3], including positive impact on increasing the number of prescriptions despite hesitancy around the logistics of implementation [6] and increasing the number of prescriptions after brief interventions [7].

Learners in the health professions are important allies for combatting overdose; interventions have been developed for health profession students to be trained in overdose education and naloxone distribution [8]. Results from student-focused overdose education and naloxone distribution interventions indicate increases in average participant knowledge around identifying and responding to suspected overdoses [8]. To extend knowledge on changes in participant thinking during educational interventions, one area to consider is the effect on participant metacognition. Here, metacognition refers to the beliefs, attitudes, and confidence related to influencing a particular cognitive task, colloquially summarized as thinking about thinking. The measurement of participant metacognitive processes is especially important in health education because of the importance of the desired outcomes and the need for a life span approach to learning in the health professions. Improvements in metacognition in health education interventions have been linked to improved content knowledge acquisition, improved clinical reasoning, and decreased avoidable errors [9]. However, assessing metacognition is not often a focus of medical education evaluation, and those who wish to evaluate metacognition are often met with a lack of clarity on how to effectively measure it [10].

The dual purposes of this brief report are to (1) present a metacognitive evaluation of a naloxone coprescription academic detailing intervention for health professions students and practitioners and (2) describe the application of a metacognitive evaluation for future medical education interventions.

Sample descriptive statistics are shown in Table 1; 307 individuals completed both pre- and posttests. As shown in Table 2, analysis of overall test scores showed a statistically significant improvement in content knowledge after completing the educational intervention, both in CTF score and binary correct-incorrect score. For both, the effect size of the intervention was moderate.

Significant differences in metacognitive scores suggest potential improvements in metacognitive monitoring occurred during the intervention. There is a statistically significant increase in absolute AC with a moderate effect size, suggesting that after intervention individuals are better able to gauge when they are confident in correct answers and unconfident in incorrect answers. For BS, median response values changed from negative to positive with a strong effect size, suggesting an overall change from being underconfident (negative values) in answer choices to appropriately confident (null or positive values) after intervention. Both CCP and CIP had a significant, positive change after intervention with strong effects. There was a significant decrease in DIS score after intervention with a very low effect size, which likely reflects an underlying increase in confidence in incorrect answers after academic detailing.

Table 3 shows the CTF and binary T/F frequencies for each item and an indication of significant change from pre- to posttest using McNemar test. Most items saw their binary correct answers increase at posttest; only 1 item (item 15) saw a significant decline in correct answers (t₃₀₆=−4.41; P=.04). This item was part of a conceptual group of questions (items 7, 12, and 15) on determining individual risk of overdose using the Risk Index for Overdose or Serious Opioid-Induced Respiratory Depression (RIOSORD) tool. From a metacognitive perspective, this group of questions also saw the frequency of confident incorrect answers increase at posttest between 117% and 350%.

In summary, findings suggest that the naloxone coprescription academic detailing intervention was effective at delivering content area knowledge and stimulating metacognition about coprescription practices. From a knowledge gain perspective, the intervention saw increases in participant knowledge along the key objectives of a naloxone coprescription program. In addition, metacognitively, results suggest that individuals were more likely to be confident in their answer choices after the intervention. While the confidence gain was seen mostly among participants who chose correct answers, a small number of participants also became overconfident in their incorrect answers. This finding could support the development of refresher courses as a tactic to reexpose those who were overconfident to the material to correct any misunderstanding of course content [15], and for naloxone-prescribing programs, refreshers would be needed to account for the changing nature of the naloxone marketplace or clinical guidelines for overdose risk. The absolute AC significantly improved after intervention. Participants were better able to confidently discern correct and incorrect answers at posttest.

Across medical education settings, metacognitive evaluations have been implemented successfully, which has resulted in improvements in metacognition itself [16], the learning and retrieval of basic science information [17], and moderation of performance test anxiety in observed clinical examinations [18]. Even withstanding the complexity of metacognitive measurement concepts [10], CTF presents itself as a simple mechanism for metacognitive evaluation available to medical educators and evaluators, allowing them to assess potential areas of weakness in content delivery and specific areas where students may struggle with concepts [14]. Academic detailing programs applying metacognitive evaluative processes may be best served by developing feedback loops for learners and curriculum designers driven by the results CTF tests. In our results, learners were the most confident in incorrect answers for questions detailing the specifics of assessing individual risk of overdose. Feedback to learners could provide clarification on application of the RIOSORD tool through follow-up emails, refresher courses, or the development of learning communities to support implementation and adoption. Feedback to curriculum designers may prompt an evaluation of course content to identify what course updates were needed to ensure key concept delivery.

This specific intervention was self-paced and web-based, a common format available for CME. Electronic interventions have been shown to be no different for metacognition than in-person interventions, despite having no formal educator to guide the process [19]. This is important evidence to bolster the benefit of web-based continuing education [9], especially given the proliferation of web-based education that occurred during the COVID-19 pandemic [20]. Evidence suggests that if learners are going to engage in a self-paced curriculum, adding a metacognitive layer forces learners to critically think about their content knowledge acquisition [21]. The identified potential overconfidence observed in this study after receiving education is consistent with other metacognitive evaluations [22].

This study is not without limitations. The evaluation used a 1-group pretest, posttest design, which limits generalizability of the findings. While the course on best practices for coprescribing was brief and designed to be completed in 1 session, it is unknown how or whether other naloxone initiatives may have influenced participants. The academic detailing program’s enrollment was open to RUSOM and its affiliates; it is not possible to rule out selection bias as 1 factor influencing score improvements.

While metacognitive processing was shown to be important for behavior change, we do not have a long-term measure to determine whether the intervention resulted in increased naloxone prescription or even whether learners went on to implement coprescription initiatives in their practice settings. As referenced earlier, there are multiple ways to assess metacognition, of which CTF is one, and the validity of one accepted measure of metacognition has yet to be established. However, this particular method of assessing metacognition with multiple conceptual domains allows evaluators to use several diagnostic measures to understand the conditions under which knowledge gain is occurring in educational interventions. Future research could measure long-term changes in these particular scores, tracking metacognitive monitoring as skills are applied, and potentially correlate both cognitive and metacognitive changes with on-the-ground prescribing and implementation behaviors.