The use of ambient artificial intelligence (AI) scribes for chart documentation has seen rapid adoption among practicing physicians [1,2]. In clinical settings, AI scribes listen in the background during a patient visit and automatically create a structured medical note from the conversation [3]. Multiple recent studies have demonstrated that the use of AI scribes among practicing physicians is associated with outcomes such as reductions in time spent on documentation, cognitive load, task load, and burnout [1,4-8].

Given the degree to which documentation burden can negatively impact the graduate medical education training experience [9], Wright et al [10] conducted a pilot of AI scribes with resident physicians. Similar to the experiences of practicing physicians, residents found that AI scribes decreased task load and documentation burden.

Documentation skills are broadly taught and assessed in undergraduate medical education curricula [11-14], and the Association of American Medical Colleges lists the ability to document a clinical encounter as 1 of 13 core entrustable professional activities that medical students should be able to perform when entering residency [15]. Additionally, adult learning theory in medical education supports experiential learning, in which students engage in hands-on practice with the concepts and tools they will encounter in the workplace [16,17].

Despite the emphasis on teaching documentation skills, acceptance of the value of experiential learning in medical education, and rapid growth of AI in undergraduate medical education curricula overall [18], to our knowledge, the use of AI scribes with medical students has not been described previously.

Documentation in medical education serves not only as a communication tool but also as a cognitive exercise through which students organize data, generate differential diagnoses, and demonstrate clinical reasoning skills [19]. Introducing automation into this developmental process may have complex effects. Automation can reduce cognitive load, which may alleviate working memory constraints during demanding tasks, thereby facilitating learning and performance [20,21]. However, the cognitive load associated with tasks such as clinical reasoning is desirable in medical education for learners to develop essential skills [20].

The possibility of harm with AI scribes has led some experts to advise against early introduction of this technology in medical training [22], although the educational implications remain unclear. Shielding current medical students from technology that they will almost certainly encounter in practice may be pedagogically unsound. In the absence of empirical evidence, there is an urgent need to understand the educational effects of AI scribes. Therefore, the purpose of this study was to examine the impact of an ambient AI scribe on the quality of preclerkship medical student note writing and explore how a human-AI hybrid workflow influences note structure and content, specifically as relates to clinical reasoning.

All 104 students submitted the required human-only write-up. A total of 45.2% (n=47) of the students consented to write a hybrid note; all completed this assignment and the student experience survey (Figure 1). Across all QNOTE elements, median evaluation scores of human-only notes were high (range 81.3-100) and were similar between students who submitted hybrid notes (n=47, 45.2%) and those who did not (n=57, 54.8%; Table 1).

In paired analysis of QNOTE elements between human-only and hybrid notes among AI scribe participants, the largest element-level change was a decline in chief complaint scores (P=.05) driven by fewer “fully acceptable” (16/47, 34% vs 21/47, 44.7%) and more “unacceptable” (7/47, 14.9% vs 3/47, 6.4%) hybrid notes as compared to human-only notes, respectively. Scores on all other elements were similar (Table 1 and Multimedia Appendix 1). Post hoc analysis of AI notes revealed that symptom duration, an explicit criterion required for full QNOTE credit in the chief complaint element, was mentioned in the chief complaint section in 17% (8/47) of the AI notes.

Of the 12 participants in the bottom quartile of human-only composite QNOTE scores, 9 (75%) exhibited improvement with hybrid scores, whereas 8 (66.7%) in the top quartile exhibited a score decline. Overall, 10.6% (5/47) exhibited score declines of at least 20 points between human-only and hybrid notes (Figure 2).

Participants agreed or strongly agreed that the AI write-up “was similar in useful content compared to my original note” (39/47, 83%), “was more concise than my note” (37/47, 78.7%), and would be “helpful as a first draft before writing my own note” (31/47, 66%). A total of 55.3% (26/47) agreed or strongly agreed that the AI write-up “left out important details that should be documented.” In total, 21.3% (10/47) agreed or strongly agreed that the AI write-up “when used regularly, may reduce my ability to learn how to write a good note” (Figure 3).

After writing high-quality “human-only” notes, interaction with notes generated by the AI scribe had little impact on the quality of the “hybrid” notes written by preclerkship medical students. Across 10 discrete note elements, the only element-level change between human-only and hybrid notes was a decline in chief complaint scores. To our knowledge, this study is the first to examine the use of ambient AI scribes for chart documentation among medical students.

Multiple prior studies have focused on the use of AI scribes among practicing physicians [1,4-8]. Tierney et al [33] and Cain et al [34] used different structured assessment instruments to demonstrate the high quality of notes after incorporation of an AI scribe in a large health system. Similarly, You et al [7] found general utility and satisfaction with AI notes in a qualitative analysis in 2 health systems. Our study suggests similar note quality with an AI workflow based on high QNOTE scores on hybrid notes and student survey responses indicating useful language and content in the AI note.

Despite these similarities with existing literature, the fundamental purpose of chart documentation for practicing physicians is different than that for medical students. In practice, the chart note serves as a communication tool between clinicians; the basis for billing, quality improvement, and performance evaluation; and a legal document [26]. While students learn about these features, they primarily write notes for educational purposes as they learn how to organize, prioritize, and synthesize information [15] and hone their ability to serve as narrators of the patient’s story [35]. Outcomes of interest in studies of AI scribes among practicing physicians, such as efficiency, cognitive load, and burnout related to note writing, are not immediately relevant to medical students. As such, in this study, we focused on students’ ability to structure a note using a conventional format while reasoning through a clinical scenario.

The QNOTE instrument emphasizes note structure with aspects of clinical reasoning embedded throughout the checklist. QNOTE elements reward clear, logical narratives that mirror a well-reasoned diagnostic process. Documentation of pertinent findings and prioritized lists are necessary for full credit on the “history of present illness” and “assessment” elements, respectively, among others [23]. As the AI scribe used in this study is designed to convert descriptions of findings into diagnostic considerations, one might expect an impact on students’ clinical reasoning. However, students on average received high scores for all elements on human-only notes with little impact after reviewing AI notes, suggesting no detriment to or augmentation of clinical reasoning overall.

The lack of impact on clinical reasoning may be due to the note writing workflow used in this study. We used a cognitive forcing strategy requiring students to generate independent judgments before exposure to potentially biasing information [29]. Cognitive forcing has been proposed as a strategy to reduce overreliance on AI [36]. In our study, requiring students to produce a human-only note prior to reviewing AI output may have preserved analytic processing and mitigated anchoring effects. While this differs from real-world clinical workflows in which AI-generated notes are the starting point, cognitive forcing through deliberate sequencing within human-AI collaboration offers a potential solution to concerns about introducing ambient AI scribes early in medical training [22].

The only observed difference between human-only and hybrid notes was a decline in “fully acceptable” scores mirrored by an increase in “unacceptable” scores in the chief complaint element. The difference in chief complaint scores was likely due to conciseness in AI notes that often omitted symptom duration, an explicit criterion for full credit on QNOTE. While errors of omission are common among AI scribes [37,38], the consistency with which duration was omitted from the chief complaint section suggests that this was due to prompt engineering [3]. In our study, over half (26/47, 55.3%) of students recognized that the AI note “left out important details that should be documented.” These omissions had minimal impact on hybrid note quality for most QNOTE elements, with scores on average approaching “fully acceptable” when written without AI. However, chief complaint was most susceptible as, on average, the scores for this element were mostly in the “partially acceptable” range among human-only notes in the first place. This decline in quality serves as a reminder of the power of automation bias, the overreliance on automated systems among early learners with AI-generated content as it can result in undesired cognitive offloading [39].

While the risk of automation bias is not unique to students, given their lack of domain expertise, early learners may be particularly vulnerable [39]. Experience and confidence in one’s own decisions decreases reliance on automated systems [40] and susceptibility to error in the face of imperfect automated outputs [41,42]. Automation bias may not be completely avoidable when confronted with complex tasks [43], but various educational safeguards have been proposed to mitigate its effect. Deliberate design of systems can help [40]. In this case, using prompt engineering better aligned with the curriculum rather than using an off-the-shelf commercial solution may have prevented the decline in chief complaint scores. However, such a solution may not be practical given resource constraints and is not a replacement for engaging students in training on the appropriate use of AI tools [22,40]. The finding that 21.3% (10/47) of the students recognized that the AI scribe “may reduce my ability to learn how to write a good note” suggests some awareness of this issue among students themselves. Our findings support the conjecture that AI tools should be released to trainees with appropriate educational safeguards and scaffolding [20,22].

The hybrid workflow may have been particularly beneficial for lower-performing students, most of whom demonstrated score increases on hybrid notes. Their improvement must be interpreted cautiously in light of potential regression to the mean as some of the highest performers saw score declines on hybrid notes. High-performing students’ human-only note scores were near the maximum possible, so ceiling compression may have limited measurable improvement and could partially explain the shift in the top quartile. As the QNOTE composite score has an upper limit, clustering near this boundary may have amplified apparent declines in the highest-performing group. It is possible that these observations may indicate true signal as they are consistent with findings in economics in which generative AI improves the quality of output for low performers while potentially having the opposite effect on high performers [44]. Establishing a causal signal will require designs that mitigate regression and ceiling effects. Future research should incorporate randomized controlled trial designs with repeated baseline measurements, longitudinal follow-up of performance trajectories, and counterbalanced AI-first vs human-first workflows (with educational safeguards) [22] to isolate the effect of AI exposure.

There are limitations to our study. This study was conducted at a single medical school within a preclerkship curriculum, limiting generalizability. Replication across multiple institutions and clerkship settings and in graduate medical education is necessary to clarify applicability longitudinally across training levels. Next, students self-selected to write hybrid notes rather than being randomly selected. However, the similar evaluation scores on human-only notes between students who submitted hybrid notes and those who did not offsets some complaint for selection bias, although unmeasured differences in motivation or attitudes toward AI may remain. Additionally, we used educational safeguards as this study took place within a summative assessment. Given the high-stakes nature of the OSCE, it was critical that the OSCE itself was unaffected by the study. Therefore, students passively received AI notes after their encounters. This workflow, which differs from ambient scribe use in workplace settings, leaves us unable to comment on the user experience with the technology. To support development of foundational skills prior to AI-supported learning [22,45], we had students write human-only notes before receiving their AI notes. This decision upheld educational best practices but represents a deviation from standard clinical workflows, which limits generalizability beyond workflows that use cognitive forcing. Finally, we did not separately analyze the AI notes because prior work has demonstrated their quality [33,34] and we had no expectation that such a note would stand alone without human review. However, future work should incorporate systematic scoring of AI notes to better understand subsection-level strengths and weaknesses to guide educational efforts.

In conclusion, the use of an ambient AI scribe following a summative OSCE for preclerkship medical students had little impact on note quality. Students found that AI notes were concise and prone to omissions and, on average, submitted high-quality notes with and without AI assistance. For low-performing students, the ambient AI scribe may have helped improve note quality. Given these findings, AI scribes could educationally enhance students’ own note writing by helping with brevity and affirming the quality of human-only notes. Enthusiasm for ambient AI scribes in medical student education should be balanced with the finding that omission of key information on an AI note can also yield lower-quality hybrid notes for elements of already lower quality without AI assistance. Our study offers a first snapshot of the impact of AI scribes on chart documentation for medical students, but future research must explore the longitudinal use of such tools. Earlier introduction could potentially jump-start skill development, although it could simultaneously trigger feelings of inadequacy [46]. Deployment later in training or with AI notes as a true first draft could increase efficiency at the expense of creating a cognitive debt due to overreliance on AI [10,22,38]. Given the rapid adoption of ambient AI scribes in clinical settings, it is crucial to further explore the educational opportunities, impact, and trade-offs of these tools.