We conducted a randomized crossover pilot study during the spring 2026 semester in the Endocrine and Reproductive System block of the integrated second-year curriculum at USC Floyd SoM. This block includes 2 faculty-led pathology SGDs: one focused on Cushing disease and one on prostate cancer. Both cases required students to complete SDL preparation using a standardized SDL form consistent with LCME Standard 6.3 SDL competency domains. The complete study design is summarized in Multimedia Appendix 1. This study is reported in accordance with the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist (Multimedia Appendix 2).

Second-year medical students were invited to participate voluntarily. Forty-four students enrolled and were randomly assigned to 2 groups of 22 using a computer-generated random sequence; allocation was concealed from faculty evaluators until group assignments were finalized. Group 1 used the custom GPT for the Cushing disease preparation and traditional resources for prostate cancer; Group 2 completed the reverse sequence. Each student served as their own control to enable pairwise testing. Participation was voluntary and did not affect grades. The 2 SGD sessions were separated by approximately 13 days. Students were instructed not to discuss their preparation method or custom GPT access with peers during the study period, consistent with the academic honor code; they were also informed that all enrolled students would receive access to the custom GPT tool at the conclusion of the study, removing any incentive to share access prematurely. As a pilot study, the sample size was determined by voluntary enrollment within the block rather than a formal a priori power calculation; the observed effect sizes (Cohen d=0.71-0.78) provide a basis for powering a future, larger trial. The study was granted an exemption by the University of South Carolina Institutional Review Board (IRB; see the “Ethical Considerations” section). Because FERPA (Family Educational Rights and Privacy Act) regulations precluded access to student academic records from prior blocks, randomization was unstratified. End-of-block final grades were comparable between groups (Group 1: mean 87.1%, SD 6.4%; Group 2: mean 86.8%, SD 6.7%); age, sex, and prior AI experience were not collected due to these access restrictions.

We developed 2 curriculum-aligned custom GPTs, one for each case, using the GPTs feature of ChatGPT. The system prompts were shaped with input from faculty to ensure alignment with the learning goals, while still allowing flexibility in how students engaged with the material. Faculty also reviewed the case flow and embedded resources to ensure alignment with the depth and reasoning expectations of the pathology curriculum. They also refined the Socratic prompts to encourage clinical and pathophysiological reasoning rather than simple answer retrieval. The goal was to create a structured but interactive tool that could guide students through the cases in a way that supports reasoning rather than simply providing answers. The design focused on several key elements: (1) keeping the interaction tightly aligned with each case’s learning objectives; (2) using a Socratic approach that required students to explain and justify their clinical and pathophysiological reasoning; (3) guiding students through the case stages, aligned with the structure of the SDL forms; (4) incorporating case-specific clinical vignettes, imaging descriptions, and laboratory data provided in the study guide students received before the discussion sessions; and (5) directing students to open-access, peer-reviewed resources. The tools were reviewed by pathology faculty with content expertise and piloted and refined with input from students participating in the AI in Medicine Extracurricular Track at USC Floyd SoM to ensure usability and alignment with the learner experience and needs. Students assigned to the GPT condition were instructed to use the custom GPT as part of their SDL preparation. Objective usage metrics were not available to independently verify the extent or duration of engagement. Students accessed each custom GPT through their institutional USC ChatGPT account via the link provided in an email at no cost, as part of required SDL preparation for each case; no separate account creation or payment was required. Students were not given a specific time or interaction quota and used the tool ad libitum before each SGD. No automated prompts or reminders were sent to encourage use beyond standard course scheduling communications. Students did not receive formal orientation or training on use of the custom GPT before the study, although the discussion identifies a brief orientation as a potential improvement for future implementations. The content and configuration of both custom GPTs were frozen for the duration of the spring 2026 data collection period, with no functional or content changes made once data collection began.

Students in the traditional condition completed SDL preparation using conventional resources: course lecture notes, pathology textbooks, peer-reviewed literature via PubMed, and other clinical resources at their discretion. Access to the custom GPT tool was not provided to students in this condition, as the tool link was distributed exclusively to students in the GPT condition.

All students completed a standardized SDL form documenting learning goals, identified knowledge gaps, information sources, clinical reasoning synthesis, and self-assessment. SDL forms were scored by 1 of the 3 pathology small group facilitators using a grading rubric (range 6-12), assessing quality of gap identification, information synthesis, pathophysiological reasoning, and resource use. SGD performance was assessed by the faculty pathology facilitator using a grading rubric scored out of 16, evaluating diagnostic depth, clinical-pathological integration, communication, and professionalism. Both scoring rubrics are provided in Multimedia Appendices 3 and 4. Each SDL form and SGD performance was scored by a single facilitator; interrater reliability was not assessed. All faculty evaluators (IC and 2 additional pathology faculty facilitators) were blinded to study participation: they evaluated their entire small group cohort under routine conditions and were unaware of which students had enrolled in the study or which preparation method any student had used. The faculty facilitators rotated across all 9 small groups on a fixed schedule established at the start of the academic year by the pathology content leader; this schedule was followed for all pathology small groups throughout the year. The group assignment process was computer-generated and was not influenced by the principal investigator (IC), who was aware of the overall study enrollment but was blinded to each student’s preparation condition for the case being assessed on that date. Every participant was assessed by a different facilitator for each of the 2 cases, and both study conditions were represented among each facilitator’s groups in both sessions. Short-term retention was assessed using student performance on 2 in-house multiple-choice questions per quiz administered 2-4 days after each SGD, while delayed knowledge retention was assessed using 2 multiple-choice questions per case from the end-of-block examination, a customized National Board of Medical Examiners (NBME) examination (NBME Customized Assessment Services). The SDL documentation form was completed and submitted immediately before each SGD as a required deliverable, capturing the student’s self-identified learning needs, resources used, information synthesis, and self-assessment of gaps; it preceded and directly informed the discussion. Representative examples of student interactions with the custom GPT tool for each case, provided by consenting study participants, are available in Multimedia Appendices 5 and 6. Students completed surveys immediately after preparation, via a REDCap (Vanderbilt University) questionnaire accessed through a link distributed by email by KR, including 5-point Likert items on pre- and postpreparation understanding, confidence, knowledge impact, mechanism understanding improvement, and resource helpfulness, along with open-text reflections. Potential AI-related concerns were assessed across 5 prespecified domains informed by previous reports on AI use in educational settings: academic integrity, time burden, overreliance, content accuracy, and fairness across performance tiers. Other potential concerns, including hallucinations, data privacy, and inappropriate clinical guidance, were not formally evaluated.

We analyzed the quantitative data using paired t tests for within-student (crossover) comparisons and Mann-Whitney U tests for between-group survey responses (GPT condition: n=44 survey observations; traditional condition: n=42 survey observations). For the delayed retention score (range 0-2, based on 2 binary items per case), the Wilcoxon signed-rank test was used as the primary analysis because of the discrete nature of the outcome. For the SDL rubric and SGD scores, paired t tests were used because these outcomes were treated as approximately continuous, given the number of possible score values and the within-subject crossover design. We acknowledge that treating rubric-based scores as continuous represents an analytical assumption and therefore a potential limitation [13]. Chi-squared tests were used for categorical variables, and effect sizes were reported using Cohen d for paired samples, based on within-subject differences, with 95% CIs for the mean differences (t distribution) and for Cohen d (noncentral t distribution). All tests were 2-tailed, with statistical significance set at α=.05. Faculty-assessed SDL and SGD rubric scores were complete for all 44 enrolled students (N=44 paired observations). Two students did not complete the postpreparation survey in the traditional condition, yielding n=42 survey observations for that condition; these were treated as missing data under a complete-case approach, and no imputation was performed.

We also conducted exploratory post hoc analyses by grouping students into mutually exclusive tertiles based on end-of-block academic performance (low: final grade <84.5%, n=15; middle: 84.5%-90.3%, n=15; high: >90.3%, n=14) to explore whether the impact of the GPT differed by academic performance tier. Because final block grade was calculated after the intervention period, it is a concurrent rather than preintervention measure. These analyses were not prespecified, were not powered for subgroup comparisons, and should be considered hypothesis-generating. Sequence effects were examined by comparing within-student GPT-minus-traditional differences between the 2 crossover groups using a Mann-Whitney U test.

This study was reviewed by the University of South Carolina IRB and determined to be exempt from IRB review under Category 1 (research conducted in established or commonly accepted educational settings) on August 12, 2025 (study ID: STUDY00000007). As an exempt protocol, formal documented informed consent was not required by the IRB; all participants were informed of the study’s purpose, voluntary nature, and that participation would not affect grades before enrolling.

This pilot study was not registered in a public trial registry. As a minimal-risk, exempt educational pilot study conducted within a single course block, formal registration was not pursued; the authors note this as a limitation and recommend prospective registration for future, larger-scale studies of this intervention.

All 44 enrolled students were randomly assigned (22 per group), completed both cases, and were included in all analyses of faculty-assessed outcomes and delayed retention; 2 students in the traditional condition did not complete the postpreparation survey (Figure 1).

Students who used the custom GPT performed significantly better on both faculty-assessed outcomes, with moderate-to-large effect sizes (Table 1; Figure 2). SDL rubric scores were higher in the GPT group (mean 10.3, SD 1.4 vs mean 9.4, SD 1.5 out of 12; mean difference 0.9, 95% CI 0.5-1.3; P<.001; Cohen d=0.71, 95% CI 0.38-1.04). Small group discussion performance showed a similar pattern (mean 14.9, SD 0.8 vs mean 13.7, SD 1.5 out of 16; mean difference 1.2, 95% CI 0.7-1.7; P<.001; Cohen d=0.78, 95% CI 0.44-1.11). Scores were higher in the GPT condition for both cases. A significant sequence effect was observed for SGD scores (P=.007), with a larger GPT-associated difference for the prostate cancer case (+1.8 points) than for the Cushing disease case (+0.6 points). No significant sequence effects were identified for SDL scores (P=.09) or delayed retention (P=.98).

No significant differences were observed between groups in quiz performance 2-4 days after the SGD (1.8 vs 1.9 out of 2; P=.32). The primary analysis of delayed knowledge retention did not reach the conventional significance threshold (Wilcoxon signed-rank P=.052; Figure 3B). Mean scores were 1.7 versus 1.5 out of 2. Among the 44 paired observations, scores were identical for 26 out of 44 (59%) students, higher under the GPT condition for 13 out of 44 (30%), and lower for 5 out of 44 (11%). In an exploratory post hoc analysis, we divided students into 3 groups based on final block performance (low <84.5%, n=15; middle 84.5%-90.3%, n=15; high >90.3%, n=14). The difference was +0.4 in the lower-performing group, +0.1 in the middle group, and +0.1 in the higher-performing group (Figure 4A). These subgroup findings are exploratory and based on small numbers.

Students using the GPT reported larger gains in their understanding of the disease from pre- to postpreparation (+1.1 vs +0.5; P<.001). While both groups improved significantly within-group (P<.001), the GPT group ended at a higher overall level of understanding (4.3 vs 3.9; Figure 5A). Perceived impact on knowledge also differed between groups (χ²₁=9.1; P=.003; Figure 5B): 32 out of 44 (73%) GPT users reported a high or very high impact, compared with 16 out of 42 (38%) students using traditional resources. No students in the traditional group reported a “very high” impact (ie, a fundamental change in understanding), whereas 6 out of 44 (14%) GPT users did. A similar pattern was seen for mechanism understanding: 28 out of 44 (64%) GPT users reported significant or extreme improvement, compared with 12 out of 42 (29%) in the traditional group (P=.003; Figure 3A).

When we examined SDL form content in relation to the competency domains described in LCME Standard 6.3, differences emerged in how students approached SDL (Table 2). Students using the custom GPT were more likely to focus on disease mechanisms and pathophysiology as their main learning need (33/44, 75% vs 20/42, 48%), while students using traditional resources more often emphasized treatment and management algorithms (12/42, 29% vs 4/44, 9%). AI chatbot users also reported identifying knowledge gaps they were not previously aware of—a metacognitive skill closely aligned with the competency domains described in LCME Standard 6.3.

These differences were reflected in the SDL rubric scores as well, particularly in goal setting and the quality of learning documentation, which showed a moderate-to-large effect size (Cohen d=0.71) and were consistent across both cases.

Overall satisfaction with the GPT was high (Figure 4B). Most students reported meaningful benefits, with 34 out of 44 (77%) noting improved clarity in their reasoning and 31 out of 44 (70%) reporting better conceptual understanding. Only 7 out of 44 (16%) students reported improved clinical application ability. Additionally, student reflections highlighted the value of the coaching-style interaction. One student shared that “this custom AI engine is extremely impressive; having it in a coaching mindset was phenomenal in terms of making sure I truly grasped the material,” while another noted that “the customized ChatGPT was extremely helpful in mastering lecture content and breaking down complex pathophysiology into manageable chunks.” The most common worry was the time required to use the tool, mentioned by 16 out of 26 (62%) GPT-condition students who provided optional comments. Students also suggested a few practical improvements, including fewer simultaneous questions per exchange, clearer signals for when they had reached the end of a task, and occasional support for recalling foundational facts when needed. Findings across the 5 prespecified AI-related concern domains are summarized in Table 3.

This crossover pilot study suggests that a curriculum-aligned custom GPT can improve second-year medical students’ SDL preparation and small group discussion performance. The improvements observed across 2 different cases, in both faculty-assessed and student-reported outcomes, suggest that thoughtfully designed AI tools may have meaningful educational value in undergraduate medical education.

It is also important to distinguish between general-purpose AI tools and curriculum-aligned custom GPTs. General-purpose large language models are not specifically designed around individual medical curricula or the development of SDL competencies. While they may provide accurate information, that information is not necessarily aligned with course learning objectives, may not address specific reasoning gaps, and often lacks the structured, iterative engagement that supports deeper learning [4,5]. In contrast, the curriculum-aligned custom GPTs in this study were intentionally built around specific cases and learning objectives. They asked students to explain and justify their reasoning while directing them toward relevant, curriculum-aligned resources. The improvements observed, particularly in pathophysiological reasoning and SDL documentation quality, may reflect this more intentional and guided learning approach. These findings are consistent with recent controlled studies showing that structured, curriculum-aligned AI tools improve SDL outcomes compared with unassisted preparation [11,12] and add faculty-assessed objective measures that were absent from prior work.

The patterns observed in SDL content are especially interesting in the context of LCME Standard 6.3. Students using the GPT were more likely to identify disease mechanisms and pathophysiology as learning needs, a pattern consistent with the intervention’s intended emphasis on mechanistic reasoning. Students also reported identifying previously unrecognized knowledge gaps, suggesting potential support for metacognitive aspects of SDL. These findings should be interpreted as patterns consistent with SDL-related competencies rather than evidence that LCME Standard 6.3 competencies were achieved. The qualitative finding that students were able to identify knowledge gaps they were not previously aware of is particularly noteworthy, as this level of metacognitive awareness can be difficult to achieve with unguided SDL and is essential for lifelong learning [14]. By contrast, only 7 out of 44 (16%) students reported improved clinical application ability, which may reflect the preclerkship context, in which students have limited clinical experience against which to judge such changes; this could be examined in clerkship-stage learners. In the post hoc analysis, the largest retention difference was observed in lower-performing students (Δ=+0.4), and GPT users in the lowest academic tier reached SGD scores of 14.7, compared with 13.8 among higher-performing students using traditional preparation. While these subgroup findings are exploratory and based on relatively small numbers (n≈14-15 per group), they suggest that the GPT may offer helpful structure for students who struggle more with self-directed learning. This potential equity-related benefit is worth exploring further [15].

The primary analysis of delayed knowledge retention did not reach statistical significance (Wilcoxon signed-rank P=.052). Interpretation was limited by the outcome measure, which was based on only 2 end-of-block examination items per case. This result is therefore inconclusive and warrants confirmation in larger studies using more comprehensive, multi-item measures of knowledge retention.

A main limitation was related to the time required to use the tool. Students perceived preparation with the GPT as taking longer than more traditional, reading-based approaches, as reported by 16 out of 26 (62%) students who provided comments. Another factor to consider is timing. The study was conducted in the middle of the spring semester, at a point when students had already established their approaches to preparing for pathology small group discussions. Adopting a new method at that stage may have added to the perceived time burden, as students were not only learning the content but also adjusting to a different way of studying. Some targeted design changes, such as asking fewer questions at a time, providing clearer end points, and offering a brief orientation, could help reduce the time burden without compromising the educational value. Part of the observed educational benefit may reflect increased time-on-task rather than the GPT’s design alone, as 16 out of 26 (62%) GPT students who provided comments reported spending more time than usual on preparation; objective time-on-task was not measured. Future studies should measure preparation time and engagement metrics to disentangle these effects.

Several limitations should be considered when interpreting these findings. First, this pilot study was conducted at a single institution using 2 pathology cases from one preclinical block, which limits the generalizability of the results. In addition, the study was not powered by a formal a priori sample-size calculation; the enrolled sample (n=44) reflected voluntary participation within the course. Although moderate-to-large effect sizes were observed (Cohen d=0.71-0.78), these should not be interpreted as evidence that the study was adequately powered, particularly given the width of their 95% CIs. The observed effect sizes may, however, provide a useful basis for sample-size calculations in future multisite studies. Subgroup (tier-based) analyses were exploratory and should be interpreted cautiously. The delayed knowledge retention outcome was also limited by using only 2 end-of-block examination items per case, reducing the precision and sensitivity of the measure to detect modest treatment effects. In the traditional preparation condition, 16 out of 42 (38%) survey respondents reported using general-purpose AI tools. Consequently, the comparison was not between AI and no AI, but rather between a curriculum-aligned custom GPT and usual self-directed preparation, during which some students independently used general AI tools. The direction and magnitude of any resulting bias cannot be determined. It is important to note that this study did not include a condition in which students used general-purpose AI tools without curriculum alignment. Accordingly, the findings do not permit conclusions about whether a curriculum-aligned custom GPT is more effective than unrestricted general-purpose AI use. The advantage demonstrated here is relative to unguided traditional preparation, a comparison that also involves some degree of general AI use in the control condition. Future studies using 3-arm designs (traditional preparation, general-purpose AI, and curriculum-aligned AI) would be needed to test the incremental benefit of curriculum alignment directly. Additionally, objective usage metrics for the custom GPT (eg, number of logins, message counts, or time-on-task) were not collected. Engagement with the custom GPT was inferred from completion of the required SDL form rather than measured directly, limiting our ability to distinguish the effects of the tool’s design from the effects of increased time-on-task. Baseline participant characteristics from prior coursework were unavailable because of FERPA restrictions. Although end-of-block performance was comparable between the randomized groups (Group 1: mean 87.1%, SD 6.4%; Group 2: mean 86.8%, SD 6.7%), prior academic performance and prior AI experience could not be fully characterized. However, all students had institutional access to the ChatGPT platform and AI training resources provided by the University of South Carolina. A significant sequence effect was observed for SGD performance, reflecting a larger GPT-associated benefit for the prostate cancer case than for the Cushing disease case. Because case and intervention order are linked in the crossover design, case-specific and order effects cannot be fully distinguished. Notably, sequence effects were not significant for SDL documentation or delayed retention. Each SDL form and SGD performance was scored by a single facilitator, and interrater reliability was not assessed, although facilitators were blinded to preparation condition. Finally, participants could not be blinded to their assigned preparation method because of the nature of the intervention. In addition, multiple outcomes were evaluated without formal adjustment for multiplicity, increasing the possibility of a type I error. Accordingly, the findings should be interpreted in the context of the overall pattern of results rather than any single statistical test.

In this single-institution crossover pilot study of 44 second-year medical students, access to a curriculum-aligned custom GPT was associated with significantly higher faculty-assessed SDL documentation and small group discussion performance compared with unguided traditional preparation (Cohen d=0.71 and 0.78, respectively). These findings were consistent across both pathology cases. The primary analysis of delayed knowledge retention did not reach statistical significance. Assessment across 5 prespecified AI-related concern domains did not identify major concerns (Table 3). The main drawback was perceived increased preparation time, reported by 16 out of 26 students who provided comments (62%). These findings support further investigation of curriculum-aligned AI tools in undergraduate medical education in larger, multisite, prospectively registered studies.