A single-arm pilot feasibility study with a pre-post design was conducted at the Department of Plastic and Reconstructive Surgery, University Hospital BG Bergmannsheil Bochum, an academic medical center with an accredited plastic surgery residency program in Bochum, Germany. The study spanned a 12-week intervention period from July through September 2025. Reporting was guided by relevant elements of the CONSORT (Consolidated Standards of Reporting Trials) extension for pilot and feasibility studies [24]. The intervention followed a rolling production model in which faculty evaluation and selection of subsequent modules occurred concurrently with the biweekly delivery of active modules to residents, ensuring continuous content delivery throughout the study period (Figure 1A).

All residents enrolled in the plastic surgery residency program were eligible for participation using a census sampling approach (N=11). No a priori sample size calculation was performed, consistent with the pilot feasibility design; the sample comprised the entire residency cohort to maximize enrollment and provide preliminary data across training levels.

All residents actively enrolled in the plastic surgery residency program at any postgraduate year (PGY) level were eligible; no exclusion criteria were applied.

The 11 enrolled residents were distributed across junior (PGY 1-2; n=4), mid-level (PGY 3-4; n=2), and senior (PGY 5-6; n=5) training years. Participation was voluntary, and no incentives were provided beyond access to the educational content. All 11 participants completed both pre- and postintervention assessments. Two postassessments could not be linked to their corresponding pre-assessments due to inconsistent pseudonym entry, yielding 9 matched pairs for paired analyses (Figure 1B).

All AI-generated content was independently evaluated by 2 attending plastic surgeons, each with more than 10 years of clinical experience and board certification in plastic surgery. Both faculty members were blinded to each other’s assessments during independent evaluation. To ensure standardization, faculty were provided with a written grading guideline outlining the evaluation rubric, scoring anchors, and examples of acceptable versus unacceptable item features. Faculty evaluated each question and its accompanying explanation across five dimensions using standardized 5-point scales (1=poor, 5=excellent):

The overall quality index for each item was calculated as the mean across all 5 dimensions. The complete rubric with detailed scoring criteria is provided in Multimedia Appendix 1. Faculty reviewed all 60 items in a structured, module-by-module sequence before pilot implementation. Interrater reliability was assessed using intraclass correlation coefficients (ICC; two-way mixed-effects model, single measures, ICC[3,1]) [27].

Prior to module release, all residents completed a baseline questionnaire. Following the 12-week intervention, residents completed an identical questionnaire with additional post-only measures. Table 1 lists all questionnaire items with verbatim wording, response formats, and assessment timing.

Knowledge change was operationalized as the difference between postintervention and pre-intervention self-reported subspecialty knowledge scores on the 7-point Likert scale. Composite knowledge change was calculated as the mean of individual change scores across all 6 subspecialty topics. Confidence change was calculated identically for the confidence domain.

Explanation quality was operationalized as the mean faculty evaluation score across the 5 quality dimensions for the explanation component of each item, as rated independently by both evaluators. This metric was used for correlation analyses examining the relationship between content quality and learning outcomes at the module level.

Outcomes were organized into a hierarchical framework reflecting the pilot nature of this study.

As a pilot feasibility study designed to assess implementation feasibility and generate effect size estimates for future trials, formal a priori power calculations were not conducted. The sample included all eligible program residents (N=11).

This study was reviewed by the Ethics Committee of the Medical Chamber of Westfalen-Lippe (Ärztekammer Westfalen-Lippe). The committee confirmed that formal ethical approval was not required, as the study exclusively involved anonymized data and therefore posed no risk to participant rights or safety, in accordance with the Declaration of Helsinki [29] and section 15 of the Professional Code of Conduct (reference 2025-605-f-N). All participants provided informed consent.

All 11 residents enrolled in the plastic surgery residency program consented to participate and completed both baseline and postintervention assessments. The cohort comprised junior (PGY 1-2; n=4), mid-level (PGY 3-4; n=2), and senior (PGY 5-6; n=5) trainees. Two postassessments could not be linked to their corresponding preassessments due to inconsistent pseudonym entry, yielding 9 matched pairs for paired analyses (Figure 1B). Module completion ranged from 9 to 11 submissions across the 6 modules, with modules 1-4 completed by 10 residents (91%), module 5 by 11 (100%), and module 6 by 9 (82%) (Figure 1C). Estimated cumulative time spent was 60 to 120 minutes over 12 weeks, equivalent to 10 to 20 minutes per module and comparable to 1-2 traditional didactic sessions.

Content engagement was moderate to high: 8 of 11 residents (73%) reported answering questions and reflecting on explanations, and 2 (18%) reported additionally consulting related topics (Table S1 in Multimedia Appendix 3). Postintervention acceptability ratings were uniformly positive, with all 6 items exceeding the scale midpoint of 4 on the 7-point scale (1 = strongly disagree, 7 = strongly agree) (Figure 1D). Usability received the highest rating (median 6, IQR 6-6; 10/11 scored 5 or higher). Willingness to continue was similarly high (median 6, IQR 5-7; 9/11 scored 5 or higher). Knowledge gap identification and overall training value each had 9 of 11 residents scoring 5 or higher. Engagement quality had a median of 6 (IQR 4.5-6.0; 8/11 scored 5 or higher). Perceived knowledge improvement received the lowest acceptability rating (median 5, IQR 4-6; 7/11 scored 5 or higher). Item-level response distributions are provided in Table S1 in Multimedia Appendix 3.

Of the 60 AI-generated items evaluated by 2 board-certified faculty members, 42 were selected for the pilot (7 per module). The overall quality index for selected items was 4.68 out of 5 (SD 0.44), compared with 4.28 out of 5 for excluded items (Figure S3 in Multimedia Appendix 2). Question and explanation quality showed strong positive correlations across dimensions (pedagogical value r=0.94, accuracy r=0.91; Figure S2 in Multimedia Appendix 2), while the safety dimension showed negligible correlation with other metrics (r=–0.05 with relevance), indicating that clinical safety standards were maintained independent of variation in other quality dimensions. Score distributions for questions and explanations showed close alignment within each subspecialty module (Figure 2B). Among the 5 evaluation dimensions, safety received the highest mean rating (4.80), followed by pedagogical value (4.70), accuracy (4.68), clarity (4.63), and relevance (4.60) (Figure 2C). The Safety dimension remained consistently high across all modules (range 4.71-4.89), independent of variation in other dimensions. Module-level quality varied, with burns and critical care scoring highest (4.89) and hand and peripheral nerve surgery scoring lowest (4.19; notably, this was the only module generated with Gemini 2.5 Pro Preview rather than the stable release) (Figure 2A). Individual item ratings are provided in Table S2 in Multimedia Appendix 3. Faculty quality ratings for all 60 generated items and interdimensional correlations are shown in Figures S1 and S2 in Multimedia Appendix 2.

Interrater reliability for the overall quality evaluation was poor (ICC(3,1)=0.231, 95% CI 0.154-0.305; P<.001). Among individual dimensions, pedagogical value showed the highest agreement (ICC=0.314, 95% CI 0.143-0.466) and relevance the lowest (ICC=0.180, 95% CI 0.002-0.348). The low agreement is consistent with the subjective nature of content quality judgments between 2 independent raters and the limited precision afforded by a 2-rater design.

Overall person-level accuracy across the 42 selected items averaged 72.5% (SD 14.1%), falling within the recommended 65% to 75% zone for formative assessment (Figure 3A-C). Module-level accuracy ranged from 55.8% (sarcoma and oncology) to 92.1% (core surgical and practice) (Table 2).

Item-level psychometric properties varied across modules (Table 2). Mean item difficulty ranged from 0.56 (sarcoma and oncology) to 0.92 (core surgical and practice). mean item discrimination (point-biserial correlation) ranged from -0.34 (core surgical and practice) to 0.46 (burns and critical care; sarcoma and oncology). Internal consistency was low overall, with 3 of 6 modules yielding negative Cronbach ɑ values (–0.82 to –0.20). Each module’s 7 items spanned distinct subtopics rather than targeting a single construct, which likely accounts for this pattern. Burns (ɑ=0.69) and sarcoma (ɑ=0.74) were the only modules with positive internal consistency, and both also had the highest item discrimination.

At the module level, perceived difficulty showed a strong negative association with MCQ accuracy (r=–0.75; Figure 3A), indicating that modules rated as more challenging yielded lower scores. Perceived relevance (r=0.75; Figure 3C) and content quality (r=0.59; Figure 3B) showed positive associations with accuracy. Content quality correlated most strongly with knowledge change (r=0.79; Figure 3E), followed by perceived relevance (r=0.69; Figure 3F), while difficulty showed only a weak negative association (r=–0.31; Figure 3D). Conversely, core surgical and practice showed the highest accuracy (92.1%) but no knowledge change (d=0), suggesting a ceiling effect. Module-level perception outcomes and the complete correlation matrix are provided in Table S3 in Multimedia Appendix 3.

Among the 9 matched pairs, composite self-perceived confidence improved significantly (median change 0.33; d=0.65; P=.047), while composite knowledge showed a medium effect that did not reach statistical significance (median change 0.17; d=0.71; P=.09) (Figure 4A; Table 3).

Topic-specific outcomes are summarized in Table 3. The largest effects emerged in hand and peripheral nerve surgery, with significant improvements in both knowledge (d=1.48; P=.02) and confidence (d=1.41; P=.02). Aesthetic and breast surgery showed a large knowledge effect size (d=0.93) that did not reach significance (P=.06). Burns and critical care was the only module with a negative knowledge trajectory (d=–0.23), while core surgical and practice showed no knowledge change (d=0). A consistent inverse relationship between baseline scores and improvement was observed across modules (r=–0.85; Figure 4B), with the largest gains occurring in topics with the lowest baseline self-assessments.

A forest plot of all effect sizes is available in Figure S4 in Multimedia Appendix 2. Complete parametric statistics are provided in Table S4 in Multimedia Appendix 3 for methodological transparency.

No significant changes in educational resource use were observed over the 12-week intervention period (textbooks P≥.99, journal articles P=.25, departmental materials P=.25, YouTube videos P≥.99, databases P=.62, and AI tools P=.62; Figure 5A; Table S5 in Multimedia Appendix 3). Weekly study time remained stable (pre median 2 hours, IQR 1-3; post median 2 hours, IQR 1.5-2; P=.45). Traditional resource use, including textbooks (P≥.99), journal articles (P=.25), and departmental materials (P=.25), did not change. This pattern suggests the intervention operated as an additive supplement rather than displacing established learning resources. All 5 sense-of-community dimensions remained stable (program addressing shared needs P≥.99, peer discussion P≥.99, connection to peers P=.50, psychological safety P≥.99, and peer support P≥.99; Figure 5B), with median ratings at or above 5 on the 7-point scale throughout. This stability provides preliminary evidence that asynchronous digital learning did not disrupt the apprenticeship model central to surgical training. AI tool familiarity showed the largest contextual change (pre median 3, IQR 2-5; post median 4, IQR 4-5; d=0.70; P=.12), though this did not reach statistical significance (Figure 5C).

This pilot study provides preliminary evidence that AI-generated microlearning modules can be feasibly integrated into surgical residency education. Over 12 weeks, all 11 enrolled residents completed postintervention assessments, and module submissions ranged from 9 to 11 across the 6 subspecialty topics. Cumulative time spent of 60 to 120 minutes was distributed across brief, self-paced sessions (10-20 minutes per module), suggesting compatibility with clinical schedules. Faculty evaluation confirmed high content quality across all modules, and postintervention acceptability ratings were uniformly positive. Among 9 matched pairs, composite self-perceived confidence improved significantly (d=0.65; P=.047), while composite knowledge showed a medium effect size (d=0.71) that did not reach statistical significance (P=.09). These preliminary findings support the feasibility of this approach and provide effect size estimates to inform the design of future confirmatory trials.

Our findings extend the emerging literature on LLM-generated medical education content. Prior studies have established that AI systems can generate MCQs achieving quality ratings comparable to human-authored items when evaluated by content experts [15,16,22]. However, most investigations have examined question quality as an endpoint rather than deploying AI-generated content within authentic educational interventions [21,22]. Our study moves beyond quality benchmarking by embedding curated AI content into a 12-week residency curriculum and measuring educational outcomes, albeit in an exploratory pilot design with a small sample.

When positioned within the broader literature, our composite effect sizes (d=0.65-0.71) align with the upper range of effects reported in continuing medical education interventions, where effect sizes corresponding to Cohen d of approximately 0.4 to 0.7 are typical [30]. These are notably smaller than effects reported in undergraduate medical education meta-analyses (SMDs of 1.2-3.4 for case-based and simulation-based learning) [31,32], likely reflecting the greater baseline knowledge and clinical experience of postgraduate learners. Against this benchmark, our preliminary findings fall within the upper range of reported continuing medical education effect sizes, despite the minimal time burden, asynchronous format, and reliance on AI-generated content.

Module-level analyses offered exploratory insight into content characteristics associated with educational outcomes. Perceived content quality (r=0.79) and relevance (r=0.69) showed strong positive associations with knowledge change, whereas perceived difficulty showed only a weak negative association (r=–0.31). These patterns are consistent with prior work demonstrating that perceived clinical relevance fosters knowledge retention [33]. If confirmed in larger samples, these associations carry practical implications: faculty reviewers and developers of AI-generated educational content might prioritize clinical relevance and pedagogical quality as primary selection criteria.

A consistent inverse relationship between baseline self-assessed knowledge and improvement emerged across modules (r=–0.85), with the largest gains in hand and peripheral nerve surgery (d=1.48; P=.02) and aesthetic and breast surgery (d=0.93; P=.06), where baseline scores were lowest. Conversely, modules with higher baseline scores showed negligible or no change (core surgical d=0; burns d=–0.23). This floor-ceiling pattern is consistent with targeted learning theory and suggests that AI-generated microlearning may be particularly suited for addressing identified knowledge gaps rather than reinforcing already-strong domains.

The structured faculty evaluation process resulted in the selection of 42 items from an initial pool of 60, elevating the mean quality index from 4.28 to 4.68 on a 5-point scale. The safety dimension remained consistently high across all modules (range 4.71-4.89) and showed negligible correlation with other quality dimensions, suggesting that LLM outputs can maintain appropriate clinical safety standards even when other aspects of content quality vary. This structural independence is noteworthy given concerns about AI-generated medical content potentially propagating harmful recommendations. Despite this independence, the expert curation process proved essential for overall content quality, suggesting that human expert oversight remains important for deploying AI-generated content in medical education contexts where content errors carry safety implications.

However, interrater reliability between the 2 faculty evaluators was poor (ICC(3,1)=0.231, 95% CI 0.154-0.305). This finding highlights the inherent subjectivity of content quality judgments and the limited precision afforded by a 2-rater design. Future implementations should consider larger evaluation panels and consensus-based approaches to improve reliability. Despite the low ICC, both raters independently selected similar items for inclusion, suggesting agreement on which items were clearly strong or clearly weak, even when absolute ratings diverged.

Internal consistency was low, with negative Cronbach ɑ values in 3 of 6 modules, reflecting the heterogeneous content within each module where items spanned distinct subtopics. The 2 modules with positive internal consistency (burns, ɑ=0.69; sarcoma, ɑ=0.74) also showed the highest item discrimination, suggesting that future item development might benefit from greater subtopic coherence within modules.

The stability of residents’ learning ecosystems throughout the intervention provides preliminary evidence that asynchronous AI-based microlearning can supplement existing educational resources without displacement. Weekly study time, traditional resource use, and all 5 sense-of-community dimensions remained unchanged, suggesting the modules functioned as an additive intervention. This nondisruption is relevant given concerns that digital learning tools might undermine the apprenticeship model central to surgical training.

AI tool familiarity showed the largest contextual change (d=0.70), though this did not reach statistical significance (P=.12). Exposure to faculty-evaluated AI content may function as experiential onboarding, potentially preparing residents for the AI-augmented clinical environments they will increasingly encounter. Whether curated AI educational content can serve as an entry point for broader AI literacy warrants investigation in future studies.

This study has several methodological strengths. The rolling production model enabled continuous content delivery while maintaining rigorous faculty evaluation, demonstrating a workflow for AI-assisted curriculum development. The standardized prompting strategy and comprehensive evaluation rubric provide reproducible frameworks for other programs seeking to implement similar interventions. The multidimensional assessment captured not only efficacy outcomes but also feasibility, acceptability, and ecosystem integration metrics essential for translating pilot findings into sustainable educational practice.

Several limitations warrant consideration when interpreting these findings. First, the single-arm design without a control group precludes causal attribution; secular trends, Hawthorne effects, or concurrent clinical experiences may have contributed to observed changes. Second, the small sample size (n=9 matched pairs) limited statistical power, as reflected in wide confidence intervals, and some outcomes that showed medium-to-large effect sizes did not reach statistical significance. Third, self-reported knowledge and confidence outcomes may not reflect objective performance on standardized examinations or clinical competency assessments. Fourth, interrater reliability for the faculty evaluation was poor (ICC=0.231), limiting the precision of content quality judgments. Fifth, the single-institution setting within a German academic plastic surgery program, the 12-week follow-up period, and the reliance on a single LLM (Google Gemini 2.5 Pro) with a specific prompting strategy all constrain generalizability. However, the evaluation framework and curriculum structure remain applicable regardless of which AI system generates initial content.

This pilot study provides evidence that AI-generated microlearning modules are feasible and acceptable for plastic surgery residency education. Faculty-evaluated content achieved high-quality ratings, and residents showed strong engagement with meaningful preliminary improvements in self-perceived confidence. The intervention was integrated into existing learning ecosystems without disrupting traditional resources or peer learning dynamics. The methodology described here, combining structured prompt engineering with subspecialty-specific content generation, systematic faculty evaluation, and learner-centered outcome assessment, may provide a framework for programs exploring similar AI-assisted educational interventions, though replication across institutions and specialties is needed. These findings, while preliminary, support proceeding to adequately powered randomized trials evaluating AI-generated microlearning against standard educational approaches, with attention to objective outcome measures, longer follow-up, and multi-institutional settings.