Hospitals that want to harness AI for clinical documentation, decision support, and administrative automation [1-4] face a workforce problem: the technologies that make these applications possible, retrieval-augmented generation (RAG) [5], the model context protocol (MCP) [6], and orchestration frameworks such as LangGraph [7], are evolving faster than the people who must build, adapt, and evaluate them [8,9]. The question is no longer whether to train hospital staff in advanced AI, but how.

Existing training efforts offer limited guidance. The 2024 Best Evidence Medical Education (BEME) scoping review by Gordon et al [10] confirmed that published AI education programs remain predominantly foundational—covering AI literacy, prompt engineering, and learners’ perceptions of generative AI tools such as ChatGPT [11-13]—and target mainly medical students and residents [14,15]. Evidence on intensive, multiweek programs that teach advanced technologies to the broader hospital workforce—IT specialists, health information managers (HIM), clinicians, researchers, and administrative staff—is scarce [16-18]. No published study, to our knowledge, has reported the design and outcomes of training that covers RAG, MCP, AI agent design, and workflow orchestration together.

Self-efficacy theory [19] predicts that mastery experiences, particularly hands-on practice with novel technologies, are the strongest driver of sustained technology adoption [20-22]. Consistent with this, recent ChatGPT education studies report that structured training significantly raises perceived AI competency among health care professionals [8,12], and experiential learning approaches outperform didactic instruction for skill-based outcomes [23]. However, most evaluations remain limited to satisfaction surveys (Kirkpatrick level 1 [24,25]), with few assessing knowledge gains (Level 2) or behavioral transfer intentions (Level 3) [26,27]. Our previous evaluation of a health informatics analyst program at the same institution demonstrated that intensive education can reshape job roles and skill profiles [18]; this study extends that work to advanced generative AI technologies.

This study has two complementary aims: (1) to describe the design rationale, curriculum structure, and implementation experience of an 8-week, 56-hour intensive generative AI training program for a multidisciplinary hospital workforce at a tertiary academic medical center; and (2) to evaluate its outcomes using the Kirkpatrick framework across Levels 1‐3, providing empirical evidence that contextualizes the lessons learned. We addressed four research questions (RQs):

This was a single-group, pre-post survey study using an independent samples design conducted at a single tertiary academic medical center in Seoul, South Korea. The training program was administered from September to November 2025, with surveys distributed immediately before (pretraining) and after (posttraining) the program. Because survey responses were collected anonymously without personal identifiers, individual-level matching between pre- and posttraining responses was not possible; therefore, pre- and posttraining groups were treated as independent samples. This study adhered to the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines for cross-sectional studies [28].

Eligible participants were employees of Asan Medical Center, Ulsan University Hospital, or Gangneung Asan Hospital who enrolled in the advanced generative AI training program. The program was delivered on-site at Asan Medical Center with simultaneous online broadcast to the 2 regional affiliate hospitals, enabling multisite participation. Inclusion criteria were (1) current employment at one of the participating institutions, (2) voluntary enrollment in the program, and (3) age 18 years or older. Participants who attended fewer than 80% of the total training hours were excluded; all 83 enrollees met this threshold. Recruitment did not use an open call; instead, the program was promoted internally within candidate departments through departmental announcements and circulated notices, and interested employees enrolled voluntarily. Because the program’s primary aim was to build AI-agent development capability, enrollment was prioritized in the order of IT workforce, health-information staff, clinicians, and researchers, with IT and related departments approached first.

All 83 enrollees completed the pretraining survey. Of these, 64 responded to the posttraining survey (response rate: 77.1%). A total of 147 survey responses were thus analyzed (pre: n=83; post: n=64). Demographic and occupational characteristics, professional group and years of work experience, were self-reported by participants as part of both the pre- and posttraining surveys, without collection of personal identifiers, and are detailed in the Results section and Table 1.

Survey items were developed by the research team specifically for this program, grounded in the competency domains of the training curriculum (MCP, RAG, LangGraph, AI agent design, and domain-specific applications) and informed by Compeau and Higgins [20] computer self-efficacy scale, adapted to the generative AI domain. The instrument underwent internal review by the program instructors prior to deployment. Post hoc psychometric analyses (Cronbach α, item-total correlations, exploratory factor analysis [EFA]) are reported in Tables S5-S6 in Multimedia Appendix 1.

Survey instruments consisted of 100 pretraining items and 154 posttraining items organized into 11 sections. All attitudinal and self-efficacy items used a 5-point Likert scale (1=strongly disagree to 5=strongly agree). The measures were mapped to the Kirkpatrick 4-level training evaluation model [24,25] as follows:

Primary outcomes (pre-post comparison):

Secondary outcomes (pre-post comparison):

Posttraining only outcomes:

All analyses were performed using Python 3.x (pandas v2.0, SciPy v1.11, factor_analyzer). The anonymous design required treating pre- and posttraining groups as independent samples. Mann-Whitney U tests [30] were used for pre-post comparisons, with the rank-biserial correlation r (= |Z| / √N; Rosenthal [31]) as the primary effect size, interpreted as small (≥.10), medium (≥.30), or large (≥.50) per Cohen [32]. Cohen d was additionally reported to facilitate comparison with prior literature. The significance threshold was α=0.05 (2-tailed); posttraining-only measures (S9, S10, and S11) were summarized descriptively. Chi-squared tests assessed demographic comparability between the 2 independent samples.

Subgroup analyses by professional group were exploratory: Cohen d with 95% CIs was computed without formal hypothesis testing or multiple-comparisons correction, given small cell sizes (n=7‐9 for non-IT groups). Full inferential statistics are available in Table S2 in Multimedia Appendix 1.

Internal consistency was assessed using Cronbach α (acceptable≥0.70, good≥0.80 [33]). EFA (principal axis factoring, direct oblimin rotation) and corrected item-total correlations provided post hoc construct validity evidence (Tables S5-S6 in Multimedia Appendix 1).

A worst-case sensitivity analysis assigned all 19 nonrespondents’ posttraining scores equal to their professional group’s pretraining mean (Table S7 in Multimedia Appendix 1). A post hoc power analysis indicated the minimum detectable effect at 80% power was r=.24 (Table S4 in Multimedia Appendix 1).

The study protocol was approved by the Institutional Review Board of Asan Medical Center (IRB number 2025‐1070; approved August 28, 2025). Written informed consent was waived because data were collected anonymously through voluntary online surveys, participation posed no more than minimal risk, and the study could not practicably be conducted without the waiver. The first page of the survey informed participants of the study purpose, voluntary nature, and data handling procedures. Participants received no monetary compensation; program tuition was fully funded through the national continuing-education initiative administered by the KOHI.

All 83 enrollees completed the program and responded to the pretraining survey; 64 responded to the posttraining survey (response rate: 77.1%; pre: n=83; post: n=64; Figure 1). Demographic characteristics of the pre- and posttraining groups are presented in Table 1. The occupational distribution was comparable between the 2 groups (IT specialists: 57.8%, 48/83 pre vs 51.6%, 33/64 post; HIM: 12%, 10/83, vs 14.1%, 9/64; clinicians: 10.8%, 9/83, vs 10.9%, 7/64; researchers: 10.8%, 9/83, vs 12.5%, 8/64; administrative staff: 8.4%, 7/83, vs 10.9%, 7/64); retention was lower among IT specialists (33/48, 68.8%) than among non-IT groups (78%, 7/9 to 100%, 7/7; see Limitations). The majority of participants in both groups had over 11 years of experience (59%, 49/83 pre vs 54.7%, 35/64 post). Chi-squared tests confirmed no statistically significant differences in occupational (χ²₄=1.23; P=.87) or experience-level (χ²₃=0.80; P=.85) distributions between the 2 independent samples.

Posttraining AI knowledge self-efficacy scores were significantly higher than pretraining scores (mean 3.40, SD 0.69 vs mean 2.34, SD 0.83; U=875.5, r=.574, d=1.37; P=3.50×10^–¹²; Table 2, Figure 2). Internal consistency was excellent (α=0.909 pre, α=0.915 post; Tables S5-S6 in Multimedia Appendix 1). At the domain level, the technologies that received the greatest curricular investment and had the lowest baseline scores showed the largest between-group differences: MCP (d=1.57), medical Q&A systems (d=1.39), and AI agent design (d=1.39), whereas foundational domains with higher baselines showed smaller differences (basic AI understanding: d=0.55; Table 2).

All 5 professional groups showed the largest between-group difference in MCP, and non-IT groups consistently showed larger differences than IT specialists (Table S2 in Multimedia Appendix 1). For MCP, clinicians showed the largest difference (d=2.88, 95% CI 1.42-4.33), followed by HIM (d=2.45, 95% CI 1.23-3.67), researchers (d=2.07, 95% CI 0.87-3.28), and administrative staff (d=1.83, 95% CI 0.55-3.11); IT specialists showed the smallest difference (d=1.36, 95% CI 0.87-1.85). These subgroup findings should be considered exploratory given small cell sizes (n=7‐9 for non-IT groups; Table S2 in Multimedia Appendix 1).

Job-specific AI competency (S6) scores were significantly higher posttraining (mean 2.77, SD 0.95 vs mean 1.95, SD 1; P=3.59×10^–⁶, r=.382, d=0.85; α=0.938-0.973 pre, α=0.877-0.981 post; Table 2). However, unlike general AI self-efficacy (S4), job-specific competency remained below the scale midpoint (2.77/5.0), indicating a knowledge-practice gap.

Three secondary pre-post comparisons did not reach significance (Table 2; Table S4 in Multimedia Appendix 1): AI attitudes (S2: r=.025; P=.76), with the mean of the 6 positively worded items (Q4-Q9) already at ceiling at baseline (M=4.29/5); the full 9-item S2 scale mean, with reverse coding applied to Q1-Q3, was 3.79 (Table 2); digital competency (S5: r=.138; P=.09); and AI daily usage (S3), where the proportion using 3+ hours daily increased modestly from 9.6% to 17.2%.

Overall training satisfaction was 4.03/5.0 (SD 0.85), while the domain-averaged score was 3.48 (SD 0.63; α=0.939; Table 3, Figure 3). Satisfaction varied substantially across domains: team project experience was rated highest (mean 3.98, SD 0.84), followed by instructor quality (mean 3.91, SD 0.72), teaching methods (mean 3.58, SD 0.69), and content & practice (mean 3.40, SD 0.67). Curriculum design was rated lowest (mean 2.95, SD 0.87), particularly learning pace (mean 2.91, SD 1.11). Item-level results for all posttraining scales (S9-S11) are reported in Table S3 in Multimedia Appendix 1.

Of the 12 capstone teams, 11 presented functional AI prototypes spanning clinical, administrative, and IT applications (representative examples in Table 4; full list in Table S8 in Multimedia Appendix 1); one regional affiliate team was unable to present due to scheduling constraints. One project, a medical dictionary-based RAG/query chatbot system (“WorksBot”), received the Minister of Health and Welfare Award. One prototype, the hospital document knowledge retrieval RAG system, has since been developed into an institutional service (“AI-Docs”) and has been in active pilot use since May 2026 across multiple departments, including the Department of Nursing and the Big Data Research Center, with hospital-wide deployment planned within 2026. WorksBot remains a prototype and is not yet in operational service; a refined version is planned following the RAG pilot. Both efforts are being advanced by the Department of Digital Innovation and Support.

Perceived changes (S10; α=0.928; Figure 4) showed high perceived knowledge improvement (mean 4.27, SD 0.62) and technology acceptance (mean 4.20, SD 0.62), but lower workplace application confidence (mean 3.50, SD 0.84), reinforcing the knowledge-practice gap observed in S6. Workplace application intention (S11; α=0.909; Figure 5) was high overall (mean 4.02, SD 0.70), with continued learning rated highest (mean 4.39, SD 0.75) and novel problem solving rated lowest (mean 3.78, SD 0.95).

All 64 posttraining respondents answered the open-ended question: “What aspect of change or growth did you most strongly feel through this 8-week program?” Eight responses (12.5%) were too brief to classify; the remaining 56 responses were grouped by the research team into 5 recurring themes. Because 8 of these responses expressed 2 distinct themes (none was assigned to more than 2), this grouping yielded 64 theme-level assignments in total. The most prevalent theme was conceptual clarity (24 assignments, 37.5% of all 64 respondents), with participants reporting that previously abstract concepts, particularly MCP and RAG, became concrete and actionable. The second was confidence and reduced fear (14 assignments, 21.9%); participants described a shift from anxiety about AI to self-efficacy, with representative responses including “the vague fear I had has been resolved to some extent” and “my first goal was confidence recovery, and I fully achieved it.” Third, hands-on tool experience (12 assignments, 18.8%) was frequently cited, with participants specifically naming vibe coding, Cursor, and Claude Code as transformative experiences; notably, several nondevelopers described coding for the first time. Fourth, workplace application ideation (10 assignments, 15.6%) emerged, with participants reporting that they began identifying specific AI applications in their own work contexts. Finally, a smaller group reported a general attitude shift (4 assignments, 6.2%).

This study describes the design, implementation, and outcomes of an 8-week intensive generative AI training program for a multidisciplinary hospital workforce. To our knowledge, this is the first study to report both the design and the pre-post evaluation of a hospital-workforce program integrating 4 advanced, agent-level generative AI technologies, MCP, RAG, LangGraph orchestration, and AI agent design, across both IT and non-IT roles, spanning IT specialists, clinicians, HIM, researchers, and administrative staff. This advances AI education research beyond its prevailing focus on AI literacy and prompt engineering [10-13] toward agent-level competencies, and, through the capstone-to-deployment pathway, links classroom learning to institutional adoption. Five principal lessons emerged from the convergence of quantitative survey data and participants’ open-ended reflections.

The program’s most consequential design decision was allocating 37% of instructional time to MCP, a technology virtually none of the participants had encountered. The quantitative data validated this bet: MCP showed the largest between-group self-efficacy difference across all professional groups (d=1.57; Table 2), and the 3 novel domains (MCP, medical Q&A, and AI agent design) consistently outperformed foundational topics. The open-ended responses corroborated this pattern—MCP and RAG were the most frequently named technologies when participants described their growth, and the dominant theme was that “previously abstract concepts became concrete and actionable.” This inverse relationship between baseline familiarity and learning yield is consistent with self-efficacy theory’s prediction that mastery experiences are most impactful where prior competency is lowest [19], and suggests that training programs should resist the temptation to stay at the prompt-engineering level.

Non-IT professionals showed consistently larger between-group differences than IT specialists across most domains (Table S2 in Multimedia Appendix 1), particularly for MCP (clinicians: d=2.88; HIM: d=2.45). This finding challenges the assumption that advanced AI training should be restricted to technical staff. Qualitatively, nondevelopers described transformative experiences: several reported coding for the first time through vibe coding, and one described the capstone as an opportunity to “bring MCP and RAG into hospital work and implement visible results through hands-on practice—a fundamentally different experience from watching lecture slides.” The mixed-expertise team structure, which received the highest satisfaction rating (mean 3.98, SD 0.84), provided the technical scaffolding non-IT professionals needed while leveraging their domain knowledge for health care-specific applications.

Despite significant self-efficacy gains (r=.574) and high perceived knowledge improvement (mean 4.27, SD 0.62), job-specific competency remained below the scale midpoint (2.77/5) and workplace application confidence was rated only 3.50—a gap that converged across 3 independent measures (S6, S10, and S11). Yet, the open-ended responses revealed a more nuanced picture: 15.6% of respondents spontaneously reported generating specific workplace application ideas, and participants described shifts from passive understanding to active problem identification (“I started thinking about improvements to the systems we currently operate”). The Unified Theory of Acceptance and Use of Technology (UTAUT) framework [22] suggests that training without corresponding organizational change management may produce motivated but unsupported learners [34,35]; health care settings compound this through restricted electronic medical record (EMR) environments, data governance requirements, and concerns about LLM hallucination. The gap is not a failure of training but a predictable transition challenge—one that infrastructure support (funded tool subscriptions, secure internal cloud environments) and structured posttraining mentoring can help bridge [36,37].

Curriculum pacing was rated lowest among satisfaction domains (mean 2.91/5.0, SD 1.11), even as overall satisfaction was substantially higher (4.03/5.0). This signals that participants valued the program but struggled to keep pace—a predictable tension when teaching rapidly evolving technologies to professionals ranging from experienced developers to clinical staff encountering code for the first time. The qualitative data illuminated both sides: some participants celebrated “coding fearlessly for the first time,” while the pacing score suggests others felt left behind. In practice, hands-on exercises were already split into application and development tracks matched to baseline skill; future iterations should additionally consider prerequisite assessment and supplementary catch-up sessions to accommodate varying baseline competencies.

Teaching a mixed-expertise cohort posed distinct challenges. IT and health-information staff were released from their regular duties to attend, whereas some clinicians and researchers used personal annual leave. At the outset, the wide range in baseline skill, from staff who had never coded to experienced IT specialists, was anticipated as a key challenge; 2 design choices mitigated it (Methods): the curriculum minimized AI theory in favor of practical concepts, agentic methods, and concrete MCP use cases, and each capstone team paired an intact IT-subdepartment core with non-IT domain members. Collaboration across backgrounds still required deliberate team-level scaffolding, and recruitment skewed toward IT staff, consistent with the program’s IT-first prioritization and department-based, voluntary enrollment—with comparatively fewer clinical participants.

The capstone component, supported by 3‐6 hours of weekly mentoring per team from a dedicated instructor, culminated in functional prototypes from 11 of 12 teams. The qualitative data revealed why: participants described the capstone as where abstract knowledge “became real”—the transition from understanding concepts to building something that works. More importantly, this bridge extended beyond the classroom: one capstone project has since advanced into institutional pilot use (AI-Docs; see Results). This progression from capstone prototype into pilot clinical use provides emerging Kirkpatrick level 3 evidence that training-generated projects can move into institutional adoption, suggesting the capstone model can serve as a pathway for AI innovation that outlasts the training period itself.

This program was conceived as the deliberate first phase of a staged, institution-wide strategy: we began with staff positioned to enable broad adoption (IT, health-information, related-department clinicians, and researchers), expecting their agent-level competency to seed hospital-wide AI deployment—a trajectory already visible in the AI-Docs pilot. Subsequent phases are planned to extend structured training to frontline clinicians and other departments and to embed the curriculum as a recurring continuing professional development (CPD) track, a required or elective module, or a template for other hospitals through the national framework that funded it. So framed, this voluntary first cohort is a starting point, not an endpoint: audiences not yet aware of agent-level AI are reached through later, role-targeted phases rather than left behind.

The observed between-group difference in AI knowledge self-efficacy (r=.574) is larger than improvements typically reported in prior digital health education studies, which have largely relied on uncontrolled before-after designs and reported heterogeneous, generally smaller gains [38]. The larger effects likely reflect the intensive, hands-on design (71% of on-site time practical; 84% experiential including the capstone), consistent with experiential learning evidence [19,23]. Recent ChatGPT education studies for health care professionals report similar directions but with shorter interventions and smaller effects [11,12]; our previous health informatics analyst education program at the same institution [18] provides a direct comparator, and the present study extends that evidence to advanced generative AI technologies.

The absence of significant attitudinal change (S2; P=.76) aligns with ceiling effects in health care AI attitude studies [13,39], where voluntary enrollees already hold favorable views (4.29/5). Notably, however, the qualitative data captured attitudinal shifts that the Likert scale missed: 21.9% of respondents spontaneously described reduced fear or increased confidence as their most significant growth—suggesting that self-efficacy gains may be a more sensitive indicator of training impact than attitudinal scales in already AI-positive populations. Future studies should recruit participants with more heterogeneous baseline attitudes [40] and consider AI adoption readiness [21] as an alternative construct.

This study has several important limitations that should be considered when interpreting the findings.

This study suggests that transforming a multidisciplinary hospital workforce into AI-capable professionals is achievable through intensive, hands-on training centered on agent-level technologies. The curriculum’s investment in MCP, the technology anticipated to underpin future health care AI infrastructure, was associated with the largest self-efficacy gains, and non-IT professionals showed the largest between-group differences, challenging the assumption that advanced AI training should be restricted to technical staff. The team-based capstone model proved particularly valuable, with prototypes that progressed toward institutional adoption, providing early evidence that training programs can serve as a pathway for institutional AI innovation. However, the knowledge-practice gap identified across multiple measures underscores that training alone is insufficient; posttraining support structures, mentoring, communities of practice, and supervised implementation are essential to translate self-efficacy gains into sustained workplace practice. Future research should use longitudinal designs with matched cohorts to assess the durability of training effects and to determine whether capstone-to-deployment pathways can be replicated across institutions.