Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study

Introduction

Psychiatric consultation requires advanced interviewing skills, diagnostic reasoning, and the ability to integrate complex psychosocial information into treatment planning [1,2]. Because the clinical interview and the mental status examination (MSE) remain the cornerstone of psychiatric diagnosis, residents must develop these competencies through repeated, feedback-rich practice [3,4]. Yet, training environments often limit opportunities for sustained supervision and deliberate rehearsal [5,6], leaving residents to encounter challenging cases without sufficient low-stakes practice [7]. Standardized patients offer structured opportunities for experiential learning, but their use is resource-intensive, difficult to scale, and typically limited to major assessments or dedicated simulation events [6,8,9]. As a result, opportunities for iterative practice and formative feedback remain limited, despite their importance for competency development in medical education [10].

In the context of cognitive learning taxonomies such as Bloom's Taxonomy, formative feedback plays a central role in helping learners move beyond basic recall toward higher-order competencies, including analysis, synthesis, and clinical reasoning [11]. For psychiatric interviewing specifically, repeated low-stakes practice enables learners to progress from identifying key symptoms (remember and understand) to formulating differential diagnoses (analyze) and developing treatment plans (evaluate and create). Simulation-based formative feedback therefore supports the deeper cognitive processes required for psychiatric consultation competence by providing structured, iterative opportunities to test and refine reasoning. This mismatch between training needs and available learning opportunities contributes to persistent gaps in consultation skill development.

Digital simulation has emerged as a promising avenue to address this challenge, with AI increasingly enabling applications in diagnosis, risk assessment, and educational evaluation [12,13]. Within this context, virtual patient systems represent a key application of AI-driven simulation in medical education. Early virtual patient systems demonstrated that computer-mediated psychiatric dialogues were feasible and effectively supported specific tasks (eg, obtaining informed consent for antipsychotics) to improve trainee confidence and knowledge in controlled studies [14], as well as enhance diagnostic reasoning [15] and skill acquisition [16]. However, their rule-based logic and fixed response structures constrained realism and adaptability. Case-specific virtual patients, such as Virtual Justina (University of Southern California Institute for Creative Technologies), a virtual adolescent with posttraumatic stress disorder [17], provided helpful structured practice but required extensive manual scripting and lacked the flexibility necessary for diverse, dynamic encounters. As these systems evolve, it is essential that performance measurement for agentic AI systems be aligned with the agent’s expected role. In educational contexts, this means evaluating the system as a training tool rather than as a therapeutic provider. For example, when an AI system is used to help learners practice cognitive behavioral therapy interviewing, its evaluation should focus on whether its responses appropriately model cognitive behavioral therapy–consistent communication and reasoning, rather than implying that the system functions as an actual clinical cognitive behavioral therapy provider [18].

Recent advances in large language models (LLMs) have transformed the capabilities of digital simulation. Transformer-based LLMs [19], trained on terabytes of data, demonstrate improved realism and adaptability in simulating a wide range of patient scenarios relevant to consultation practices [20] and education [21,22]. LLM-generated simulations demonstrate clinically relevant psychiatric exchanges [23-25] and generate automated feedback aligned with expert judgment [26], offering a cost-efficient and scalable alternative to real patients for clinical training. Newer models outperform their predecessors by a significant margin [27]. Tools such as MedSimAI (UCSF [University of California, San Francisco] School of Medicine, Weill Cornell Medicine, Yale School of Medicine, and Ohio State University), which uses GPT-4o to simulate patient encounters and provide immediate formative feedback [28], exemplify the rapid evolution of AI-enabled clinical education. However, current platforms are not designed specifically for psychiatric consultation and often lack validated psychiatric symptom realism, emotional and behavioral nuance, structured MSE-level detail, or alignment with specialty-specific competency frameworks.

In Canada, psychiatry residents’ consultation competence is formally assessed using the structured assessment of a clinical encounter report (STACER) [29], which evaluates performance on the MSE, the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition])-aligned diagnostic formulation [30], and treatment planning. STACER assessments are infrequent and predominantly summative, limiting opportunities for repeated practice. Kolb's experiential learning cycle [31] underscores that skill development requires iterative cycles of concrete experience, reflective observation, conceptual integration, and active experimentation, conditions difficult to satisfy when assessment opportunities occur only episodically.

To address these limitations and build on our previous works in generative AI and prompt-based reasoning [32,33], we developed the STACER Agentic System (SAS), an LLM-powered simulation and assessment platform designed to support psychiatric consultation training. SAS integrates 2 coordinated AI agents: a patient agent, which simulates realistic multiturn psychiatric interviews, and a rater agent, which produces structured formative feedback aligned with STACER criteria. By unifying simulation and assessment within a single environment, SAS aims to create a scalable pathway for repetitive practice, self-directed learning, and progressive refinement of consultation skills.

The objective of this technical feasibility study was to evaluate SAS’s ability to (1) simulate clinically realistic psychiatric interactions and (2) generate structured, STACER-aligned formative feedback to support the development of consultation competence.

Methods

SAS Overview

The SAS is an LLM-powered educational platform designed to provide psychiatry residents with structured, simulation-based practice in consultation interviewing. Grounded in principles of simulation-based education [34], the system operationalizes the Kolb experiential learning cycle [31] by coordinating 2 complementary agents: a patient agent, which provides the concrete experience of a simulated psychiatric interview, and a rater agent, which supports reflective observation and abstract conceptualization through structured formative assessment, narrative feedback, and guided cross-examination [10]. Figure 1 illustrates how these agents collectively enact the 4 stages of the Kolb cycle to support competency-based medical education (CBME) [35,36].

‎

The patient agent simulates psychiatric presentations derived from detailed patient profiles, while the rater agent acts as a formative, stabilizing comparator and produces structured formative feedback aligned with the Core of Discipline-STACER form [29]. A verbal chat interface allows residents to ask clarifying questions and engage in reflective dialogue. The system was implemented using LangGraph (LangChain, Inc) [37], an open-source framework that allows for the creation and coordination of fully customizable multiagent systems using graph-based workflows. Patient dialogue is generated using a narrative sense-making process, which draws on research showing that language models naturally use storytelling to make incomplete information feel coherent and realistic [38]. This helps the patient agent produce responses that resemble how real patients naturally describe their experiences and is delivered through integrated audio and speech-to-text tools (supported through ElevenLabs’ AI Audio Application Programming Interface) [39]. For details, please refer to the GitHub repository for the workflow code and the prompt files in Multimedia Appendix 1. Detailed descriptions of the patient agent and the rater agent can be found in Multimedia Appendix 2.

Study Design

This study used a two-phase prospective single-arm cohort study design consisting of (1) an automated technical validation and (2) feasibility testing with human participants (Figure 2). To ensure transparent and comprehensive reporting of this LLM-based system, we adhered to the Development, Evaluation, and Assessment of Large Language Models guidelines [40]; the completed Development, Evaluation, and Assessment of Large Language Models checklist is provided in Checklist 1.

‎

Participants and Setting

Participants included 2 groups: (1) members of the clinical research team and (2) psychiatry residents. The clinical research team consisted of 5 individuals, including 1 student and 4 postdoctoral fellows in psychiatry or psychology. These individuals simulated the role of psychiatry residents during feasibility testing. The second group consisted of 9 psychiatry residents enrolled in the University of Alberta Psychiatry Residency program.

Testing was conducted in 2 settings. Clinical research team members were provided with instructions to complete the simulation sessions in person using a laboratory computer at St. Michael’s Hospital. Psychiatry residents were provided login credentials without further instructions and completed the sessions remotely using the online platform.

No formal sample size calculation was performed, as the study aimed to evaluate feasibility and preliminary system performance rather than to test a hypothesis or detect between-group differences. Participants were recruited using convenience sampling.

Study Procedure

For technical validation, a psychiatrist agent external to SAS was developed to conduct automated multiturn interviews with the patient agent. The psychiatrist agent, grounded in Core of Discipline-STACER expectations and informed by the DSM-5, ensured standardized elicitation of clinically relevant information across all synthetic profiles. Automated testing on the synthetic dataset was performed prior to human user testing.

The feasibility testing unfolded across 4 sequential phases as illustrated in Figure 3: (1) the participant selects a patient profile to interview; (2) a simulated consultation occurs with the psychiatric patient agent (powered by GPT-4.1-mini), mirroring the 55-minute STACER interview; (3) the participant delivers a 20-minute case presentation to the rater agent; and (4) the rater agent (powered by GPT-4.1) combines the case study file, interview dialogue, presentation, and DSM-5 diagnostic criteria to offer feedback presented in a Core of Discipline-STACER assessment form. A chat interface was provided for participants to converse with the rater agent about the feedback, explore opportunities to extract new knowledge, and engage in active reflection. Simulation results from the 14 participants were manually analyzed and compared with the actual behavior and symptoms of the cases.

‎

Participants rated the patient agent on realism, psychiatric nuance, and rapport building using a Likert-type scale (1=very poor to 5=excellent) and through a free-text option. Two practicing psychiatrists independently reviewed each simulation and rater agent output, providing parallel STACER assessments for comparison.

Synthetic Benchmark Dataset

Existing datasets are not suited for multiturn psychiatric consultations [41-43], so we developed a comprehensive synthetic benchmark of 227 DSM-5-Text Revision–based [44] major depressive disorder cases to systematically evaluate patient agent performance (Table 1). A detailed synthetic case study is provided in Multimedia Appendix 3.

Real-Patient Case Study Dataset

A separate set of 100 manually curated case studies adapted from DSM-5-Text Revision Clinical Cases was created for testing. Overall, 5 noncomorbid major depressive disorder cases (5/100, 5%) were selected for feasibility evaluation. Each case included a narrative summary, behavioral description, depressive symptom list, patient profile, and confirmed diagnosis (Multimedia Appendix 3).

Performance Measure

Performance was evaluated separately for each agent using role-specific metrics, including automated testing for the patient agent using DeepEval (Confident AI); accuracy, clarity, professional tone, medical faithfulness, bias, toxicity, turn relevance, and role adherence), and participant sessions were used in reliability analyses for the rater agent. User feedback questionnaires were used to provide insights into the limitations of the learning experience. The rater agent’s scoring was compared with the psychiatrists’ using the intraclass correlation coefficient (ICC), which was computed using the psych package in R (version 4.5.1; The R Foundation for Statistical Computing). Standard interpretive thresholds were applied: ICC <0.50 for poor, 0.50‐0.75 for moderate, 0.75‐0.90 for good, and >0.90 for excellent reliability, respectively [45]. Fidelity of the patient agent’s behavior and symptoms was then analyzed. Full details on performance measures are provided in Multimedia Appendix 4.

Ethical Considerations

This work was conducted as a preliminary technical feasibility and operational testing study of the SAS prototype prior to future controlled educational evaluation studies. Psychiatry residents and members of the clinical research team voluntarily participated as pilot testers. Participants were informed about the purpose and procedures of the study through email communication and voluntarily chose whether to participate. Given the preliminary operational testing nature of this work, the study focused on prototype functionality and usability assessment rather than formal evaluation of educational or clinical outcomes. Formal research ethics board review was not sought because this work was conducted as preliminary operational and technical feasibility testing of a prototype system prior to future controlled educational evaluation studies.

No personal health information or sensitive participant information was collected. Participant initials were used only for study coordination purposes and were removed prior to analysis. All transcripts, ratings, and feedback were deidentified before reporting to protect participant privacy and confidentiality. No financial compensation or incentives were provided for participation. No identifiable participant information or images are included in the manuscript, figures, or supplementary materials, and no participant assessments or feedback can be linked to individual residents. Future studies designed to formally evaluate learner outcomes, educational effectiveness, or clinical performance will undergo formal research ethics review prior to implementation.

Results

Patient Agent Performance

Using the synthesized patient profile and the psychiatrist agent, the Patient Agent’s behavior was validated with the eight DeepEval evaluation metrics. Table 2 and Figure 4 summarize the 8-performance metrics, including mean, SD, 95% CI, token cost, and success rate.

‎

User Experience With the STACER Agentic System

Detailed ratings from participants’ first-session responses are presented in Table 3. Participants indicated that the patient agent effectively expressed depressive symptomatology. However, rapport building was more limited (3/5). Ratings of the patient agent’s ability to convey emotional distress varied between the 2 groups (4.6/5 vs 2.78/5). Using voice and text-based nonverbal cues alone, both the patient and rater agents demonstrated moderate levels of emotional expressiveness.

Figure 5 illustrates the differences between the 2 participant groups. One of the psychiatry residents assigned 3 scores of “0” and one score of “1,” citing limitations related to typing. The other scores of “1” were given by 1 resident who noted the absence of visual cues and by another who highlighted typing constraints alone.

‎

Free-text feedback indicated that participants found the SAS both informative and educational, not only because of the feedback provided but also due to the ability to ask questions to the rater agent, which enhanced their learning experience and supported their planning. However, the absence of visual cues limited the expression of emotions and hindered rapport building, as facial expressions, body language, and overall appearance are important components of communication.

Performance Evaluation of the Rater Agent

We measured the consistency of the rater agent at both the item-level scores and the section subtotals. The one-way ICC(1) model, where each target is rated by a different judge and judges are randomly selected, was reported because it yields the most conservative (lowest) correlation estimates. Table 4 summarizes the intrarater reliability of the rater agent across all runs for each recorded session. One of the psychiatry residents (resident 7) did not complete the STACER process, and the session was discarded in the analysis. At the item level, assessments for both groups demonstrated ICCs in the moderate-to-high range (4/5 and 8/8). Reliability improved when item scores were aggregated into section subtotals, with 2 of 5 and 7 of 8 in excellent agreement. Missing data adversely affected reliability estimates, as observed for participants 4 and 5. Resident 7 did not complete the interview and presentation and was excluded from the analysis. The missing data are summarized in Table 4. All missingness belonged to the same items: interrupts politely when required, redirects when required, facilitates organization of disorganized patients, engages patients in a culturally safe manner, attends to timing, and provides a pertinent closing statement. Participant 5 had 3 additional items: nonverbal behavior encourages patients to tell their story, listens attentively, and note taking does not distract from the interview. As these items were not evident and some were impossible to capture from the transcripts, they were replaced with 0.

Table 5 summarizes the interrater reliability between the 2 psychiatrists (P1 and P2) and between each psychiatrist and the rater agent. At the item level, all comparisons demonstrated poor agreement, with ICCs below 0.50. The 2 groups exhibited very different results at the section subtotal level. For the clinical team, 1 psychiatrist showed moderate agreement with the rater agent, while the other achieved good agreement. The agreements for the psychiatry residents were much higher, from 0.89 to 0.93, indicating good-to-excellent agreements.

Discrepancies between raters were also evident in overall scoring patterns, especially for the clinical research team. One psychiatrist rated scores 51% higher than the rater agent, whereas the other rated scores 33% lower. The rater agent’s scores consistently fell between those of the 2 psychiatrists. On the other hand, both human raters scored 14% higher than the rater agent for the psychiatry residents.

As shown in Tables 4 and 5, aggregating item scores into section subtotals generally improved agreement. Section-level aggregation provides a broader impression of skill categories rather than focusing on individual items, thereby reducing instability arising from ambiguity or differing interpretations of item definitions.

Linking Back the Patient Agent’s Behavior and Symptom Expressions

As the participants were given the choice to select the case studies, based only on the case name, the number of behavior traits and symptoms varies (summarized in Table 6). A total of 4 case studies (a-d) were selected by more than 1 tester. Case studies (b) and (c) achieved 100% in behavioral expressions, while case studies (a) and (d) achieved 87.50% (7/8) and 92% (11/12), respectively. Case studies (a), (b), and (d) achieved 100% symptom expression, while case study (c) achieved 88.89% (16/18). Resident 3 selected a Schizophrenia patient profile where behaviors could not be mimicked in the consultation. Resident 9 selected a more complicated comorbid case study. The overall behavioral fidelity across both groups was over 91.07% (51/56), and symptom fidelity reached 95.45% (105/110), consistent with the high medical faithfulness (0.993) from the automated test.

Discussion

This study evaluated the technical feasibility of an agentic AI platform designed to simulate psychiatric consultations and deliver structured formative feedback aligned with the STACER framework. The findings demonstrate that LLM-driven agents can emulate clinically coherent psychiatric dialogue, support structured feedback, and create an accessible environment for deliberate practice. At the same time, the results highlight specific constraints, including rapport formation and multimodal fidelity, that shape how agentic systems should be incorporated into psychiatric education. These findings suggest that while AI-supported simulation may effectively extend opportunities for experiential learning, there are areas requiring refinement and testing before widespread curricular integration.

Fidelity and Educational Value of the Patient Agent

The patient agent exhibited high psychiatric fidelity, maintaining diagnostic, behavioral, and linguistic coherence across multiturn interviews. This aligns with evidence that LLM-based virtual patients can support clinically realistic interactions in domains such as communication and history taking [28]. For instance, MedSimAI demonstrated that GPT-4o can sustain coherent exchanges, allowing learners to rehearse clinical questioning strategies [28]. Similarly, Virtual Justina showed that structured, DSM (Diagnostic and Statistical Manual of Mental Disorders)-based simulations can facilitate practice with sensitive symptom domains, including trauma-related questioning and rapport building [17]. The SAS builds on these foundations by coupling psychiatric realism with adaptive, multiturn dialogue and structured competency mapping aligned to STACER domains, while offering more sustained internal consistency across extended interviews than earlier systems, which were constrained by fixed dialogue trees [17]. The near-perfect turn relevance and role adherence further support its role as a reliable platform for the “concrete experience” stage of experiential learning.

A key finding from both quantitative and qualitative data is that rapport formation was consistently limited (2.78‐3.00 out-of 5 across groups). Rather than a minor usability issue, this represents a defining boundary of current agentic AI systems: while effective for supporting the cognitive components of psychiatric consultation (eg, history taking and diagnostic reasoning), they are less suited for affective and relational skills [17,28,46]. This limitation is further clarified when mapped onto the STACER framework [29]. The patient agent demonstrated strong alignment with domains related to information gathering, diagnostic formulation, and MSE structure; however, lower rapport scores indicate reduced capacity to support domains involving affect, engagement, empathy, and relational skills. These competencies rely on subtle interpersonal and embodied cues, particularly nonverbal communication, that are not fully captured in text- and voice-based interfaces.

These findings are consistent with prior literature indicating that embodied conversational agents vary widely in their use of human communication modalities; although some systems incorporate verbal and nonverbal behaviors, many rely on simpler, less expressive designs [46]. Because advanced multimodal interaction is technically demanding and still seldom evaluated in clinical settings, the extent to which embodied conversational agents can support human-like therapeutic connection remains uncertain. Complementing this, Tay et al [47] found that immersive and augmented reality environments enhance empathy and reduce stigma by enabling learners to experience affective and perceptual dimensions of mental illness. Incorporating audio-prosodic variation, avatar-based facial animation, or sensor-based detection of the learner’s nonverbal cues may partially address this limitation by creating a more ecologically valid simulation of the mental status examination and the relational aspects of psychiatric care [48]. Similarly, any multimodal extension would also need to capture and interpret the resident’s own nonverbal communication, as authentic rapport relies on bidirectional cues [49].

Performance and Bias in the Rater Agent

The rater agent produced detailed, criterion-referenced feedback and structured STACER scoring, reflecting the growing literature demonstrating that LLM-based evaluators can generate coherent assessments [26,28]. Prior work by Holderried et al [26] showed that GPT-4 achieved “almost perfect” agreement with human raters in history-taking evaluations, while Hicke et al [28] reported that LLM feedback enhanced learner reflection and engagement in simulated patient interactions. In this study, all participants perceived the rater agent’s feedback as specific, actionable, and aligned with expectations for psychiatric assessment. These findings, echoed by Cook et al [23], who observed that LLM-powered virtual patients generated personalized, high-quality performance feedback rated comparably to human evaluators, substantiate the potential of AI-assisted formative assessment in clinical education.

With the use of a more structured and regimented prompt, the rater agent demonstrated improved rating consistency, particularly at the section subtotal level, where agreement ranged from moderate to excellent across both participant groups. Notably, agreement between the rater agent and human raters was higher in the psychiatry resident group than in the clinical research team, suggesting that the system may better approximate expected scoring patterns within its intended learner population. For the clinical research team, the rater agent’s scores consistently fell between those of a more generous and a stricter human rater, indicating a stabilizing, middle-ground evaluation rather than systematic bias. In contrast, for psychiatry residents, both human raters consistently assigned higher scores than the rater agent. This shift, from disagreement in overall scoring for the clinical research team to consistently higher ratings by both psychiatrists for residents, may reflect a favorable bias toward psychiatry residents, potentially influenced by expectations of trainee competence [50]. Blinding raters to participant groups would have helped mitigate this possibility and represents an important consideration for future study design.

Contrary to concerns raised in MedSimAI [28], no evidence of a “supportive” bias was observed. Through iterative prompt engineering, we mitigated several known limitations of LLM-based evaluators, including (1) a predisposition toward overly supportive or affirming judgments [51], (2) overreliance on linguistic polish as a proxy for empathic or clinical competence [52], and (3) the absence of external evaluative pressure that typically constrains human raters [50]. In psychiatric education, such unchecked leniency, while potentially reducing learner anxiety, risks misrepresenting clinical competence and may ultimately diminish the formative value of assessment. To strengthen reliability, future development should incorporate expert-in-the-loop calibration, reinforcement learning informed by clinician judgments, and iterative psychometric validation. Structured prompting strategies, stability constraints, and inconsistency penalties may further reduce variability. Since STACER is used nationally to evaluate readiness for independent practice, any AI-generated scoring must be tightly aligned with established competency standards. Even when used purely for formative purposes, calibration is critical to ensure that learners receive accurate guidance that meaningfully informs their development.

Alignment With Competency-Based and Experiential Learning

A key contribution of SAS is the way it operationalizes competency-based medical education [35,36] and the Kolb experiential learning cycle [31] within a unified, self-directed, digital ecosystem. Unlike traditional STACER assessments [29], which are episodic and summative, the SAS supports repeated low-stakes practice with immediate feedback, enabling residents to iteratively refine interviewing, diagnostic reasoning, and communication skills. The patient agent provides concrete experience as learners engage in full psychiatric interviews. The rater agent delivers reflective observation and abstract conceptualization through structured narrative feedback, cross-examination questions, and STACER-aligned performance indicators. Reengagement with the patient agent allows for active experimentation, completing the experiential cycle [31]. This sequence mirrors established mechanisms of learning in simulation-based education, where structured debriefing is consistently identified as the primary driver of skill acquisition [53,54]. Cheng et al [53] demonstrated that structured debriefing is an influential instructional component, facilitating reflection, performance analysis, and knowledge transfer to clinical contexts. For example, the Promoting Excellence and Reflective Learning in Simulation framework integrates self-assessment, guided discussion, and directive feedback [54]. The rater agent’s structured questioning and targeted feedback map onto these principles, helping learners identify gaps and refine the interpretive steps that underlie psychiatric consultation competence.

By embedding simulation and assessment within one system, SAS presents a distinct pedagogical affordance: the ability to support adaptive, longitudinal skill development outside of resource-intensive, faculty-led simulation centers. This positions the technology as a complement, not a replacement, to supervised clinical encounters and traditional Objective Structured Clinical Examinations–style assessments [55]. However, these findings should be interpreted cautiously, as the small sample size and open-label feasibility design primarily support conclusions regarding technical feasibility rather than educational effectiveness, thereby limiting generalizability to broader curricular integration.

Limitations and Future Directions for Educational Implementation

Several limitations must be acknowledged when interpreting these findings. First, phase 1 relied on automated testing using the LLM-as-a-judge paradigm [56,57], enabling high-volume evaluation but limiting the ecological validity of results. Automated benchmarking is useful for characterizing agentic behavior but cannot substitute for human educational outcomes. Second, although psychiatry residents were included, the sample size was small, and variability in interface use (text vs voice) introduced heterogeneity in the user experience, limiting generalizability and precluding conclusions about educational efficacy. Future studies should use randomized controlled designs with larger samples focused on psychiatry residents to evaluate competence development, diagnostic accuracy, and changes in consultation confidence. Third, and most notably, the limited capacity for rapport formation and nonverbal communication directly impacts multiple STACER domains, including affect, engagement, and relational skills. These findings reinforce that current text- and voice-based agentic systems cannot fully replicate the interpersonal complexity of psychiatric encounters. Future development should prioritize multimodal enhancements to better capture both patient and learner nonverbal behaviors and improve ecological validity.

Fourth, the current evaluation of SAS focused mainly on major depressive disorder. A broader evaluation of DSM-5-Text Revision diagnostic categories is needed for the system to serve as a comprehensive training tool. Finally, refinement of the rater agent, including calibration to professional standards and psychometric validation, is necessary for improving reliability, fairness, and consistency across learners. Integrating expert-driven feedback loops and controlled prompting strategies may reduce variability in scoring behavior and align the rater agent more closely with human evaluators. To strengthen its educational impact, future iterations of the SAS should also include learning-progress tracking and adaptive difficulty scaling to personalize training trajectories and support longitudinal skill development. Collectively, these limitations inform a clear roadmap for future development: expanding diagnostic breadth, enhancing multimodal realism, increasing participant diversity, and strengthening assessment reliability.

Conclusion

This technical feasibility prospective cohort study demonstrates that agentic AI systems can simulate clinically coherent psychiatric encounters and generate structured, STACER-aligned formative feedback within a single educational platform. Unlike earlier virtual patient systems that depended on scripted interactions or lacked psychiatry-specific competency assessment, the SAS combines adaptive multiturn psychiatric dialogue with structured evaluation aligned to competency-based medical education frameworks. The findings suggest that the SAS may provide a scalable and accessible approach for supporting deliberate practice, particularly for cognitive aspects of consultation such as history taking and diagnostic reasoning. At the same time, the study highlights an important limitation of current agentic AI systems: reduced capacity to support affective and relational skills, including rapport building, which rely on multimodal and interpersonal cues. By expanding access to repeated low-risk practice and structured feedback, the SAS has the potential to complement traditional supervision and simulation-based training, particularly in resource-constrained educational settings. Although preliminary findings from psychiatry residents are encouraging, the small sample and feasibility design preclude conclusions about educational effectiveness or curricular integration. Future studies should include larger controlled evaluations, multimodal enhancements, and further calibration of the rater agent to strengthen reliability and educational impact.