AI for Clinical Competency Assessment: Scoping Review of Methods and Applications

Introduction

Universal health coverage (UHC) cannot be realized without adequate human resources for health—a recognition that has shaped the UHC agenda since its inception and integration into the Sustainable Development Goals [1]. Achieving UHC depends not only on expanding service coverage but also on ensuring that care is delivered safely, effectively, and consistently. Despite decades of investment, with an average of more than 70% of health system expenditures spent on human resources, the global health workforce continues to face persistent shortages and wide heterogeneity in skills and practice quality, especially in resource-constrained settings [2-4]. These workforce quality gaps have direct and severe consequences, contributing to preventable deaths and compromised patient trust in health systems. Globally, more than 3 million deaths occur annually due to unsafe care, with estimates suggesting that in low- and middle-income countries, as many as 4 in 100 people die from unsafe care [5]. Tackling health workforce challenges, as well as improving the efficiency of health systems worldwide, requires valid, timely, and scalable measures of clinical competencies.

Clinical competency assessments are commonly used in 2 main contexts: medical education, where they support learning and progression across undergraduate, graduate, and continuing medical education; and health system monitoring, where they help track the skills of practicing providers. Written knowledge tests, such as multiple-choice questions, offer one scalable and standardized approach to assessment. However, because they are inherently closed-ended, they often fail to capture the complexity and interpersonal dimensions of real clinical encounters [6]. To address this, performance-based assessments exist to offer a more authentic approach that evaluates skills more comprehensively, including history-taking, counseling, clinical reasoning, and professionalism. The “gold standard” performance-based assessment is the Objective Structured Clinical Examination (OSCE), in which learners interact with actors who role-play as standardized patients (SPs) and are scored using checklists or global rating scales by an observer on the side [7]. Yet, OSCE programs are widely recognized as resource-intensive, requiring SP recruitment and training, examiner time, facilities, and coordination, all creating barriers to frequent deployment, especially at scale or in low- and middle-income countries [8,9]. Large-scale health workforce assessments, such as the World Bank’s decade-long Service Delivery Indicators surveys, further illustrate both the value and difficulty of generating representative measures of provider readiness using clinical vignettes [9]. These efforts can cost millions of dollars and require complex field operations, which are especially difficult in the reduced funding environment for global health when governments and international organizations face more limited capacity for financing.

Besides their resource intensity, another major limitation of OSCEs is that the standardization of cases does not fully eliminate subjectivity in evaluation. Even when clinical scenarios, SP scripts, checklists, and rating forms are designed to create comparable testing conditions, learner scores may still be shaped by evaluator-related bias and other sources of variation. Recent literature has emphasized that fairness in medical education assessment requires greater attention to the sociocultural processes through which judgments are produced [10]. SP clinical scenarios may encode gendered or stereotyped assumptions, while the observation and the checklist completion components may be affected by the social identities of both learners and assessors [11,12]. Evaluator effects are also well documented, including halo effects, contrast effects, and systematic differences in stringency or leniency across examiners, with examiner stringency alone accounting for up to 29% of score variation in some OSCE settings [13,14]. Thus, the need for scalable assessment is inseparable from the need for fairer and more consistent assessment systems.

Recent advances in AI create an opportunity to develop conversational, case-based simulations that mimic human SP encounters, potentially improving scalability and reducing the risk of evaluator bias. AI, including large language models (LLMs), has the potential to engage learners or practitioners in naturalistic clinical encounters at scale, responding dynamically to history questions, laboratory test requests, and more based on prespecified case scenarios. Several reviews have examined adjacent areas, but none fully address the assessment-focused scope of this review. Earlier reviews of virtual patients in health professions education established that digital patient simulations can support knowledge acquisition and skills development, but these reviews largely evaluated virtual patients as educational interventions rather than as systems for competency assessment [15,16]. More recent reviews have focused on LLM-based virtual patients, describing their growing use for medical training [17-20]. Broader reviews of AI and generative AI in medical education have also summarized applications across curriculum design, feedback, and educational support while highlighting ongoing concerns about validity, implementation, and evidence quality [21-23]. However, these reviews do not distinguish whether the AI system functions as the patient, the assessor, the feedback generator, or all three; nor do they systematically characterize what clinical competencies are assessed, how learner performance is scored, whether scoring is human-generated or AI-generated, and what evidence supports reliability, validity, or implementation readiness. This gap is important because AI–standardized simulated patStandardized Simulated Patients (SSPs) are not only educational simulations but also may become infrastructures for more scalable, routine, and fairer competency assessment. A dedicated scoping review is therefore needed to map the designs, assessed domains, and scoring and feedback approaches of AI-SSP systems for clinical competency assessment.

Given these gaps, this scoping review aimed to map the existing literature on AI-enabled SSP systems for clinical competency assessment. Specifically, we sought to characterize (1) the study characteristics, populations, and clinical contexts in which AI-SSPs have been evaluated; (2) the system architectures, model types, platforms, and interaction designs used to simulate SP encounters; (3) the clinical competencies assessed; (4) the quantitative scoring and qualitative feedback approaches used, including whether scores were generated by humans, AI, or hybrid methods; and (5) the maturity of available evidence, including feasibility, usability, scoring accuracy, learning outcomes, and implementation evidence. By synthesizing these dimensions, this review identifies how AI-SSPs are currently being used for competency assessment and what further validation, design, and implementation research are needed.

Methods

Protocol and Registration

This scoping review was conducted according to the Joanna Briggs Institute Manual for Evidence with special attention to Chapter 10, Scoping Reviews [24,25]. The reporting of this review is aligned with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) checklist [26] and the PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Literature Search Extension) checklist (Checklist 1) [27]. Our protocol was registered prospectively with the Open Science Framework on June 10, 2025 [28].

Eligibility Criteria

Studies were eligible for inclusion if they described an AI-based SSP system with a clinical competency assessment component. For this review, “assessment” was defined as the production of a quantitative score that could support comparison either within a learner over time or across learners. The score did not need to be AI-generated; studies were eligible if the simulation itself was AI-based and the resulting performance was scored by any method, including by AI or by human raters. Eligible systems also had to be implemented or evaluated as a functioning tool, prototype, or product. Purely conceptual frameworks, design proposals, and systematic reviews were excluded.

Inclusion criteria were as follows:

• Primary research source: Reported original empirical data.
• AI-based SSP: The interaction between the user and the SSP must be AI-based.
• Health professions context: Focused on learners or practitioners in medicine, nursing, pharmacy, or other health professions.
• Clinical competency focus: Involves clinical skills such as history-taking, clinical reasoning, diagnosis, counseling, and so on.
• Assessment component: Produced a quantitative competency score, either as part of the AI-SSP platform itself or assigned post hoc by human raters based on the AI-SSP encounter. Platforms used for formative assessments (“assessment for learning”) and summative assessments (“assessment of learning”) are both eligible.
• Any scoring source: The score could be generated by AI or human raters reviewing AI-simulated encounters.
• Functioning system: Described a functioning tool, prototype, platform, or product.

Exclusion criteria were as follows:

• Nonprimary source: Systematic review, editorial, commentary, opinion piece, or protocol without results.
• No AI-based SSP interaction: The SSP encounter was not AI enabled, for example, video-based or enacted by human SPs.
• No competency score: Unstructured, narrative feedback only, which does not allow for comparability across learners.
• No clinical competency focus: Did not assess clinical or health professional competencies.
• No functioning system: Described only a conceptual model, design proposal, or system architecture without evaluating a functioning tool or prototype.

Information Sources

A comprehensive literature search was conducted by searching MEDLINE (National Library of Medicine, OVID), Embase (Elsevier, embase.com), Cumulative Index to Nursing and Allied Health Literature (CINAHL Complete, EBSCOhost), Education Source (EBSCOhost), and Web of Science Core Collection (Clarivate), covering the period from their inception to an initial search date of June 12, 2025. This search was updated on May 7, 2026, to expand the search vocabulary, with the addition of Inspec (Institution of Engineering and Technology, EBSCOhost), a technology-oriented database to identify articles from the engineering literature, and a second education database, the Education Resources Information Center (ERIC, EBSCOhost), as well as the Cochrane Central Register of Controlled Trials (Cochrane Library, Wiley) to identify unpublished trials. Each database was searched individually on the specified platform.

The databases searched included meeting abstracts and proceedings, dissertations, and reports, in addition to articles published in the scholarly literature. A supplementary manual search was conducted by reviewing reference lists of relevant review articles retrieved by the database searches and by conducting targeted searches in Google Scholar using keywords. No online or print resources were purposefully browsed, and no additional studies or data were sought from authors or experts.

Search

The search strategies were designed and carried out by a health sciences librarian (CM) based on discussion with the research team and predetermined eligibility criteria. No substantial portions of strategies from previously published reviews were used. The search terms encompassed three concepts: (1) patient simulations of clinical case–based vignettes, (2) AI and LLMs, and (3) clinical competency assessment. Controlled subject vocabulary (ie, MeSH, Emtree, and subject headings from CINAHL, Education Source, ERIC, and Inspec) was included when available and appropriate. No language, search filters, or publication date limits were applied. The searches did not undergo peer review. Search reporting was guided by PRISMA-S recommendations. The exact search strategies from each database are provided in Multimedia Appendix 1.

Selection of Sources of Evidence

Records from all database searches were imported into Covidence systematic review software (Veritas Health Innovation Ltd), where duplicates were removed prior to screening. Using Covidence, 2 reviewers (BD and WZ) independently screened titles and abstracts from the search results. Conflicts were resolved through discussion. Full texts were then retrieved and reviewed by the same 2 reviewers independently. Any discrepancies between reviewers were discussed among reviewers and resolved by discussion. A formal critical appraisal of the included studies was not conducted, as the goal was to map the extent of the literature.

Data Charting

A structured data charting template was developed in Covidence to extract information from the included studies. The template captured study characteristics, study population, SSP system features, clinical scenario design, learner interaction modalities, feedback and scoring mechanisms, backend model architecture, and product evaluation methods and results. The charting form was developed iteratively and pilot tested on an initial set of included studies by the 2 independent reviewers (BD and WZ).

Data Items

Extracted variables included general study information (study ID, title, authors, country, and relevant context on why the tool was developed); study population (target learner or practitioner group, health profession, training level, total number of respondents, and any reported demographic or clinical experience information); SSP system characteristics (tool name, frontend interface, online or offline support, and compatible devices); clinical scenario design (number and type of clinical scenarios and how they were developed or validated); scenario display features (whether the SSP was displayed using text, avatar, images, or other modalities); user interaction (respondent mode of input and available clinical actions); quantitative scoring (the type of scores used and the logic by which they were derived); qualitative feedback (the type of qualitative feedback and the logic by which they were generated); backend AI model (response generation model of SSP interaction, any prompting strategy for LLM-based systems, knowledge graphs for rule-based systems, and so on); language localization; and product evaluation (methods and results used to evaluate scoring accuracy and reliability and respondent experience and perceptions). Any unclear or missing information was clearly indicated as “not reported” with potential reasons specified.

Synthesis of Results

Extracted data were first reviewed and synthesized through discussions between the 2 reviewers to reach consensus. Thematic analysis was then conducted using the completed data charting spreadsheet. Each extracted variable was tabulated and compared across studies, with themes developed inductively from observed patterns and summarized using frequency counts and narrative synthesis where appropriate.

Results

Study Selection

A total of 5987 records were first identified through database searches and imported into Covidence. After removing 2036 (34%) duplicates, 3951 (66%) unique records were available for title and abstract screening. Two reviewers independently screened all titles and abstracts against the predetermined eligibility criteria, which focused on AI-based SSP systems with competency assessment components. Conflicts were resolved through discussion. At this stage, 3906 (98%) records were excluded because they did not meet the inclusion criteria. The full texts of 45 (1%) reports were then retrieved and assessed independently by the same 2 reviewers. Of these, 27 (60%) reports were excluded, with the most common reasons for exclusion including not involving an AI-based interaction with SP, no quantitative competency score produced, or not an empirical primary study. Four additional records were then identified through citation searching and Google Scholar, of which 3 (75%) studies were included. A final total of 21 studies were included in the final scoping review. The PRISMA flow diagram (Figure 1) provides further details on the screening process and reasons for exclusion.

‎

Study Characteristics

Table 1 summarizes the characteristics of the 21 studies included in this scoping review. The included studies were published between 2008 and 2026, with most studies published in recent years. Fifteen (71%) of the 21 studies were published in 2024 or later, reflecting the growth of AI-enabled SSP systems following the broader availability of LLMs. Earlier studies primarily evaluated rule-based or knowledge-based virtual patient systems, while all studies published after 2022 examined LLM-based SSPs.

The included studies were conducted across a range of geographic settings, with the largest number from the United States or involving US-based study sites or author affiliations [32,33,40-43,45-47,49,50]. Other countries represented included Germany [36-38], China [39,44], Sweden [30], Israel [31], Australia [34,41,45], Singapore [35], and Taiwan [48]. Overall, the evidence base was concentrated in high-income countries, suggesting that the generalizability of findings across diverse medical educational contexts remains uncertain.

Study designs varied substantially, but most reflected early-stage development, feasibility testing, or validation rather than large-scale implementation research. Most studies conducted single-arm pilot feasibility testing [30,32,33,37,38,40,41,44,45,47,50]; others were framed as proof-of-concept, technical, or computational evaluations [34,39,46], while a limited number of studies used randomized controlled trials, crossover, or quasi-experimental designs [31,36,42,43,48,49]. Only 2 (10%) studies provided evidence beyond initial development or short-term evaluation [31,49]. Overall, the evidence base remained concentrated in pilot, feasibility, proof-of-concept, and validation work, with fewer controlled evaluations and large-scale implementation studies.

Study Populations

Medical students across all academic years were the most frequently cited population, including students at the start of clinical training [31], first-year students in medical interviewing curricula [33], early-semester students without clinical rotation experience [36], second-year students [40], third-year students in history-taking training but with limited SP experience [44], and fourth-year medical students in surgery [30]. Several studies extended beyond medical students to include residents, junior physicians, practicing clinicians, or mixed groups of health professional trainees and practitioners [32,41-43]. Fewer studies examined other health professions learners, including physiotherapy students [35], midwifery students [37,38], nursing students [48], pharmacy students [50], and physician assistant trainees [42,43]. One technical evaluation study did not involve a conventional learner trial and instead evaluated AI-SSP quality and scoring using a combination of nonmedical evaluators and senior physicians. Overall, the included studies primarily centered on medical education rather than clinicians in practice, with more limited but emerging representation from nonmedical health professions.

Sample sizes and units of analysis varied widely across studies. Among studies reporting learner-level participation, sample sizes ranged from very small pilot studies with 3 to 4 participants [41,45] to larger evaluations involving more than 100 learners [30,31,38,49], and in one case, 410 unique learners across 3 institutions [49]. Several studies reported encounter-, transcript-, utterance-, or dialogue-level data rather than only learner-level sample sizes, including 112 transcripts [32], 80 dialogue records [39], 602 student encounter transcripts [47], 1024 completed encounters [49], and more than 2300 conversation logs or students’ utterances [46]. Some studies did not clearly report learner sample size or were development focused without a completed learner evaluation [35,50].

AI-SSP System Architecture and Interaction Design

Systems Overview and Intended Use

As summarized in Table 2, the included studies evaluated a diverse set of AI-enabled SSP systems, ranging from dialogue simulation platforms to immersive virtual reality environments. Several studies involved formally named platforms, such as USC Standard Patient [32], CureFun [39], SOPHIE [42,43], AMTES [44], Turing Interactive Case Learning [45], 2-Sigma [46,47], VR-AI-ECS [48], and MedSimAI [49], while others described the intervention more generally as a “virtual patient,” “simulated patient chatbot,” or “AI-supported online platform” [31,33,35-38,40,41,50]. Almost all studies justified their systems by pointing to the same underlying problem: traditional SPs are resource intensive, costly, logistically difficult, and hard to scale. The systems varied in format and implementation, but most were designed to create repeatable clinical encounters in which learners could practice, receive feedback, or be assessed in a standardized environment. Overall, 20 unique AI-SSP systems were identified across 19 unique study teams, with 1 system evaluated by 2 studies [42,43] and 1 study team evaluating 2 variations of a system [37,38].

Across these 20 unique AI-SSP systems, their primary intended use fell into 3 broad categories. The largest group, 9 (45%) of 20 systems, framed AI-SSPs primarily as deliberate practice or educational tools, emphasizing safe, repeatable, and low-stress environments for learners to practice history-taking, communication, empathy, counseling, or case-based clinical reasoning [33,35-38,40-43,48]. While they do not list assessment as a primary purpose, they were included in this review because they all produce some form of quantitative score, and their functionality demonstrates strong potential for direct adaptation to assessment purposes. Six (30%) of 20 systems were positioned as hybrid training and assessment tools, combining SP interaction with structured feedback and scoring [31,32,34,39,44,49]. The remaining 5 systems focused explicitly on assessment using AI-SSP systems to evaluate user performance. Taken together, the literature suggests that AI-SSPs, even the ones with scoring or assessment components, have most often been introduced as formative learning environments. However, there exists an increasing subset of systems that explicitly emphasize assessment purposes, including automated scoring, structured feedback, and performance analytics.

Backend Architecture and AI Model

Backend architectures varied substantially across studies and showed a clear temporal shift in model type, as summarized in Figure 2. Earlier AI-SSP systems, particularly those published before 2022, relied on rule-based natural language processing, scripted dialogue structures, decision trees, keyword matching, or intent classification approaches. These systems used preauthored cases and expected learner inputs to be matched against predefined system response pathways, making the interaction more predictable but less conversationally fluid. For example, Maicher et al [33] used approximately 2500 ChatScript rules to support the virtual patient dialogue, Yang et al [34] developed a knowledge-based 3D dialogue system using classification rules, and Lee et al [35] used DialogFlow intents to classify learner questions and return scripted patient responses. These earlier architectures emphasized control, standardization, and reproducibility but were constrained by the range of prespecified utterances, intents, and response rules built into the system.

‎

After 2022, all included studies either partially or fully relied on LLMs to power patient dialogue, scoring, or feedback generation. In contrast to rule-based systems, LLMs are AI models trained on large text datasets to interpret and generate natural language, which allows them to support more open-ended, contextually responsive SSP dialogue. Specifically, 2 (10%) studies used the public ChatGPT interface directly, representing a relatively low-barrier implementation model in which learners interacted with an existing chatbot rather than a fully custom AI-SSP platform [36,40]. In a study by Brugge et al, contextual prompts were entered into ChatGPT right before participants arrived at the computer, while Chiu et al instructed students to write their own prompt specifying that ChatGPT should act as a patient before they began the dialogue.

The majority of the newer studies, however, embedded LLMs within more controlled custom platforms or structured prompt pipelines rather than using an open-ended public chatbot interface. These systems typically combined a system prompt with case information, behavioral instructions, and specific constraints to define how the simulated patient should respond. For example, Holderried et al [37,38] used a 2-part prompt with behavioral instructions and illness scripts, Haut/SOPHIE [42,43] used serious illness communication scenarios with target skills such as agenda setting and responding to emotion, and Hicke et al/MedSimAi [49] used instructor-provided templates to build system prompts and separate evaluation prompts with JSON output. Several other studies also separated the simulated patient prompt from the scoring or feedback prompt, indicating that the same LLM was not simply prompted once to “be a patient” but was instead used through multiple task-specific prompt layers [37,38,40,42-44,49].

A smaller subset of studies incorporated additional backend structures to constrain LLM output, improve consistency, and reduce hallucinations. These included retrieval-augmented generation, structured case graphs, templates, or guardrails that limited the information available to the model and specified how the virtual patient should behave. For example, Li et al [39] used a structured case graph so that the model could retrieve relevant patient facts before generating a response, Haut/SOPHIE [42,43] aligned dialogue with target communication frameworks, and Hicke et al [49] used templates specifying patient demographics, history, communication style, emotional state, and vocabulary. These approaches suggest that newer AI-SSP systems are mostly “LLM wrapped” rather than fully open-ended. The model generates naturalistic conversation, but the architecture has constraints in place for what information it can use, how the patient should respond, and what learning or assessment goals the interaction is intended to support.

Clinical Case Scenarios and Their Development

The number of simulated cases ranged from 1 to 25 across studies. Most used a relatively small set of cases rather than a large clinical case bank [30,33,35,38,40]. The clinical content spanned multiple domains, including gastrointestinal and metabolic conditions, cardiopulmonary and respiratory presentations, neurology and neurosurgery, musculoskeletal complaints, oncology and cancer communication, obstetric and nursing empathy scenarios, and pharmacy counseling.

Earlier systems often used cases to assess biomedical clinical reasoning and diagnostics [30-32,34,36,45,47]. Later studies increasingly expanded case content to include communication-oriented and interpersonal competencies, including empathy, breaking bad news, counseling, professionalism, serious illness communication, and patient-centered communication behaviors [40,43,46,48-50]. Across studies, cases typically included patient demographics, chief complaint, symptom history, and expected learner tasks, with some systems also incorporating physical examination findings, investigation results, differential diagnosis, treatment decisions, emotional context, and grading rubrics.

Case development was mostly expert-led or faculty-led rather than AI-generated. Fourteen (67%) studies reported cases that were authored, adapted, or reviewed by clinical experts, faculty, residents, or other domain specialists [30-33,35,37,38,40-45,49]. Human expertise informed scenario writing, checklist or rubric design, natural language processing rules, and other chatbot response logic. Only 1 (5%) study explicitly described cases as being developed by an LLM using contextual prompts [36]. Overall, case development remained largely a human-authored process, with AI more commonly used as an interaction and/or scoring mechanism rather than as a generator of clinical case content.

Platform, Access, and Frontend Interactions

Frontend design ranged from simple text-based chat interfaces to avatar-based immersive virtual environments. Fourteen (67%) of 21 studies used primarily text- or chat-based interfaces, making this the most common display format overall [31,35-41,44-47,49,50]. These systems presented the simulated patient through written dialogue and allowed learners to type questions in a conversational format. Text-based interfaces were especially common among recent LLM-enabled systems, suggesting that newer AI-SSPs prioritized conversational fluidity over visual embodiment.

Seven (33%) studies incorporated avatars, patient images, or other visual representations to add realism to the simulated encounter [30,32-34,42,43,48]. These systems were common before the wide adoption of LLMs and often included patient portraits, animated avatars, or graphical interfaces for selecting clinical actions beyond conversations. One (5%) study in particular used an even more immersive avatar-based virtual reality (VR) environment, in which learners wore vests and a VR headset to interact with the AI-SSP in a 3D simulated clinical setting [48]. Overall, earlier systems tended to be more visually elaborate but less conversationally flexible, whereas newer LLM-enabled systems were visually simpler but allowed more open-ended dialogue.

Most systems were delivered through online or computer-based platforms, although access details were inconsistently reported. Three systems were hosted offline, including earlier virtual patient systems [30,34,48]. Two studies used smartphone or tablet-based devices [33,44]. One study used a VR headset to support embodied interaction in an immersive environment [48]. The included studies suggest that AI-SSP frontend design has evolved from visually rich, platform-specific simulation environments toward more lightweight, chat-based interfaces that can be deployed through web, mobile, or existing LLM platforms.

Clinical Competency Assessment

Clinical Skills Assessed

Table 3 summarizes the clinical competencies each included study assessed. Across the 21 studies, the most commonly assessed domain was history-taking or clinical inquiry, which was assessed in 11 (52%) studies. Diagnostic accuracy was assessed in 8 (38%) studies, and communication, empathy, professionalism, or counseling was also assessed in 8 (38%) studies. Only 4 (19%) studies assessed clinical examination or laboratory test selection, and only 2 (10%) evaluated disease management and treatment planning skills. Temporally, the focus of assessment also shifted from more structured clinical reasoning tasks, such as diagnosis and information gathering, toward more interactional competencies, including communication, empathy, counseling, and professionalism, largely mirroring the broader transition from rule-based virtual patient systems to more fluid, conversational LLM-enabled SSPs.

Quantitative Scoring Approaches and Accuracy Validation

Although all included studies generated some form of learner performance measure, the source of scoring varied across studies. In 3 (14%) systems, the AI-SSP functioned primarily as the simulated patient, while performance was scored by human faculty, raters, or coders after the encounter using transcripts [36,42,43,48]. In other studies, scoring was automated by the AI-SSP system itself [38,39,44-46,49,50]. Several studies used both scoring approaches, and AI-generated scores were compared directly with human-led results [32,40,44,45,48-50].

Checklist-based scoring was the most common strategy. Ten (48%) studies used this approach where they translated expert-defined case content into required history-taking questions and clinical actions, then scored performance as item coverage, percent completion, or weighted checklist scores [30-33,37-39,44,45,49]. Nine (43%) studies used rubric-based or Likert scale scoring, particularly for communication-intensive competencies, such as empathy, professionalism, counseling, clarity, and ethical appropriateness [35,36,40,42,43,46,48-50]. A smaller number of studies used hybrid scoring formats, combining checklist completion with rubric ratings, classifier-generated process metrics, quizzes, diagnosis accuracy scores, or missed mandatory topic counts [34,42,43,49].

Where scores were AI generated, their accuracy and reliability were evaluated unevenly across the literature. Several studies did not formally validate automated scoring against a human benchmark [30,35,37,47], while others relied on internal consistency, classifier performance, or post hoc learning outcomes. Among studies comparing AI-generated scores with human ratings, the results were generally promising but variable. Correlation-based validation showed moderate to strong associations in Bond et al (Spearman ρ=0.80) [32], Li et al (Spearman r=0.66‐0.95 and Pearson r=0.77‐0.94) [39], Gorcilov et al (Pearson r=0.69‐0.88) [41], and Qian et al (Pearson r=0.67‐0.90) [45]. However, Reynolds et al [50] reported wider variability across checklist categories, with correlations ranging from negative values to 0.72.

In addition to correlation tests, agreement and reliability analyses similarly showed mixed but overall positive findings. Holderried et al [38] reported strong GPT-4 versus human agreement, with Cohen κ of 0.832 across 106 conversations and 45 categories [38]. Liu et al [44] reported high human-AI agreement across 3 cases, with intraclass correlation coefficients (ICCs) of 0.92 to 0.98 and item-level consistency of 95.75% to 97.13% [44]. Qian et al [45] reported ICCs of 0.78 to 0.82 after adjusting for consistent differences, while Chen [48] reported an AI versus educator average ICC of 0.81, comparable to human-human educator ICCs of 0.82 to 0.88. Classification-based evaluations were more variable: Haut [42,43] reported bidirectional encoder representations from transformers classifier accuracy of 72% and an F₁-score of 51%, Zheng et al [46] reported accuracy of 0.82 to 0.84 and a weighted F₁-score of 0.82 to 0.84, and Hicke et al [49] reported exact accuracy of 32.5%, off-by-one accuracy of 64.1%, and thresholded accuracy of 87.0%.

Several studies also identified important AI-based scoring limitations. Chiu et al [40] found that ChatGPT scores were substantially higher than faculty scores and sensitive to prompt design, while Qian et al [45] reported that LLMs were consistently around 12 percentage points higher than expert scores before calibration. Reynolds found that AI-human correlations varied widely by rubric category, including negative correlations for certain counseling-related items [50]. Although human-human reliability was sometimes reported as a benchmark [32,36,42,43,48], it was not consistently used to determine whether automated scoring achieved human-level performance [32,36,42,43,48]. Overall, while several systems demonstrated moderate to strong alignment with human ratings, validation approaches remained inconsistent, and evidence for reliably automated scoring was still limited.

Qualitative Feedback Approaches

Qualitative feedback was common across studies but varied substantially in specificity and actionability. Earlier systems often provided checklist- or gap-based feedback, identifying missed questions, omitted clinical actions, or incomplete case elements [31-35,40,49]. Newer LLM-enabled systems increasingly generated narrative feedback that included strengths, weaknesses, transcript-based examples, and personalized improvement suggestions [36,38,40-43,48]. Several studies used hybrid feedback approaches that combined checklist completion and narrative comments [39,42,43,48,49]. Overall, feedback evolved from relatively simple “what was missed” summaries toward more individualized coaching focused on communication, clinical reasoning, and next steps for improvement.

AI-SSP Product Evaluation and Evidence Maturity

Beyond scoring accuracy, studies also evaluated AI-SSP systems using a range of product-level outcomes, including learner experience, AI-SSP response quality, and learning outcomes. Several studies reported high learner satisfaction, perceived usefulness, ease of navigation, or willingness to recommend the system [30,32-34,37,38,40,42-44,49]. Response quality and realism were also commonly evaluated, assessing whether AI-SSP responses were plausible, realistic, or aligned with expected case content [30,32,38,39,41-43]. Evidence for downstream skills improvement was more limited, with only 6 (6/21, 29%) studies assessing this outcome [36,40,42,43,48,49].

As illustrated in Figure 3, the available evidence was concentrated primarily in early-stage product evaluation rather than learning outcomes, controlled comparisons, or sustained implementation. Feasibility, usability, or response quality was the most frequently assessed evidence category, reported in 18 (86%) studies, followed closely by scoring accuracy or reliability in 16 (76%) studies. In contrast, only 8 (38%) studies evaluated learning outcomes or incorporated a controlled comparison, and this evidence was unevenly distributed across competency domains, with the strongest concentration in communication-related skills and history-taking. Evidence was particularly limited for treatment and management and for clinical examination or laboratory testing, suggesting that the literature has not yet established whether AI-SSPs can support assessment across the full clinical workflow. Only 2 (10%) studies reported evidence of implementation or deployment, indicating that most systems remain at the pilot, proof-of-concept, or validation stage. Studies of training-oriented systems contributed mainly to feasibility, usability, and early learning evidence, whereas studies of hybrid training and assessment systems demonstrated the broadest evidence profile, accounting for all identified implementation evidence and a substantial share of scoring validation evidence. Assessment-focused systems were more narrowly concentrated in scoring accuracy and reliability, with comparatively little evidence on learning effects or real-world adoption. Overall, the evidence gap map shows that the field has made meaningful progress in demonstrating technical feasibility and initial scoring accuracy and reliability, but stronger evidence is still needed on educational effectiveness, sustained use, scalability, integration into high-stakes assessment, and performance across the full range of clinical competency domains.

‎

Discussion

This scoping review aimed to map how AI-enabled SSP systems have been used for clinical competency assessment, with attention to system architecture, interaction design, assessed clinical competencies, scoring and feedback approaches, and evidence maturity. Across 21 studies and 20 unique AI-SSP systems, we found that the field is expanding rapidly, with 15 (71%) studies published in 2024 or later and a clear temporal shift from earlier rule-based virtual patient systems to LLM-enabled conversational platforms after 2022. The evidence base remained concentrated in high-income academic settings, focused primarily on medical students, and dominated by pilot, feasibility, and proof-of-concept studies rather than large-scale implementation research. Although all included studies produced some form of competency-related performance measure, AI-SSPs were most often positioned as formative learning tools rather than formal assessment systems, with 9 systems focused primarily on deliberate practice, 6 combining training and assessment, and 5 explicitly assessment-oriented. Across studies, history-taking was the most commonly assessed competency, while laboratory tests, diagnosis, treatment planning, communication, empathy, and professionalism were assessed less consistently. Quantitative scoring and qualitative feedback were common but heterogeneous: checklist-based scoring was most frequent, rubric-based scoring was used mainly for communication-intensive competencies, and validation of automated scoring remained inconsistent. Overall, our findings suggest that AI-SSPs are increasingly capable of supporting scalable and repeatable competency assessment, but the current literature is more mature in demonstrating feasibility and usability than in establishing validity and implementation readiness.

Previous research on virtual patients has shown that digital patient simulations can give learners consistent clinical cases to practice with, helping them build medical knowledge, clinical reasoning skills, and confidence [15,16]. However, these systems have historically functioned primarily as educational simulations rather than scalable competency assessment infrastructures. Similarly, recent reviews of LLM-based virtual patients describe a newer wave of systems focused on conversational realism, history-taking, and medical training, reflecting broader enthusiasm for generative AI in health professions education [17-19]. Our findings build on this literature by showing that AI-SSPs are beginning to move beyond simulation alone toward competency measurement but remain in a transitional stage. The shift from rule-based virtual patient systems to LLM-enabled platforms has improved conversational fluency, responsiveness, and perceived realism, but most systems still rely on human-authored cases, structured prompts, and rubrics, which represent important guardrails against hallucinations. Current AI-SSPs are not fully agentic assessors of clinical competence, but semistructured assessment environments in which generative AI is layered onto human-defined clinical content and scoring logic.

The concentration of assessed competencies also has important implications for how AI-SSPs should be interpreted as assessment tools. Consistent with the broader virtual patient literature, the included systems most often assessed history-taking, diagnosis, and communication-related skills, domains that map naturally onto text-based or conversational interaction [16,17,51]. However, fewer studies assessed physical examination, laboratory or investigation selection, and management planning. This pattern suggests that current AI-SSPs are strongest for evaluating how learners gather information and communicate with patients but less mature for assessing the full clinical workflow from examination to diagnostic interpretation and treatment decision-making. For formative education, this may be sufficient and even valuable. For competency assessment, however, the narrow domain coverage raises concerns about how a learner may perform well in an AI-SSP conversation without demonstrating the broader set of skills required for safe and effective clinical care.

In terms of automated scoring of performances, our review showed that checklist-based scoring was most common, reflecting the need to convert complex encounters into observable items. Rubric-based scoring was used mainly for communication-intensive domains, where performance depended not only on whether an item was mentioned but also on how the learner expressed empathy, professionalism, clarity, or ethical reasoning. Several included studies reported moderate to strong agreement between AI-generated and human or expert scores, suggesting that automated scoring may reduce faculty burden and enable more frequent formative assessment. However, there is also evidence of consistent score inflation, prompt sensitivity, and variable performance across rubric categories in other studies. This pattern mirrors findings from the comparison of AI and human evaluation of the same video-recorded OSCE performances by Tekin et al [52], in which AI models consistently assigned higher scores than human evaluators across clinical skills, including a mean score of 28.23 versus 25.25 for intramuscular injection. Similar tendencies appeared in our included studies: Chiu et al [40] found that ChatGPT scores were substantially higher than faculty scores, and Qian et al [45] reported that LLM scores were approximately 12 percentage points higher than expert scores before calibration. These findings suggest that AI scoring may be scalable and internally consistent, but not necessarily equivalent to human judgment. Additionally, while several studies reported high correlations or ICCs between AI and human raters, evidence regarding construct validity, fairness, and suitability for high-stakes summative competency assessment remains limited. Stronger validation should therefore include transparent scoring logic, calibration against expert benchmarks, comparison with human-human reliability, and domain-specific evaluation rather than reliance on aggregate correlations alone.

The overall evidence maturity of AI-SSPs remains limited compared to their technical promise. This pattern is consistent with broader reviews of digital simulation and AI in health professions education, which have found that usability, satisfaction, and short-term learning outcomes are more commonly evaluated than large-scale trials, implementation, or real-world clinical impact [22]. In this review, most studies demonstrated that AI-SSPs were feasible, acceptable, realistic, or capable of producing structured scores and feedback, but fewer examined whether they improved clinical competency, could be integrated into existing curricula, or performed reliably across various types of users, especially in low-resource contexts. As promising prototypes do not automatically translate into scalable assessment infrastructure, future studies should move beyond product-level feasibility toward larger-scale validation, assess longitudinal learning outcomes, and test in diverse geographic, socioeconomic, and health professional contexts.

A further implementation concern is the need for AI safety protections when AI-SSPs are used for assessment, especially for future high-stakes summative assessments beyond formative ones. As these systems may collect detailed learner transcripts, performance scores, and interaction logs, institutions will need clear data governance policies covering consent, storage, and secondary use of assessment data [53]. Technical guardrails are also needed to prevent unsafe or inappropriate patient responses, protect confidential case content, and ensure that automated feedback does not provide misleading clinical guidance [54]. These governance and safety issues are central to whether AI-SSPs can be trusted as assessment tools. Furthermore, assessment-oriented AI-SSPs also create risks related to new ways of cheating, plagiarism, and the authenticity of performance, particularly if learners can use external AI tools, reuse shared prompts, or access case materials outside the assessment setting. Future systems should therefore include procedural safeguards such as secure testing environments, controlled access to cases and rubrics, and identity verification when appropriate.

This review has several limitations. First, it synthesizes heterogeneous study designs and tool implementations, many of which were small-scale pilots or descriptive prototypes rather than fully validated systems. Second, the rapid evolution of LLM technologies also means that some findings may become outdated as models and deployment practices change. Third, most included studies originated in high-income settings, limiting generalizability to low- and middle-income health systems where SSPs may offer the greatest potential value. At the same time, this review has important strengths. It is, to our knowledge, the first to systematically characterize AI-led SSPs for clinical competency “assessment.” We explored frontend design, backend architecture, and evaluation methods, offering a holistic view of how these tools are currently constructed and assessed. By identifying common architectural patterns and persistent gaps, this review provides a conceptual framework for future SSP development and highlights key design and validation priorities to advance scalable, reliable clinical competency assessment.

In conclusion, AI-enabled SSPs represent an important emerging direction for clinical competency assessment, but the field remains in an early stage of development. This review shows that recent advances in LLMs have accelerated a shift from rule-based virtual patient systems toward more conversational and flexible assessment environments. At the same time, most systems continue to rely on human-authored cases, structured prompts, checklists, rubrics, and other guardrails, underscoring that AI-SSPs are not autonomous assessors of clinical competence. The current evidence base suggests strongest readiness for formative assessment of history-taking, communication, and selected clinical reasoning skills, while broader clinical workflows, including clinical examination, laboratory tests, treatment planning, and management, remain less consistently evaluated. For AI-SSPs to move from promising prototypes to trustworthy assessment infrastructure, future studies must establish stronger evidence of reliability, validity, fairness, and implementation feasibility across diverse learners, professions, and health system contexts. With rigorous validation and responsible governance, AI-SSPs could help expand access to more frequent, standardized, and scalable competency assessments, particularly in settings where traditional SP-based assessments are difficult to sustain.