We conducted a single-blind, parallel-group, RCT to evaluate whether an AI-powered virtual patient (SOPHIE) could improve clinician communication skills in serious illness conversations. Participants were randomized in a 1:1 ratio to the intervention (SOPHIE training) or the control (reading modules) group. The study is reported in accordance with the CONSORT 2025 Statement for randomized trials, and the completed CONSORT (Consolidated Standards of Reporting Trials) checklist is provided in Checklist 1.

We performed a power analysis using pilot data that showed an effect size of 0.82 (Cohen d) to determine the target sample size for the study. A 1-tailed, 2-sample t test at a 5% type I error rate was selected to detect improvement in the SOPHIE arm relative to the control. We determined that a minimum of 48 participants (24 per arm) was needed to achieve at least 80% power. Sessions were recorded for blinded evaluation. Control participants were offered SOPHIE training after the study. Figure 1 demonstrates the RCT study design.

We enrolled 51 participants, including medical, nursing, and physician assistant students, as well as practicing clinicians, from June to December 2024. Most participants were recruited from the University of Rochester School of Medicine and Dentistry, the University of Rochester School of Nursing, and Strong Memorial Hospital. Recruitment was conducted via email outreach, personal contacts, and campus flyers. Participants provided informed consent before participation. The study took place via the online video-calling platform Zoom (Zoom Communications Inc). Participants were randomized to the SOPHIE intervention (n=26) or the control (n=25). We used stratified block randomization to ensure balance across different participant classes (eg, trainees vs professionals). Prior participation in the SOPHIE pilot study was the only exclusion criterion. Each participant received a US $30 Amazon gift card as compensation. Table 1 presents the demographic breakdown of participants by age, gender, race or ethnicity, and clinical role.

All participants viewed an instructional video on SIC and how to use the MVP framework and 3E skills [31]. The intervention group completed 3 SOPHIE modules, each focusing on 1 of 3 communication skills—Empathize, Be Explicit, and Empower—combining real-time, interactive dialogue with a virtual patient and immediate automated feedback. The Empathize module focused on responding to emotional distress, the Be Explicit module on clearly communicating prognosis and uncertainty, and the Empower module on eliciting values and supporting a patient-centered care plan. After each module, participants received personalized feedback on effective behaviors, missed opportunities, and suggested alternatives for the specific module. Examples of feedback and interface visuals are shown in Figure 2. Figure 3 shows a participant navigating through one of the SOPHIE modules and pre-SP and post-SP interactions.

Control participants reviewed reading materials on the same SIC framework for the same equal 30-minute duration. The reading materials used in the control arm covered the same SIC framework and the same 3 target skills addressed in the SOPHIE modules: Empathize, Be Explicit, and Empower. Thus, both groups received the same core educational content, but only the SOPHIE group had the opportunity to practice these skills interactively with a virtual patient and to receive automated, personalized feedback. Control participants reviewed the reading materials for the same 30-minute duration as the SOPHIE training to maintain comparable exposure time across arms.

All participants completed preintervention and postintervention simulated encounters with a trained SP drawn from 1 of 3 patient scenarios. Each participant was paired with 1 of 13 SPs, who interacted with the participant via video conference and rated the participant’s communication skills using the previously described evaluation rubric [31,39]. The 13 SPs who directly interacted with participants were hired through an external service (Professional Communication Simulators). In each interaction, SPs were randomly assigned to 1 of 3 advanced cancer clinical cases (Table 1, case distribution) developed in collaboration with experts in communication training (RME and TC). The SPs were blinded to group assignment (intervention or control) but were aware of the encounter order (pre or post). We recorded a video of each participant-SP encounter and 4 additional actors, who work with the ACT Program (Advanced Communication Training Program, University of Rochester), which is directed by one of the authors (TC), watched the videos and provided a separate rating for both preintervention and postintervention participant-SP encounters. These actors were blinded to both the intervention and the encounter order. Both SPs and third-party (TP) raters completed an online form, described in Table S2 of Multimedia Appendix 1, to evaluate the participant.

After the intervention, all participants completed a postinteraction survey and a 10-minute interview (Tables S3-S7 in Multimedia Appendix 1). The shared survey asked both control and SOPHIE participants to rate perceived improvements in Empathize, Be Explicit, Empower, and overall communication skills, as well as stress during the SP interactions. Arm-specific survey questions differed by condition: control participants evaluated the usefulness and engagement of the reading materials, whereas SOPHIE participants evaluated the usefulness, engagement, stress, personalized feedback, and willingness to repeat the SOPHIE modules.

The interview included both structured and open-ended questions. Structured prompts asked all participants what they learned, what messages they found most important, whether the training improved their confidence or competence in the 3E skills, and whether they preferred interactive systems, passive materials, or a combination for communication training. Open-ended prompts allowed participants to describe their emotional experience, expected clinical takeaways, concerns, and suggestions for an ideal computer-based training system. We conducted and recorded the interviews through Zoom immediately after the experiment. These recordings were reviewed, and major themes were extracted from participant responses.

To our knowledge, there are no widely established benchmark scores for SIC performance. We therefore adopted the MVP framework and converted the 3 target skills—Empathize, Be Explicit, and Empower—into a structured Likert-scale questionnaire to quantify communication behaviors on a numeric scale. The full set of statements used in the rubric is provided in Table S2 of Multimedia Appendix 1. For each of the 3E skills (S), we summed the Likert scores associated with the skill and normalized them to between 0 and 1 using equation 1. For the overall score, questions related to all 3 skills were summed. This process was repeated for the ratings provided by the SP and the 4 TPs, and the resulting scores were averaged:

We averaged the evaluations across 5 raters to reduce the effects of individual biases and normalized the scores to between 0 and 1 to make them interpretable and comparable across raters.

The rubric was intended to capture within-participant improvement from before to after the intervention, not to define an ideal level of performance or estimate population-level clinical competence. Future studies may use a similar rubric to evaluate whether repeated SOPHIE practice produces larger or more durable gains over time.

We summarized participant demographics using descriptive statistics. Communication skill outcomes were analyzed for each of the 3Es (Empathize, Be Explicit, and Empower) and for the overall score. For descriptive reporting, we calculated pre-to-post change scores by subtracting each participant’s first SP encounter score from their second encounter score.

To test whether improvement differed by study arm while accounting for repeated measurements within participants, we fit separate linear mixed effects models for Empower, Be Explicit, Empathize, and overall communication score. Each model included fixed effects for study arm, time (first vs second SP encounter), and the study arm-by-time interaction, with a random intercept for each participant to account for within-participant correlation. Models were estimated using maximum likelihood under an intention-to-treat framework, including all randomized participants, and were specified as:

score ~ study arm × time + (1 | participant).

The study arm-by-time interaction was the primary test of the intervention effect because it estimates the additional pre-to-post improvement in the SOPHIE arm relative to the control arm. Statistical significance was evaluated using α of .05, and 95% CIs were reported for model estimates.

This study was approved by the University of Rochester Institutional Review Board (IRB; study ID: STUDY00006689). All participants provided informed consent after receiving an explanation of the study procedures, risks, and benefits, and their participation was voluntary. The study was conducted remotely using Zoom, and video meetings were recorded only with participants’ consent. Study data and recordings were handled according to IRB-approved confidentiality procedures, stored in secure cloud storage, and deidentified before analysis or data sharing. Only aggregate demographic information is reported.

Both participant groups were demographically balanced, with no notable differences in age, gender, or clinical background (Table 1). Logistic regression confirmed no demographic predictors of group assignment. The CONSORT diagram is displayed in Figure 8. Of the 65 eligible participants who signed up, 14 did not show up at the scheduled meeting time. Of the 51 participants who showed up and signed the consent form, all completed the full study. Because the full study took place in a single, continuous session, we achieved a zero-dropout rate.

We examined agreement between the SP ratings and the blinded TP rating using a leave-one-out comparison. The full analysis is reported in Table S1 of Multimedia Appendix 1. For each rater, we compared that rater’s score with a reference consensus formed by averaging the remaining raters’ scores. SP refers to the SPs who interacted directly with participants, while TP0 to TP3 refer to the 4 SPs who reviewed the recorded encounters as blinded TP raters. All ratings were normalized at the rater level before analysis to reduce individual scale-use differences.

The SP ratings showed moderate-to-strong agreement with the blinded TP rater consensus (Pearson correlation coefficient r=0.68; Spearman rank correlation ρ=0.67; mean absolute error=0.15; intraclass correlation coefficient [ICC] (A,1)=0.63; 95% CI 0.50‐0.74). Agreement varied across individual TP raters but was generally within a similar range, with ICC(A,1) values from 0.57 to 0.78. These results suggest that SPs and blinded TP raters captured broadly consistent communication quality while also reflecting meaningful differences in perspective between first-person interaction and TP observation.

To evaluate communication skill development, we analyzed changes in participants’ scores (Empower, Be Explicit, Empathize, and overall) between their first and second SP conversations. Each conversation (n=102) received 5 independent ratings—1 from the participating SP and 4 from blinded TP reviewers. Final scores were averaged across these ratings and change scores (∆) were calculated by subtracting preintervention from postintervention values and then aggregated as described earlier. Of the expected 510 ratings from 51 participants across 2 SP encounters and 4 independent raters, 4 ratings were missing, resulting in 506 ratings for the final analysis. Interrater reliability was strong, with an ICC of 0.88 (95% CI 0.82‐0.93) among TP reviewers blinded to intervention or control status before and after the study.

Both groups showed improvement from the first to the second SP encounter, but the SOPHIE group showed larger gains across all communication outcomes (Table 2). For Empower, mean scores increased from 0.49 to 0.55 (Δ=0.06) in the control group and from 0.41 to 0.58 (Δ=0.17) in the SOPHIE group. Similar patterns were observed for Be Explicit (control Δ=0.05; SOPHIE Δ=0.13), Empathize (control Δ=0.07; SOPHIE Δ=0.14), and overall score (control Δ=0.06; SOPHIE Δ=0.15).

To account for the repeated-measures structure of the randomized pre-post design, we analyzed communication outcomes using linear mixed effects models under an intention-to-treat framework. All randomized participants (n=51; 102 observations) were included in the analyses.

The mixed effects models showed significant study arm-by-time interactions for all outcomes. The SOPHIE arm showed greater pre-to-post improvement than the control arm for Empower (β=.11, 95% CI .04-.17; P=.002), Be Explicit (β=.08, 95% CI .03-.13; P=.002), Empathize (β=.07, 95% CI .01-.13; P=.03), and overall communication score (β=.09, 95% CI .04-.14; P=.001). In practical terms, these interaction estimates represent the additional improvement associated with SOPHIE beyond the improvement observed in the control arm.

Participants using the SOPHIE system reported markedly greater confidence in their communication skill development compared with controls on a Likert scale ranging from 1 to 5, with average perceived improvement scores of 4.31 (SOPHIE) vs 3.83 (control) on a 5-point scale (P=.009).

The training system itself received strong evaluations, particularly for its actionable feedback on communication skills—the highest-rated feature (mean: 4.7/5, SD: 0.18). Users also praised SOPHIE’s usability (mean: 4.2/5, SD: 0.36), though ratings for the virtual human’s realism were more moderate (mean: 3.4/5, SD: 0.17).

This RCT tested whether SOPHIE would lead to greater improvement in SIC skills and confidence than did written educational materials. The findings support this hypothesis: participants assigned to SOPHIE showed greater pre-to-post improvement than controls across Empower, Be Explicit, Empathize, and overall communication scores, as well as higher self-reported confidence. These gains occurred after a brief training session of less than 30 minutes, suggesting that AI-based virtual patient training may be feasible within clinical education settings.

Confidence gains were also higher with SOPHIE, indicating that interactive practice not only improved competence but also helped learners recognize their progress. Interviews reinforced this: many participants felt underprepared for the emotional demands of SIC, underscoring the value of deliberate, low-risk practice.

Participants rated SOPHIE’s actionable feedback (4.7/5) higher than the avatar’s visual realism (3.4/5), suggesting that visual fidelity has room for improvement.

Prior work on LLM-based virtual patients has largely focused on system feasibility, usability, and intrinsic quality rather than on behavior learning outcomes. Studies by Elhilali et al [25] and Cook et al [27] demonstrate that LLM-driven patient simulations can generate realistic dialogue and are perceived as usable, but they do not assess whether such systems improve clinician communication skills. A recent systematic review [26] similarly highlights the lack of consistent outcome measures and the limited evidence of skill transfer in the literature. Importantly, very little published work has evaluated AI-based training specifically for SIC, and the available literature is particularly limited in randomized trials with behavior outcomes.

Some randomized studies evaluate AI-based simulations but rely mainly on self-reported outcomes. Tyrrell et al [29] and Roux et al [30] report improvements in perceived skills or confidence, yet do not show clear gains in objectively assessed communication behaviors. Other work using virtual SPs without LLM-driven dialogue has demonstrated improvements in clinical reasoning and diagnostic skills [28], but these systems do not target conversational communication or provide generative, personalized feedback. Thus, although AI-based virtual patient systems are increasingly being studied in medical education, there remains a dearth of evidence on whether such systems can improve SIC skills that transfer to human SP encounters.

In contrast, this study evaluates an LLM-based virtual patient designed for deliberate practice in SIC and measures effectiveness using a randomized controlled design with multirater behavior assessments. This work extends prior studies by providing evidence that generative AI–based patient simulations can lead to measurable improvements in clinician communication behaviors beyond usability or self-reported learning gains.

As health systems embrace telemedicine and remote education, scalable and equitable training is essential. SOPHIE leverages generative AI to standardize, measure, and democratize communication training, with the potential to mitigate bias by enabling practice with diverse patient identities. Even modest clinician gains at scale could reduce disparities in outcomes. Applications in conflict resolution, medical error disclosure, and misinformation highlight its versatility. Figure 9 demonstrates diverse appearances of virtual avatars that can be easily integrated into our system.

These findings provide emerging empirical support for the hypothesis that AI can serve as an adaptive tutor in complex interpersonal domains [35]. By enabling deliberate practice with structured, immediate feedback [36,37,43], AI systems can support skill development traditionally cultivated through in-person mentorship.

Importantly, our assessment focused not only on participant engagement with the training platform but also on whether skills translated to performance in encounters with SPs. This approach supports the evaluation of skill transfer beyond the virtual training environment—an important consideration for tools intended to support clinical practice. Looking ahead, systems like SOPHIE may serve as a foundation for the use of AI that strengthens, rather than replaces, the clinician’s voice in moments that matter most.

Although our approach has several strengths, the long-term effects of SOPHIE-based training remain unknown. Additionally, the assessment of improvements in communication skills was conducted in a simulated setting; future work is needed to assess its real-world clinical impact. The observed learning effects may also have been influenced by novelty or motivational effects, as many participants were likely interacting with a virtual patient for the first time. In addition, the single-scenario training, the relatively small sample size, and the same-day interaction and evaluation limit the generalizability of the findings. Our control condition measured the effectiveness of SOPHIE against a static reading condition. However, how the SOPHIE intervention would compare with other forms of interactive training, such as SP-based training and feedback, remains still unknown. Future studies should include more diverse virtual patients and clinical cases, larger samples, longer follow-up periods, and comparisons of AI training with human training. Four of the expected 510 ratings were missing; given the small proportion of missing ratings, their effect was likely limited, although some measurement uncertainty remains.

Our dialogue agent captures the user’s speech by transcribing it into text and does not take into account the user’s facial expressions, prosody, intonation, body language, or other nonverbal cues. While participants recognized emotions such as sadness and concern during interactions, some noted that SOPHIE’s expressions felt “uncanny.” This highlights a common design challenge: avatars that look too human can be unsettling, while more stylized expressions may actually make emotional cues clearer. Whether intentionally favoring exaggerated, easy-to-read expressions is educationally superior to striving for subtle, lifelike realism should be explored further in future studies.

One important limitation of the SOPHIE system is its dependence on commercial providers for LLMs, speech recognition, and speech synthesis. This reliance can hinder exact reproducibility over time because cloud-based models and APIs may be updated or deprecated. In addition, transmitting conversation data to external services may limit the use of SOPHIE for protected or highly sensitive clinical conversations, even when appropriate safeguards are in place. These concerns could be mitigated by deploying local models, such as the Llama 3 [44] or Qwen 2.5 [45] families for language generation, Whisper [46] for speech recognition, and Kokorofor speech synthesis, allowing all processing to remain within institutional infrastructure. However, such deployments introduce additional engineering complexity and require dedicated hardware resources and ongoing maintenance, potentially increasing the operational burden for academic or health care institutions.

Looking ahead, systems like SOPHIE may serve as a foundation for a new kind of AI-human partnership in health care—one that strengthens, rather than replaces, the clinician’s voice in moments that matter most.