Shared decision-making (SDM) is a collaborative, patient-centered approach in which health care professionals and patients jointly make medical decisions based on evidence and patients’ preferences [1]. While SDM is often regarded as a modern model of care, its conceptual roots can be traced back to early discussions in the 1950s emphasizing patient involvement as a core element of the “modern doctor” [2]. Charles et al [3] later formalized SDM into a significant framework comprising 4 key elements: the involvement of at least 2 people, reciprocal information exchange, joint deliberation regarding treatment options, and agreement on a specific decision. SDM is widely recognized as a strategy to reduce informational and power asymmetries in clinical encounters by strengthening patient autonomy and encouraging active participation [4,5]. Empirical evidence indicates that SDM strengthens the therapeutic alliance and is associated with higher patient satisfaction, improved adherence, and better health outcomes [6-10] without increasing consultation time or health care costs [11]. Additionally, when patients experience decision-making as participatory, they report enhanced affective and cognitive outcomes, including greater perceived control, improved understanding, and increased emotional security [12]. SDM is particularly relevant in the management of chronic or complex conditions, where sustainable treatment agreements are essential [13]. Overall, SDM contributes to the quality, sustainability, and acceptance of medical decisions.

Despite its strong ethical and policy endorsement, SDM is not yet consistently implemented in routine clinical practice [14]. Although physicians often report favorable attitudes toward SDM, their actual engagement varies depending on the clinical context, the type of treatment decision, and patient characteristics [15]. Patient-related factors, including cultural beliefs, language barriers, and socioeconomic status, may further limit the consistent application of SDM in everyday care [6]. Moreover, effective participation requires both adequate knowledge and a sense of agency, which not all patients are equally equipped with [16]. Consequently, SDM remains widely endorsed in principle yet difficult to implement consistently in practice due to structural, contextual, and individual barriers [17].

Training programs aimed at improving SDM and communication skills vary considerably in content, structure, and delivery format, and evidence regarding their effectiveness remains limited [18]. While educational interventions for medical students and health care professionals can improve communication skills in the short term [19], robust evidence on long-term effects, participant satisfaction, and sustained behavioral change remains limited [20]. While patient-reported experience measures exist, they are subject to bias, exhibit low response rates, and are not widely implemented [21]. Observer-based measures rely on human raters and are therefore resource-intensive, prone to lapses in concentration, and yield only modest reliability [22-24]. In parallel, AI is increasingly being integrated into medical education, particularly in the domains of communication training, simulation-based learning, formative feedback, and reflective practice [25-27]. AI-driven systems, including large language models, enable scalable and standardized simulated patient encounters that allow learners to repeatedly practice clinical communication in a safe environment [28-30]. Such systems can also provide immediate, structured feedback, potentially supporting deliberate practice and self-regulated learning [31].

Recent studies suggest that large language models (eg, generative pretrained transformer [GPT]–based systems) can reliably evaluate communication quality in simulated interactions and distinguish between different levels of performance [24]. For example, Selvaraj et al [24] demonstrated that AI-generated assessments of SDM-related communication show strong correlations with human ratings, indicating potential for more scalable and efficient evaluation processes. However, despite these promising developments, empirical evidence on the educational impact of AI-based SDM training—particularly regarding learner outcomes such as communicative self-efficacy (CSE), attitudes, and behavioral change—remains limited. Furthermore, CSE as a key factor for effective communication and the implementation of SDM has not been addressed in this context so far. However, evidence shows that communication skills training can improve both performance and self-efficacy [32].

Several important gaps remain in the current literature. First, there is limited evidence on the use of AI-simulated patients specifically for SDM training in medical education. Second, although AI systems have shown promise in evaluating communication behavior, few studies have systematically validated AI-generated feedback against human expert ratings in SDM contexts. Third, there is limited evidence on how AI-based SDM training influences CSE as an important learner-centered outcome in communication skills development. Finally, little is known about how learners perceive AI-generated feedback in SDM training and whether such feedback is experienced as useful and credible for supporting reflective learning.

To address these gaps, this study investigates a chatbot-based SDM training intervention using an AI-simulated patient combined with automated feedback. In addition, the accuracy and perceived usefulness of AI-generated feedback in SDM training are assessed. Furthermore, the study examines changes in CSE and compares SDM attitudes and performance between medical students and physicians.

To evaluate the effectiveness of the proposed intervention, the following research questions are examined:

This feasibility study used a mixed methods pre-post design to evaluate an AI-supported SDM training intervention. The study consisted of a single-session interaction with an AI-simulated patient, followed by automated feedback and postintervention assessment. In addition, a group comparison was conducted between licensed physicians and medical students. Quantitative data were collected at baseline and immediately after the intervention, complemented by qualitative reflections.

Participants included medical students and licensed physicians. Eligibility criteria comprised sufficient proficiency in the German language and current enrollment in medical training after passing the second state examination or active clinical practice. Participants were recruited either via a study link distributed through mailing lists and professional networks or in person during psychosomatic training sessions, where they completed the study on site. Participation was voluntary, and no compensation was provided. Datasets with incomplete data and chats comprising fewer than 5 messages were excluded from analyses.

The data collection took place from November 2025 until February 2026. Initially, participants were introduced to an explanatory flowchart [33] illustrating 6 steps of SDM to ensure a basic level of SDM knowledge across all participants (Multimedia Appendix 1). Participants then engaged in a standardized 20-minute text-based consultation with an AI-simulated patient designed to elicit SDM behaviors. The AI system was configured to assume the role of a patient with a predefined clinical case and responded dynamically to participant input while maintaining consistency across interactions through structured prompting. Following the consultation, participants received AI-generated structured feedback on their SDM performance. The entire process, consisting of chat and questionnaires, took about 30 to 40 minutes. Data collection occurred at 2 time points:

A detailed flowchart illustrating the study procedure is presented in Figure 1.

This application is described in detail in previous studies [34-36]. As validated in [35], this setup produces stable, medically accurate conversations. The model GPT-4-0613 was used via the OpenAI API with the temperature set to 0.1 and other parameters left at default. Details about models and parameters are provided in Multimedia Appendix 2. The feedback system was set to request feedback on buckets of 4 items during each model call. A general description of the 5 response levels of the OPTION-12 scale was included in the prompt, as well as the complete chat. The model GPT-5 was used via the OpenAI API as it was the frontier model at the time of evaluation. Multimedia Appendix 2 provides complete prompts, as well as models and parameters. Screenshots, exemplary interactions with the AI-simulated patient, and AI-generated feedback are provided in Multimedia Appendices 1 and 3 to illustrate the training experience and feedback structure.

To enable AI-generated feedback, a training dataset consisting of 9 expert-evaluated chats assessed using the OPTION-12 scale was developed in a separate pretest phase. These pretest data were collected prior to the main study with different participants and were fully independent of the final evaluation dataset. This strict separation ensured that no overlap occurred between data used for model refinement and data used for evaluation, thereby minimizing the risk of data leakage or circular evaluation. The absolute agreement between the human and AI ratings was moderate (intraclass correlation coefficient [ICC]=0.633, df=2,1, 95% CI 0.039-0.902; P=.02) but considered sufficient for the study’s purpose. The pretest annotations were used to support prompt development and calibration of the AI feedback system. The resulting prompts were iteratively refined through a structured manual optimization process, including the integration of illustrative examples and clarification of category boundaries to improve output consistency and interpretability. The final set of prompts is provided in Multimedia Appendix 2.

Two clinical scenarios (prostate cancer and depression) were selected to represent distinct SDM contexts. The patient vignettes were specifically developed by experts. The first case describes a female student aged 28 years with recurrent depressive disorder. She is afraid of medication but is open to psychotherapy. The second case depicts a craftsman aged 62 years who presented with prostate cancer. He “just wants the thing out” and is skeptical of other procedures. The diagnosis is already known to the patients; the following conversation should therefore focus on discussing treatment options. The detailed patient vignettes are included in Multimedia Appendix 4.

Both cases involve distinct but comparable SDM challenges, including emotional burden, treatment uncertainty, and the need to balance medical evidence with patient preferences. The scenarios were also chosen to allow participants to engage with a case that was broadly aligned with their clinical background or expertise. Both cases were developed based on established clinical guidelines and structured to ensure comparability across participants. To ensure clinical validity and relevance, the cases were subsequently reviewed by additional experts in psychiatry and oncology. These experts assessed the medical accuracy, internal consistency, and appropriateness of the scenarios for SDM training. Feedback from this review process was incorporated to refine and finalize the case vignettes. The cases were integrated into the chatbot and comprehensively tested.

Chat scripts and questionnaire results were collected anonymously via the application. In total, 61 chats and 58 questionnaires were included in the analysis. Chats were segmented into question-answer pairs and compiled in a Microsoft Excel 2408 sheet [39]. Statistical analyses were performed in R (version 4.4.1; R Core Team, 2024 [40]), with significance set at P<.05. Figures were created using Microsoft Excel and R. All statistical tests were conducted using a 2-sided significance level of α=.05. Descriptive statistics were calculated for all variables. Reported metrics include means, SDs, percentages, and Pearson correlations.

SDM performance was operationalized using the total score of the OPTION-12 scale. To evaluate the applicability of the OPTION-12 scale in the AI setting, internal consistency was assessed using Cronbach α, and item-level relationships were examined using interitem correlations. To assess the quality of AI-generated feedback, OPTION-12 ratings produced by the AI were compared with human ratings of the same interactions. Agreement between raters was evaluated using ICCs and Pearson correlation. ICCs were interpreted according to established guidelines [41]. Based on the 95% CI of the ICC estimate, values below 0.50, between 0.50 and 0.75, between 0.75 and 0.90, and above 0.90 indicate poor, moderate, good, and excellent reliability, respectively. These thresholds were used to interpret the level of agreement between AI-generated and human ratings.

Normality of continuous variables was assessed prior to analysis. Nonparametric Wilcoxon signed-rank tests were used when normality assumptions were violated. Homogeneity of variance between groups was assessed using variance ratio tests (F test). Depending on the outcome, either Student t test for independent samples (in cases of equal variances) or Welch t test (in cases of unequal variances) was applied to calculate group differences. Changes in CSE (pre-post) were analyzed using the Wilcoxon signed-rank test, as the distribution of paired differences deviated from normality. In addition to inferential statistics, the effect size was estimated using Cliff delta, a nonparametric measure that quantifies the magnitude and direction of differences between groups by estimating the probability of stochastic dominance without assuming a normal distribution.

A qualitative analysis of selected cases was conducted to investigate differences between AI and human OPTION-12 scores. Chats with the largest absolute differences in the OPTION-12 total scores were identified and subjected to in-depth examination. For each selected chat, item-level ratings were compared, and those with the greatest divergences were analyzed in detail. The analysis was guided by the OPTION-12 coding manual [42], with particular attention to how rating criteria were applied. This approach allowed for a systematic exploration of underlying reasons for rating differences between AI and human raters.

The study design, hypotheses, and analysis plan were preregistered on October 23, 2025, prior to data collection on the Open Science Framework [43].

The study was approved by the Ethics Committee of the Faculty of Medicine at University Hospital Tübingen (375/2025BO2). Participation was voluntary, and no compensation was offered for participation. Informed consent was obtained from all participants prior to participation. Participants confirmed their agreement to the data collection and processing by ticking the respective box on the computer. Participants were fully informed about the study’s purpose and procedures. All data were anonymized. Appropriate measures were taken to ensure the confidentiality and security of participant information. All procedures were conducted in accordance with the Declaration of Helsinki and ethical guidelines for human participants.

Results are presented according to participant-level and encounter-level analyses and structured in line with the 3 research questions. All analyses were conducted in accordance with the prespecified primary, secondary, and exploratory outcomes.

The following results address research questions 2 and 3 at the participant level.

As can be seen in Figure 2, the ratings that AI and human raters allocated differ considerably in some cases. To understand these large discrepancies, the 3 chats with the highest differences in the total OPTION-12 score (chats 21, 38 and 48 with differences of 9, 11, and 10, respectively) were analyzed qualitatively. For each chat, item-level differences between AI and human ratings were examined, and items with the greatest discrepancies were analyzed in detail and compared against the coding manual definitions [42]. In chat 21, item 5 (“The clinician explains the pros and cons of options to the patient”), the physician focused exclusively on explaining the effectiveness and side effects of medications rather than outlining the advantages and disadvantages of all available treatment options. While the human rater assessed the item as a minimal attempt, the AI rated it as exhibiting a good standard. In chat 38, item 7 (“The clinician explores the patient’s concerns [fears] about how problem[s] are to be managed”), AI ratings credited items as fulfilled even when patients introduced relevant information without being explicitly prompted by the physician. Similarly, for item 10 (“The clinician elicits the patient’s preferred level of involvement in decision-making”), the physician only implicitly inquired about patients’ preferred level of involvement in decision-making; nevertheless, the patient provided minimal indications of their preferences, which were again recognized as sufficient by the AI but not by the human rater. In chat 48, item 2 (“The clinician states that there is more than one way to deal with the identified problem”), the physician acknowledged the existence of multiple valid treatment approaches without providing a comprehensive or detailed overview. The AI rated this as “The behavior is observed and performed at a high level,” while the human rater rated it as “The clinician gives the impression that the options are valid and need to be examined more closely.” Overall, discrepancies appear to stem from differences in how implicit communication and patient-initiated contributions are interpreted and evaluated. More details are available in Multimedia Appendix 5.

This study examined a chat-based SDM training intervention using an AI-simulated patient combined with automated feedback on SDM performance. The principal findings indicate that participants were able to engage in structured SDM interactions within the AI-simulated patient and that the system was generally perceived as feasible for communication training. Preliminary results indicate good to strong agreement between AI-generated and human ratings of SDM performance based on the OPTION-12 scale. In addition, medical students and physicians reported similarly positive attitudes toward SDM, but students achieved higher performance scores on the OPTION-12 scale. CSE slightly decreased following the SDM practice intervention and feedback. The observed engagement with the AI chatbot in SDM tasks, as well as the alignment between AI and human ratings, suggests that large language models can support structured communication training. However, given the exploratory nature of the study, the limited sample size, and the fact that the results do not provide sufficient evidence for the validity of AI-generated assessments for SDM performance, the results should be interpreted as preliminary evidence for future research.

Both groups of medical students and licensed physicians reported similarly positive attitudes toward SDM. This is in line with the consistent endorsement of SDM principles across levels of professional training described in the literature [15]. In contrast, differences emerged in observed SDM performance, with medical students achieving significantly higher OPTION-12 scores than physicians, indicating greater adherence to structured SDM approaches in recently trained students [44]. These findings are in line with previous research reporting an inverse association between SDM-related competencies and age [45]. One possible explanation is that students may adhere more closely to structured SDM models due to recent formal training, whereas physicians may rely more on routinized communication patterns shaped by clinical experience [46-48]. The discrepancy between attitudes and observable behavior highlights the potential gap between endorsement of SDM and its practical implementation [14,15].

The observed decrease in CSE following the intervention may reflect a possible recalibration effect, as has been suggested in previous educational research [49]. This effect describes a phenomenon in which engagement in structured learning and feedback increases awareness of the complexity of the task and potentially lowers self-perceived competence. However, in the absence of a control group, this interpretation remains speculative, and causal conclusions cannot be drawn.

Descriptive analyses of OPTION-12 values showed moderate overall performance and values similar to those reported in the literature [50]. Certain SDM components appeared to be implemented more consistently (eg, “The clinician states that there is more than one way to deal with the identified problem”), while others were more challenging (eg, “The clinician assesses the patient’s preferred approach to receiving information to assist decision-making”).

Regarding the comparison of AI-generated and human ratings, the 2 measures showed a strong positive correlation and good interrater reliability, indicating a high level of consistency, as previously described in similar contexts [24]. Nevertheless, the AI consistently assigned higher scores than the human rater. This tendency to agree and give socially desirable answers is a pattern that is already recognized [51,52]. In the exemplary qualitative analysis conducted as part of our study, it became clear that discrepancies appear to stem mainly from differences in how implicit communication and patient-initiated contributions are interpreted. Overall, these findings should be interpreted cautiously given the exploratory nature of the study and the absence of a fully established validation framework.

The results suggest that AI-powered chatbots may be a viable and scalable tool for teaching SDM skills in medical education and professional development. Feedback on real clinical conversations is often difficult to implement, critical to the protection of personal data, and resource-intensive to obtain, making AI a less demanding alternative. This study shows that AI can apply the OPTION-12 scale in a similar way to a human rater and can adopt the patient’s perspective during the assessment. Without having surveyed actual patients, we were therefore able to gain an initial impression of the patient’s perspective, as suggested in the literature [20,21]. Such tools can complement existing communication training formats by providing low-threshold, time-independent practice opportunities and individualized feedback. In terms of training practice, integrating AI-supported SDM training into medical curricula could help standardize the teaching of SDM principles and establish SDM on a broad scale.

Several limitations should be considered when interpreting the results of this study. First, the simulated consultations were text-based and conducted with an AI-simulated patient, which may not have fully captured the complexity of real-world clinical encounters, particularly nonverbal aspects of communication. Future research should therefore examine the transferability of SDM skills acquired in chatbot-based training to real-world patient encounters. Second, the study was based on a pre-post design without a control group, which limits causal inferences about the effects of chatbot-based SDM training. Future studies should focus on randomized controlled designs compared with traditional teaching methods. Furthermore, longitudinal follow-up assessments are necessary to evaluate sustained impact. Third, the study was conducted with a relatively small and self-selected sample, which may introduce selection bias and limit representativeness. Furthermore, the study did not assess participants’ prior familiarity with AI tools, digital literacy, or typing speed, all of which may have influenced their interaction with the chatbot. Participants with higher levels of digital competence or faster typing skills may have been better able to engage with the system, potentially confounding performance outcomes. Future studies should consider measuring and controlling for these factors to better isolate the effects of the intervention. The fairly high dropout rate could be attributed to a lack of time and competing clinical or academic obligations, since this was a voluntary study with no incentives. Participants may have clicked on the link out of initial interest but lost interest shortly thereafter, perhaps due to a lack of time. Future studies should therefore aim to recruit larger and more diverse samples across multiple institutions to improve generalizability and reduce selection bias. Fourth, the evaluation of the AI-generated feedback was conducted by only 1 rater. This introduces a potential risk of subjectivity and limits the robustness of the validation process, reducing the methodological rigor of the findings. Future research should therefore incorporate multiple raters and report interrater reliability metrics to ensure a more reliable and comprehensive validation of AI-generated feedback. Finally, performance bias cannot be excluded, as participants were not blinded to the study purpose and may have adapted their behavior based on perceived expectations. Similarly, assessment bias may have been introduced due to the lack of blinding in human ratings. Future research should implement blinding procedures for both participants and raters wherever feasible to minimize expectancy effects and rating bias.

This exploratory study provides preliminary evidence that AI-simulated patients can be integrated into communication training to support SDM. Participants were able to engage in structured SDM interactions within the AI environment, and initial results suggest a good level of agreement between AI-generated and human ratings of SDM performance. However, these findings should be interpreted cautiously due to the exploratory design, limited sample size, and methodological constraints. While AI-simulated patients may have potential as a supportive tool for SDM training, further research is required to establish validity, reliability, and generalizability using larger samples and standardized, rigorously validated AI implementations.