Interpersonal skills are an invaluable necessity for health care providers. Impactful medical practice requires the ability to adapt treatment plans to a patient’s unique preferences, which is founded on the development of the patient-provider relationship. Numerous studies have shown that effective physician communication is positively correlated with patient satisfaction [1] and positive health outcomes [2]. Patient involvement in their own medical care, often termed “shared decision-making,” has also been observed to correlate with increased overall and disease-specific quality of life scores in orthopedic patients [3]. The now near-ubiquity of internet usage has resulted in patients frequently consulting online materials for additional medical information. Nearly all American households had internet access in 2021 [4], and recent Centers for Disease Control and Prevention data suggest that more than half of adults (58.5%) use the internet for medical advice [5]. This trend is also mirrored internationally, with 1 study showing that 64.3% of Lebanese patients consulted web resources for acute symptoms, with 19.2% seeking web information before physician consultation [6].

The increasing popularity of AI technologies has introduced a new potential source for internet-accessible medical information. Large language models (LLMs), such as GPT-4o (OpenAI), are currently being evaluated for their ability to play a role in health care. AI has been shown to accurately process medical knowledge and show proficiency at the level of third- and fourth-year medical students in preparation for the United States Medical Licensing Examination [7-9]. In addition, GPT-4o has been challenged with postgraduate examination materials, including preparation materials for the American Board of Surgery In-Training Examination [10,11], neurology specialist examinations [12], and the Orthopaedic In-Training Examination [13,14] and, in most cases, has performed at the level of a human medical trainee. Recent exploration has shown that ChatGPT models can also directly respond to patient questions about common conditions across a variety of specialties with generally satisfactory answers [15-17].

Nevertheless, integration of AI into health care settings remains controversial due to a lack of clarity surrounding its strengths and limitations, and it is unclear whether AI can be safely recommended to serve as a substitute or even as an adjunct to physician consultation [18]. The “soft skills” of medical training include compassion and empathetic communication, which are essential to identifying and navigating the nuances present within each individual patient interaction. Although AI technologies have shown some capability in correctly reproducing technical knowledge and answering patient questions, their capability to identify and process user emotion is not well understood.

LLMs are modeled on neural networks, in which units or “tokens” of data are connected and linked by associations, which are strengthened or weakened based on the presence of patterns within training data [19]. Essentially, the models use the patterns identified from their training data to generate outputs that are the most probabilistically likely “correct” responses [20]. Transformer architecture is the modern evolution of neural networks and uses “self-attention” to consider and weight tokens in relation to each other [21]. Through the training process, the model adapts the level of attention given to various tokens to identify the most relevant components of the input and better generate more relevant responses in the given context [21]. Ultimately, transformer architecture improves the statistical predictive accuracy of LLMs, especially in the processing of large volumes of inputs, by more accurately weighing the relevance of input tokens [21].

Training can occur through the establishment of parameters, which guide the model in processing data inputs to promote or oppose specific outputs; positive reinforcement of a specific association teaches the model that the reinforced logic has an increased likelihood of leading to the correct output [22]. Feedback can be provided to the model based on the final output (outcome supervision) or throughout the logical process (process supervision), and the model applies this feedback and recalculated probabilities to direct the processing of new data [23]. Therefore, the veracity of AI-generated outputs is dependent directly on the quality and accuracy of the training data and subtleties within the inputs themselves, rather than solely on known truths [24]. As a result, AI models have been documented to “hallucinate” or fabricate information and present it convincingly as fact [25]. Researchers in one study found that a number of inconsistencies and hallucinations were noted when questions on sensitive or obscure topics were presented to several AI models [26], supporting the hypothesis that AI performance declines when available data are limited. “Factuality” hallucinations, when models report information that violates known facts, and “faithfulness” hallucinations, when models deviate from the information provided in the given input, are theorized to occur due to inconsistent training and model evaluation metrics [27].

Despite numerous studies evaluating AI’s accuracy in answering questions based on empiric knowledge, the literature supporting AI’s performance in health care–based medical ethical, legal, and social scenarios is limited and inconsistent. Some reports demonstrate that LLMs can answer ethical scenario questions [28] and respond to patient clinical questions with answers that are rated positively for empathy by human raters [29,30], while others found ChatGPT models performed worse on ethics questions compared to medical trainees [10,31,32]. When subjected to emotion-related testing, AI demonstrated the ability to correctly identify and manage emotions but struggled when associating emotions to rationale and scored lower than the human average on the “using emotions to facilitate thought” section of the examination [33].

Variant AI models have been designed to address the general safety and ethical concerns surrounding the outputs produced by LLMs. One such model is Claude, created by Anthropic, which describes it as “helpful, honest, and harmless” [34]. Claude was developed using a novel “Constitutional AI” method, meaning that instead of using human feedback during the training process, feedback is instead provided by the AI model itself, guided by a provided set of “constitutional principles” [35]. During the supervised training process, the model is asked prompts that will purposefully generate a “harmful” output, for example, asking for instructions on how to break into a house. The model is then asked to critique the output based on a constitutional principle (“how is the last response unethical or illegal”) and then to revise the output according to the critique (“rewrite the response to remove unethical or illegal content”) [36]. In the reinforcement learning or unsupervised phase, the model is asked to generate multiple responses to a prompt, and a different feedback model is asked to choose one of the responses also based on a constitutional principle (“which of these responses is the most ethical and legal?”) [36]. Therefore, humans are not providing feedback for each model response themselves, but instead instructing a “teaching” AI to guide the “learning” model to align with specific rules, or constitutional principles. Anthropic sources its constitutional principles from a number of ethic-centered documents, including the UN Declaration of Human Rights, as well as principles developed by other AI laboratories and Anthropic researchers themselves during model training to discourage harm, aggression, offense, violence, and so on [35]. Although Claude is intentionally designed with ethics and safety in mind, there are no available studies specifically exploring Claude’s performances in ethics-related health care scenarios.

Throughout medical school and postgraduate medical training, learners are taught how to manage complex social issues and interpersonal interactions with a high degree of emotional intelligence and empathy. Even ethical dilemmas, which are addressed using well-defined ethical principles, must be navigated within the larger context of patient emotion. Medical training examinations often use questions about end-of-life discussions, patient safety, communication strategies, and informed consent to assess these intangible skills. This study aimed to evaluate the ability of GPT-4o and Claude Sonnet 4 to answer health care–based ethical medical board–style questions accurately and consistently. We hypothesized that GPT-4o and Claude will differ in performance between question levels and categories.

All questions used for testing were scenario-based multiple-choice questions for which there was only one correct answer. Medical student–level questions were sourced from the UWorld Step 2 CK question bank and AMBOSS, 2 resources designed to prepare third- and fourth-year medical students for the Step 2 benchmark examination. The UWorld system allows for question sourcing from selected subject categories. One category of interest was the broad “Social Sciences (Ethics/Legal/Professional)” category, which involves the subcategories of Communication and Interpersonal Skills, Healthcare Policy and Economics, Medical Ethics and Jurisprudence, Patient Safety, and System-based practice and Quality Improvement. The subcategory “Communication and Interpersonal Skills” was also specifically selected as a second source of questions to assess whether performance differed on questions specifically relating to interpersonal communication scenarios, which often require interpretation of patient tone and social cues. Both types of questions typically begin with patient vignettes. Ethics or legal questions often ask, “Which of the following is the best next step in management of this patient?” in scenarios that require the physician to make a decision about common clinical dilemmas involving end-of-life care, next-of-kin decision-making power, or consent for emergent procedures. Interpersonal questions typically present a patient scenario regarding a sensitive subject such as alcohol dependence before asking, “which of the following is the most appropriate response to this patient?” The test taker is given several different responses as answer choices that vary in tact and counseling advice. Individual question specifics cannot be reproduced, as these question banks are copyright protected; however, these questions are designed to imitate the style of questions typically asked on the USMLE Step 2 CK examination. An example of this question’s style provided by the USMLE is provided as follows [37]:

As these ethical dilemmas are typically framed in the context of emotional situations, the complexity and nuance of addressing these issues often stem from the application of concrete ethical principles to emotionally charged and sensitive scenarios. Therefore, identification of the correct or “most appropriate response” requires not only an understanding of ethical principles but also the ability to apply these principles in a manner that is sensitive to the patient. The interpersonal communication questions were included to assess not only LLM performance in ethical scenarios but also to assess whether there was unique accuracy or inaccuracy in the skill of identifying the textbook “most appropriate answer.”

The UWorld software allows users to create custom tests based on subject categories with up to 40 questions per test. The individual questions available in each subject category are not visible to test takers due to the nature of the test bank, and therefore, the questions included in each test could not be individually predetermined. A 40-question test was generated from each category described previously—one test using questions from the broader “Social Sciences (Ethics/Legal/Professional) category” and one test using questions exclusively from the “Communication and Interpersonal Skills” subcategory.

As the UWorld questions were not individually selected prior to test generation, 15 questions were duplicated between the 40-question sets. The duplicated questions were not excluded as the average test taker performance score is generated based on the full 40-question set, and therefore, inclusion of all questions was necessary to compare LLM performance to human performance. This method ensured that each test would be representative of tests encountered by human testers.

The AMBOSS question bank similarly allows the creation of tests involving randomly generated questions within a subcategory, although with a higher maximum of 50 questions. The AMBOSS “Legal Medicine and Ethics” category was the only equivalent available subcategory to test the ethics-related medical competencies, and therefore, a test of 50 randomly selected questions from the AMBOSS Legal Medicine and Ethics discipline was generated. The format of these questions mirrored those of UWorld questions in that they are designed to imitate the style of questions asked on the USMLE Step 2 examination.

Another area of interest was whether model performance would vary in specialty-specific contexts. By testing questions written at a level for medical residents, LLM performance can be evaluated in scenarios at higher levels of medical complexity. This aim of the study was treated as completely exploratory, and causal relationships should not be inferred. Orthopedics was chosen as the specialty due to the author’s access to Orthobullets, an online resource intended to prepare orthopedic residents for the Orthopaedic In-Training Examination. This question bank similarly allows the generation of subject-specific tests and provides 2 question categories relevant to this study “Ethics in Orthopaedics Practice” and “Legal Considerations in Orthopaedic Practice.” A total of 70 questions were available: 21 from the Ethics category and 49 from the Legal category. All questions were included in this study. The ethics questions required test takers to identify the most ethical course of actions in orthopedic patient–specific scenarios involving potential financial conflicts of interest with research studies and transfers of value from pharmaceutical or medical device representatives, obtaining interpretation services for a non–English-speaking patient, and appropriate marketing practices. Legal consideration questions involved scenarios in which test takers were required to identify the most appropriate way to obtain consent in emergent situations or manage medical or surgical errors, altering the medical record of surgical errors, or preoperative protocol management, and professionalism violations.

All questions from UWorld and AMBOSS were text-based by nature of the test banks, while Orthobullets questions included text-only questions as well as questions with supporting images. The inclusion of questions involving images introduced a potential for confounding, but it was thought that analysis performed at the question bank–level would illuminate whether LLM performance varied by question bank. Only 7 questions of the 70 sourced from Orthobullets included images, and this volume was not considered sufficient to draw conclusions about varying performance based on question modality. For all question banks, the software provided a metric of the “average” score earned by test takers on the same test, and this is the benchmark used in this study as the “human test taker” performance metric. It is important to note that the multiple-choice questions used are not validated measures of ethical reasoning or empathetic reasoning but simply an available tool to compare performance to that of human test takers. Information on the specifics of how these benchmarks are calculated is not publicly available for any of the question banks. Therefore, questions could not be excluded in our study from the tests generated by the software, as the average human test taker benchmark could not be manually calculated.

A zero-shot approach was used to conduct the trials; no training or prompt engineering was performed to provide context to the models [38]. Instead, the free, publicly available version of GPT-4o was opened in a browser window, a singular question and its corresponding answer choices were copied and pasted verbatim as the input, and the resulting selection was recorded. If any images were provided in accompaniment to the question stem, these were also provided to the chatbot in the same input as the question stem. To prevent bias or learning for subsequent trials, a new, unique browser window was opened, and the above, zero-shot approach was repeated for each trial. If the provided answer varied between trials, the answer generated in the majority of trials was recorded as GPT-4o’s selected response. Three trials were performed for each question, and additional trials were conducted as needed to achieve a majority response if there was no consensus between the original 3 trials. Typically, GPT-4o provided reasoning for its selection without additional prompting. If the incorrect answer was chosen, the reasoning behind the incorrect choice was recorded. If no reasoning was provided, GPT-4o was then prompted with “Why is answer choice [the correct answer] incorrect?” If answers varied between trials, GPT-4o’s evaluation of the correct answer choice, incorrect answer choice, and previously selected answer choices were recorded. All questions tested required the selection of a singular answer choice. On the occasions when GPT-4o selected 2 answer choices as the correct answer, the browser was refreshed, and the trial was repeated until GPT-4o provided only one answer choice. The testing procedure can be visualized in Figure 1.

For all questions, the question banks also provided individual test taker benchmarks on the percentage of subscribers that selected each individual answer choice. It is important to note that the exact method of calculation for this metric was also not publicly available for each question bank. If GPT-4o selected the incorrect answer, it was noted whether or not its selection was the incorrect answer choice most popularly chosen by humans.

The same zero-shot testing method was used for Claude Sonnet 4. Claude users are required to pick a category of interest on sign-in. The “learning and studying” option was selected for all trials included in this study, but no additional prompt engineering or configurations were made to this chatbot.

Statistical analysis to compare the AI model’s overall performance against human performance was performed using SPSS (version 31.0.0.0 (117); IBM) with one-sample proportion tests, and McNemar tests were used to compare performance between models. Chi-square and Fisher exact tests were used to analyze model ability to select the most popular incorrect response as well as the frequency of varying responses to identical questions.

It is unclear how the question banks derived the reported human performance averages and whether these scores are truly representative of the true population of medical test takers. Therefore, statistical inferences are not valid, and comparisons between LLM and human performance are reported purely descriptively. The performance of GPT-4o was similar to the average score of the question banks’ human test takers (71% vs 70%). GPT-4o correctly answered 89 (68%) of 130 general medical student–level questions, and 51 (73%) of 70 orthopedic resident–level questions with no significant difference in performance, compared to the human averages of 71% and 73%, respectively. Claude answered 180 (90%) of 200 total questions correctly, performing better than the human average (89% vs 71%) and significantly better than GPT-4o (89% vs 70%; P<.001). Claude’s score was higher than the provided human average on all question banks. Claude also performed significantly better than GPT-4o on all question categories except the UWorld Social Sciences and Orthobullets Ethics questions; results and statistical conclusions are summarized in Table 1, with the number of questions correct and correlated weighted score as provided by each test bank.

Question banks only provided average human test taker score in percentages, and equivalent numbers of correctly answered questions were calculated based on the provided percentage correct and total number of questions used.

GPT-4o answered 60 (30%) of 200 total questions incorrectly. Of these 60 questions, the most popular incorrect response was chosen 37 (62%) times. Analysis did show a significant difference in the frequency that GPT-4o chose the most popular incorrect answer between both UWorld general social sciences and interpersonal-specific sets (P=.03), but no difference between the UWorld (general and interpersonal) and AMBOSS (legal medicine and ethics) sets (P=0.33), Orthobullets Ethics and Legal sets (P=.22), or medical student general and orthopedic resident–level (UWorld/AMBOSS and Orthobullets) sets (P=.46; Table 2).

Claude incorrectly answered 20 (10%) of 200 total questions. Analysis using Fisher exact test did not show a significant difference in the frequency that Claude chose the most popular incorrect answer between both UWorld general social sciences and interpersonal-specific sets (P=.49), UWorld (general and interpersonal) and AMBOSS (legal medicine and ethics) sets (P>.99), Orthobullets Ethics and Legal sets (P>.99) or medical student general and orthopedic resident–level (UWorld/AMBOSS and Orthobullets) sets (P>.99; Table 2).

GPT-4o varied answers on 30 (15%) of 200 total questions, while Claude varied answers on 19 (10%) of the 200 total questions. There was no significant difference between the frequency of varying answers between GPT-4o responses for both the UWorld general social sciences and interpersonal-specific sets (P=.23) UWorld (general and interpersonal) and AMBOSS (legal medicine and ethics) sets (P=.25), and Orthobullets Ethics and Legal sets (P=.90). However, GPT-4o produced variable answers between trials significantly less frequently for orthopedic resident–level questions than for medical student general questions (P=.002; Table 3).

Claude did not significantly differ responses for both UWorld general social sciences and interpersonal-specific sets (P=.75), UWorld (general and interpersonal) and AMBOSS (legal medicine and ethics) sets (P=.53), and Orthobullets Ethics and Legal sets (P=.90), or between medical student general questions and orthopedic resident–level questions (P=.06).

There was no significant difference in variability of responses between the 2 models overall (P=.09), for medical student general questions (P=.07) or for orthopedic resident–level questions (P>.99).

When both models varied their answers, the explanations between various sessions were often contradictory; one session of the model would provide reasoning as to why an answer choice was correct, and a different session of the model would cite the same reasoning as the reason that answer choice was incorrect. Examples of contradictory reasoning are provided in Tables 4 and 5.

For example, one question from the UWorld broad social sciences question bank involved identifying the next step in the management of a patient with no advance directive. This question required 7 trials to achieve a majority answer, as GPT-4o fluctuated its choice between 3 options. In one trial, GPT-4o recommended the answer of counseling the patient’s family against aggressive care measures due to the patient’s poor prognosis, explaining, “The family may still struggle with feelings of guilt or uncertainty, but presenting the situation clearly in terms of the patient’s current and likely future suffering allows the family to make an informed compassionate decision. It’s essential to approach the conversation with empathy, helping the family understand that continuing aggressive treatment may prolong suffering without offering a meaningful benefit to the patient.” In another trial, GPT-4o explained that this same answer choice would be incorrect because “reminding this family of this fact alone may come across as dismissive or judgmental, especially when they have not yet reached a consensus.” Similarly, one session of GPT-4o recommended consultation with an ethics committee because “The ethics committee can help facilitate a structured discussion, ensuring that the decision-making process aligns with the patient’s best interests and ethical principles...” while another session argued against consulting an ethics committee, stating “referral to the ethics committee can be helpful...however, in this case, the issue is more about supporting the family through the decision-making process rather than resolving a fundamental ethical dilemma.” Claude provided similar styles of contradictions for its responses, but notably correctly mentioned data from 2 relevant research studies to justify the responses. The first response cited a study performed in 1984 [39], and the second, contradictory response mentioned the first study but also referenced more recent data published in 2019 (Table 5; Orthobullets Legal) [40].