TY  - JOUR
AU  - Holderried, Friederike
AU  - Stegemann-Philipps, Christian
AU  - Herrmann-Werner, Anne
AU  - Festl-Wietek, Teresa
AU  - Holderried, Martin
AU  - Eickhoff, Carsten
AU  - Mahling, Moritz
PY  - 2024
DA  - 2024/8/16
TI  - A Language Model–Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study
JO  - JMIR Med Educ
SP  - e59213
VL  - 10
KW  - virtual patients communication
KW  - communication skills
KW  - technology enhanced education
KW  - TEL
KW  - medical education
KW  - ChatGPT
KW  - GPT: LLM
KW  - LLMs
KW  - NLP
KW  - natural language processing
KW  - machine learning
KW  - artificial intelligence
KW  - language model
KW  - language models
KW  - communication
KW  - relationship
KW  - relationships
KW  - chatbot
KW  - chatbots
KW  - conversational agent
KW  - conversational agents
KW  - history
KW  - histories
KW  - simulated
KW  - student
KW  - students
KW  - interaction
KW  - interactions
AB  - Background: Although history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback. Objective: In our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students’ performance in history taking with a simulated patient. Methods: We conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients’ responses and provide immediate feedback on the comprehensiveness of the students’ history taking. Students’ interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback. Results: Most of the study’s participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4’s role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed “almost perfect” agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model’s assessments were overly specific or diverged from human judgement. Conclusions: The GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2024/1/e59213
UR  - https://doi.org/10.2196/59213
DO  - 10.2196/59213
ID  - info:doi/10.2196/59213
ER  -