TY  - JOUR
AU  - Hudon, Alexandre
AU  - Kiepura, Barnabé
AU  - Pelletier, Myriam
AU  - Phan, Véronique
PY  - 2024
DA  - 2024/4/4
TI  - Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study
JO  - JMIR Med Educ
SP  - e54067
VL  - 10
KW  - psychiatry
KW  - artificial intelligence
KW  - medical education
KW  - concordance scripts
KW  - machine learning
KW  - ChatGPT
KW  - evaluation
KW  - education
KW  - medical learners
KW  - learning
KW  - teaching
KW  - design
KW  - support
KW  - tool
KW  - validation
KW  - educational
KW  - accuracy
KW  - clinical questions
KW  - educators
AB  - Background: Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)–based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials. Objective: The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion. Methods: This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs. Results: A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content. Conclusions: This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2024/1/e54067
UR  - https://doi.org/10.2196/54067
DO  - 10.2196/54067
ID  - info:doi/10.2196/54067
ER  -