TY  - JOUR
AU  - Huang, Ryan ST
AU  - Lu, Kevin Jia Qi
AU  - Meaney, Christopher
AU  - Kemppainen, Joel
AU  - Punnett, Angela
AU  - Leung, Fok-Han
PY  - 2023
DA  - 2023/9/19
TI  - Assessment of Resident and AI Chatbot Performance on the University of Toronto Family Medicine Residency Progress Test: Comparative Study
JO  - JMIR Med Educ
SP  - e50514
VL  - 9
KW  - medical education
KW  - medical knowledge exam
KW  - artificial intelligence
KW  - AI
KW  - natural language processing
KW  - NLP
KW  - large language model
KW  - LLM
KW  - machine learning, ChatGPT
KW  - GPT-3.5
KW  - GPT-4
KW  - education
KW  - language model
KW  - education examination
KW  - testing
KW  - utility
KW  - family medicine
KW  - medical residents
KW  - test
KW  - community
AB  - Background: Large language model (LLM)–based chatbots are evolving at an unprecedented pace with the release of ChatGPT, specifically GPT-3.5, and its successor, GPT-4. Their capabilities in general-purpose tasks and language generation have advanced to the point of performing excellently on various educational examination benchmarks, including medical knowledge tests. Comparing the performance of these 2 LLM models to that of Family Medicine residents on a multiple-choice medical knowledge test can provide insights into their potential as medical education tools. Objective: This study aimed to quantitatively and qualitatively compare the performance of GPT-3.5, GPT-4, and Family Medicine residents in a multiple-choice medical knowledge test appropriate for the level of a Family Medicine resident. Methods: An official University of Toronto Department of Family and Community Medicine Progress Test consisting of multiple-choice questions was inputted into GPT-3.5 and GPT-4. The artificial intelligence chatbot’s responses were manually reviewed to determine the selected answer, response length, response time, provision of a rationale for the outputted response, and the root cause of all incorrect responses (classified into arithmetic, logical, and information errors). The performance of the artificial intelligence chatbots were compared against a cohort of Family Medicine residents who concurrently attempted the test. Results: GPT-4 performed significantly better compared to GPT-3.5 (difference 25.0%, 95% CI 16.3%-32.8%; McNemar test: P<.001); it correctly answered 89/108 (82.4%) questions, while GPT-3.5 answered 62/108 (57.4%) questions correctly. Further, GPT-4 scored higher across all 11 categories of Family Medicine knowledge. In 86.1% (n=93) of the responses, GPT-4 provided a rationale for why other multiple-choice options were not chosen compared to the 16.7% (n=18) achieved by GPT-3.5. Qualitatively, for both GPT-3.5 and GPT-4 responses, logical errors were the most common, while arithmetic errors were the least common. The average performance of Family Medicine residents was 56.9% (95% CI 56.2%-57.6%). The performance of GPT-3.5 was similar to that of the average Family Medicine resident (P=.16), while the performance of GPT-4 exceeded that of the top-performing Family Medicine resident (P<.001). Conclusions: GPT-4 significantly outperforms both GPT-3.5 and Family Medicine residents on a multiple-choice medical knowledge test designed for Family Medicine residents. GPT-4 provides a logical rationale for its response choice, ruling out other answer choices efficiently and with concise justification. Its high degree of accuracy and advanced reasoning capabilities facilitate its potential applications in medical education, including the creation of exam questions and scenarios as well as serving as a resource for medical knowledge or information on community services. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2023/1/e50514
UR  - https://doi.org/10.2196/50514
UR  - http://www.ncbi.nlm.nih.gov/pubmed/37725411
DO  - 10.2196/50514
ID  - info:doi/10.2196/50514
ER  -