TY  - JOUR
AU  - Madrid, Julian
AU  - Diehl, Philipp
AU  - Selig, Mischa
AU  - Rolauffs, Bernd
AU  - Hans, Felix Patricius
AU  - Busch, Hans-Jörg
AU  - Scheef, Tobias
AU  - Benning, Leo
PY  - 2025
DA  - 2025/3/21
TI  - Performance of Plug-In Augmented ChatGPT and Its Ability to Quantify Uncertainty: Simulation Study on the German Medical Board Examination
JO  - JMIR Med Educ
SP  - e58375
VL  - 11
KW  - medical education
KW  - artificial intelligence
KW  - generative AI
KW  - large language model
KW  - LLM
KW  - ChatGPT
KW  - GPT-4
KW  - board licensing examination
KW  - professional education
KW  - examination
KW  - student
KW  - experimental
KW  - bootstrapping
KW  - confidence interval
AB  - Background: The GPT-4 is a large language model (LLM) trained and fine-tuned on an extensive dataset. After the public release of its predecessor in November 2022, the use of LLMs has seen a significant spike in interest, and a multitude of potential use cases have been proposed. In parallel, however, important limitations have been outlined. Particularly, current LLMs encounter limitations, especially in symbolic representation and accessing contemporary data. The recent version of GPT-4, alongside newly released plugin features, has been introduced to mitigate some of these limitations. Objective: Before this background, this work aims to investigate the performance of GPT-3.5, GPT-4, GPT-4 with plugins, and GPT-4 with plugins using pretranslated English text on the German medical board examination. Recognizing the critical importance of quantifying uncertainty for LLM applications in medicine, we furthermore assess this ability and develop a new metric termed “confidence accuracy” to evaluate it. Methods: We used GPT-3.5, GPT-4, GPT-4 with plugins, and GPT-4 with plugins and translation to answer questions from the German medical board examination. Additionally, we conducted an analysis to assess how the models justify their answers, the accuracy of their responses, and the error structure of their answers. Bootstrapping and CIs were used to evaluate the statistical significance of our findings. Results: This study demonstrated that available GPT models, as LLM examples, exceeded the minimum competency threshold established by the German medical board for medical students to obtain board certification to practice medicine. Moreover, the models could assess the uncertainty in their responses, albeit exhibiting overconfidence. Additionally, this work unraveled certain justification and reasoning structures that emerge when GPT generates answers. Conclusions: The high performance of GPTs in answering medical questions positions it well for applications in academia and, potentially, clinical practice. Its capability to quantify uncertainty in answers suggests it could be a valuable artificial intelligence agent within the clinical decision-making loop. Nevertheless, significant challenges must be addressed before artificial intelligence agents can be robustly and safely implemented in the medical domain. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2025/1/e58375
UR  - https://doi.org/10.2196/58375
DO  - 10.2196/58375
ID  - info:doi/10.2196/58375
ER  -