TY  - JOUR
AU  - Nakao, Takahiro
AU  - Miki, Soichiro
AU  - Nakamura, Yuta
AU  - Kikuchi, Tomohiro
AU  - Nomura, Yukihiro
AU  - Hanaoka, Shouhei
AU  - Yoshikawa, Takeharu
AU  - Abe, Osamu
PY  - 2024
DA  - 2024/3/12
TI  - Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study
JO  - JMIR Med Educ
SP  - e54393
VL  - 10
KW  - AI
KW  - artificial intelligence
KW  - LLM
KW  - large language model
KW  - language model
KW  - language models
KW  - ChatGPT
KW  - GPT-4
KW  - GPT-4V
KW  - generative pretrained transformer
KW  - image
KW  - images
KW  - imaging
KW  - response
KW  - responses
KW  - exam
KW  - examination
KW  - exams
KW  - examinations
KW  - answer
KW  - answers
KW  - NLP
KW  - natural language processing
KW  - chatbot
KW  - chatbots
KW  - conversational agent
KW  - conversational agents
KW  - medical education
AB  - Background: Previous research applying large language models (LLMs) to medicine was focused on text-based information. Recently, multimodal variants of LLMs acquired the capability of recognizing images. Objective: We aim to evaluate the image recognition capability of generative pretrained transformer (GPT)-4V, a recent multimodal LLM developed by OpenAI, in the medical field by testing how visual information affects its performance to answer questions in the 117th Japanese National Medical Licensing Examination. Methods: We focused on 108 questions that had 1 or more images as part of a question and presented GPT-4V with the same questions under two conditions: (1) with both the question text and associated images and (2) with the question text only. We then compared the difference in accuracy between the 2 conditions using the exact McNemar test. Results: Among the 108 questions with images, GPT-4V’s accuracy was 68% (73/108) when presented with images and 72% (78/108) when presented without images (P=.36). For the 2 question categories, clinical and general, the accuracies with and those without images were 71% (70/98) versus 78% (76/98; P=.21) and 30% (3/10) versus 20% (2/10; P≥.99), respectively. Conclusions: The additional information from the images did not significantly improve the performance of GPT-4V in the Japanese National Medical Licensing Examination. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2024/1/e54393
UR  - https://doi.org/10.2196/54393
UR  - http://www.ncbi.nlm.nih.gov/pubmed/38470459
DO  - 10.2196/54393
ID  - info:doi/10.2196/54393
ER  -