TY  - JOUR
AU  - Hsu, Hsing-Yu
AU  - Hsu, Kai-Cheng
AU  - Hou, Shih-Yen
AU  - Wu, Ching-Lung
AU  - Hsieh, Yow-Wen
AU  - Cheng, Yih-Dih
PY  - 2023
DA  - 2023/8/21
TI  - Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation
JO  - JMIR Med Educ
SP  - e48433
VL  - 9
KW  - ChatGPT
KW  - large language model
KW  - natural language processing
KW  - real-world medication consultation questions
KW  - NLP
KW  - drug-herb interactions
KW  - pharmacist
KW  - LLM
KW  - language models
KW  - chat generative pre-trained transformer
AB  - Background: Since OpenAI released ChatGPT, with its strong capability in handling natural tasks and its user-friendly interface, it has garnered significant attention. Objective: A prospective analysis is required to evaluate the accuracy and appropriateness of medication consultation responses generated by ChatGPT. Methods: A prospective cross-sectional study was conducted by the pharmacy department of a medical center in Taiwan. The test data set comprised retrospective medication consultation questions collected from February 1, 2023, to February 28, 2023, along with common questions about drug-herb interactions. Two distinct sets of questions were tested: real-world medication consultation questions and common questions about interactions between traditional Chinese and Western medicines. We used the conventional double-review mechanism. The appropriateness of each response from ChatGPT was assessed by 2 experienced pharmacists. In the event of a discrepancy between the assessments, a third pharmacist stepped in to make the final decision. Results: Of 293 real-world medication consultation questions, a random selection of 80 was used to evaluate ChatGPT’s performance. ChatGPT exhibited a higher appropriateness rate in responding to public medication consultation questions compared to those asked by health care providers in a hospital setting (31/51, 61% vs 20/51, 39%; P=.01). Conclusions: The findings from this study suggest that ChatGPT could potentially be used for answering basic medication consultation questions. Our analysis of the erroneous information allowed us to identify potential medical risks associated with certain questions; this problem deserves our close attention. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2023/1/e48433
UR  - https://doi.org/10.2196/48433
UR  - http://www.ncbi.nlm.nih.gov/pubmed/37561097
DO  - 10.2196/48433
ID  - info:doi/10.2196/48433
ER  -