This paper is in the following e-collection/theme issue:

Theme Issue: ChatGPT and Generative Language Models in Medical Education (102) Chatbots and Conversational Agents (635) Artificial Intelligence (1745) Decision Support for Health Professionals (1341) Generative Language Models Including ChatGPT (501)

Published on in Vol 9 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/46939, first published .
Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study

Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study

Putting ChatGPT’s Medical Advice to the (Turing) Test: Survey Study

Authors of this article:

Oded Nov1 Author Orcid Image ;   Nina Singh2 Author Orcid Image ;   Devin Mann2, 3 Author Orcid Image
Article Authors Cited by (83) Tweetations (21) Metrics

Journals

  1. Sallam M, Salim N, Barakat M, Al-Mahzoum K, Al-Tammemi A, Malaeb D, Hallit R, Hallit S. Assessing Health Students' Attitudes and Usage of ChatGPT in Jordan: Validation Study. JMIR Medical Education 2023;9:e48254 View
  2. Osipov M. On the question of the specifics of the formulation and use of the Turing test for the ChatGPT. Программные системы и вычислительные методы 2023;(4):1 View
  3. Turner J. Triangle of Trust in Cancer Care? The Physician, the Patient, and Artificial Intelligence Chatbot. Cancer Biotherapy and Radiopharmaceuticals 2023;38(9):581 View
  4. Erren T. Patients, Doctors, and Chatbots. JMIR Medical Education 2024;10:e50869 View
  5. Howe P, Fay N, Saletta M, Hovy E. ChatGPT’s advice is perceived as better than that of professional advice columnists. Frontiers in Psychology 2023;14 View
  6. Blease C. Open AI meets open notes: surveillance capitalism, patient privacy and online record access. Journal of Medical Ethics 2024;50(2):84 View
  7. Kerr W, McFarlane K. Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist. Current Neurology and Neuroscience Reports 2023;23(12):869 View
  8. Scquizzato T, Semeraro F, Swindell P, Simpson R, Angelini M, Gazzato A, Sajjad U, Bignami E, Landoni G, Keeble T, Mion M. Testing ChatGPT ability to answer laypeople questions about cardiac arrest and cardiopulmonary resuscitation. Resuscitation 2024;194:110077 View
  9. Alsadhan A, Al-Anezi F, Almohanna A, Alnaim N, Alzahrani H, Shinawi R, AboAlsamh H, Bakhshwain A, Alenazy M, Arif W, Alyousef S, Alhamidi S, Alghamdi A, AlShrayfi N, Rubaian N, Alanzi T, AlSahli A, Alturki R, Herzallah N. The opportunities and challenges of adopting ChatGPT in medical research. Frontiers in Medicine 2023;10 View
  10. Younis H, Eisa T, Nasser M, Sahib T, Noor A, Alyasiri O, Salisu S, Hayder I, Younis H. A Systematic Review and Meta-Analysis of Artificial Intelligence Tools in Medicine and Healthcare: Applications, Considerations, Limitations, Motivation and Challenges. Diagnostics 2024;14(1):109 View
  11. Hillmann H, Angelini E, Karfoul N, Feickert S, Mueller-Leisse J, Duncker D. Accuracy and comprehensibility of chat-based artificial intelligence for patient information on atrial fibrillation and cardiac implantable electronic devices. Europace 2023;26(1) View
  12. Tailor P, Dalvin L, Chen J, Iezzi R, Olsen T, Scruggs B, Barkmeier A, Bakri S, Ryan E, Tang P, Parke D, Belin P, Sridhar J, Xu D, Kuriyan A, Yonekawa Y, Starr M. A Comparative Study of Responses to Retina Questions from Either Experts, Expert-Edited Large Language Models, or Expert-Edited Large Language Models Alone. Ophthalmology Science 2024;4(4):100485 View
  13. Meyer A, Riese J, Streichert T. Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study. JMIR Medical Education 2024;10:e50965 View
  14. Ata A, Aras B, Yılmaz Taşdelen Ö, Çelik C, Çulha C. Evaluation of Informative Content on Cerebral Palsy in the Era of Artificial Intelligence: The Value of ChatGPT. Physical & Occupational Therapy In Pediatrics 2024;44(5):605 View
  15. Mandal S, Wiesenfeld B, Mann D, Szerencsy A, Iturrate E, Nov O. Quantifying the impact of telemedicine and patient medical advice request messages on physicians' work-outside-work. npj Digital Medicine 2024;7(1) View
  16. Wong L, Park H, Looi C. From hype to insight: Exploring ChatGPT's early footprint in education via altmetrics and bibliometrics. Journal of Computer Assisted Learning 2024;40(4):1428 View
  17. Figari Jordan R, Sandrone S, Southerland A. Opportunities and Challenges for Incorporating Artificial Intelligence and Natural Language Processing in Neurology Education. Neurology Education 2024;3(1) View
  18. Mubin O, Alnajjar F, Trabelsi Z, Ali L, Parambil M, Zou Z. Tracking ChatGPT Research: Insights From the Literature and the Web. IEEE Access 2024;12:30518 View
  19. Le M, Davis M. ChatGPT Yields a Passing Score on a Pediatric Board Preparatory Exam but Raises Red Flags. Global Pediatric Health 2024;11 View
  20. Zampatti S, Peconi C, Megalizzi D, Calvino G, Trastulli G, Cascella R, Strafella C, Caltagirone C, Giardina E. Innovations in Medicine: Exploring ChatGPT’s Impact on Rare Disorder Management. Genes 2024;15(4):421 View
  21. Tailor P, Dalvin L, Starr M, Tajfirouz D, Chodnicki K, Brodsky M, Mansukhani S, Moss H, Lai K, Ko M, Mackay D, Di Nome M, Dumitrascu O, Pless M, Eggenberger E, Chen J. A Comparative Study of Large Language Models, Human Experts, and Expert-Edited Large Language Models to Neuro-Ophthalmology Questions. Journal of Neuro-Ophthalmology 2025;45(1):71 View
  22. Moise A, Centomo-Bozzo A, Orishchak O, Alnoury M, Daniel S. Can ChatGPT Replace an Otolaryngologist in Guiding Parents on Tonsillectomy?. Ear, Nose & Throat Journal 2024 View
  23. Howard E, Chong N, Carnino J, Levi J. Comparison of ChatGPT knowledge against 2020 consensus statement on ankyloglossia in children. International Journal of Pediatric Otorhinolaryngology 2024;180:111957 View
  24. Aharoni E, Fernandes S, Brady D, Alexander C, Criner M, Queen K, Rando J, Nahmias E, Crespo V. Attributions toward artificial agents in a modified Moral Turing Test. Scientific Reports 2024;14(1) View
  25. Connors C, Gupta K, Khusid J, Khargi R, Yaghoubian A, Levy M, Gallante B, Atallah W, Gupta M. Evaluation of the Current Status of Artificial Intelligence for Endourology Patient Education: A Blind Comparison of ChatGPT and Google Bard Against Traditional Information Resources. Journal of Endourology 2024;38(8):843 View
  26. Zhang L, Shu J, Hu J, Li F, He J, Wang P, Shen Y. Exploring the Potential of Large Language Models in Radiological Imaging Systems: Improving User Interface Design and Functional Capabilities. Electronics 2024;13(11):2002 View
  27. Martynov A, Bechrakis N, Lever M. Das Aderhautmelanom im Zeitalter der generativen künstlichen Intelligenz – im Gespräch mit ChatGPT. Klinische Monatsblätter für Augenheilkunde 2025;242(02):127 View
  28. Meyer A, Soleman A, Riese J, Streichert T. Comparison of ChatGPT, Gemini, and Le Chat with physician interpretations of medical laboratory questions from an online health forum. Clinical Chemistry and Laboratory Medicine (CCLM) 2024;62(12):2425 View
  29. Imtiaz A, King J, Holmes S, Gupta A, Bafadhel M, Melcher M, Hurst J, Farewell D, Bolton C, Duckers J. ChatGPTversusBing: a clinician assessment of the accuracy of AI platforms when responding to COPD questions. European Respiratory Journal 2024;63(6):2400163 View
  30. Yao J, Aggarwal M, Lopez R, Namdari S. Large Language Models in Orthopaedics. Journal of Bone and Joint Surgery 2024;106(15):1411 View
  31. Collin H, Keogh K, Basto M, Loeb S, Roberts M. ChatGPT can help guide and empower patients after prostate cancer diagnosis. Prostate Cancer and Prostatic Diseases 2025;28(2):513 View
  32. Durmaz Engin C, Karatas E, Ozturk T. Exploring the Role of ChatGPT-4, BingAI, and Gemini as Virtual Consultants to Educate Families about Retinopathy of Prematurity. Children 2024;11(6):750 View
  33. Small W, Wiesenfeld B, Brandfield-Harvey B, Jonassen Z, Mandal S, Stevens E, Major V, Lostraglio E, Szerencsy A, Jones S, Aphinyanaphongs Y, Johnson S, Nov O, Mann D. Large Language Model–Based Responses to Patients’ In-Basket Messages. JAMA Network Open 2024;7(7):e2422399 View
  34. Elyoseph Z, Gur T, Haber Y, Simon T, Angert T, Navon Y, Tal A, Asman O. An Ethical Perspective on the Democratization of Mental Health With Generative AI. JMIR Mental Health 2024;11:e58011 View
  35. Li M, Guenier A. ChatGPT and Health Communication. International Journal of E-Health and Medical Communications 2024;15(1):1 View
  36. Chen S, Kuo H, Chang S. Perceptions of ChatGPT in healthcare: usefulness, trust, and risk. Frontiers in Public Health 2024;12 View
  37. Al Faraby S, Romadhony A, Adiwijaya . Analysis of LLMs for educational question classification and generation. Computers and Education: Artificial Intelligence 2024;7:100298 View
  38. McDarby M, Mroz E, Hahne J, Malling C, Carpenter B, Parker P. “Hospice Care Could Be a Compassionate Choice”: ChatGPT Responses to Questions About Decision Making in Advanced Cancer. Journal of Palliative Medicine 2024;27(12):1618 View
  39. Leslie-Miller C, Simon S, Dean K, Mokhallati N, Cushing C. The critical need for expert oversight of ChatGPT: Prompt engineering for safeguarding child healthcare information. Journal of Pediatric Psychology 2024;49(11):812 View
  40. Ning Y, Teixayavong S, Shang Y, Savulescu J, Nagaraj V, Miao D, Mertens M, Ting D, Ong J, Liu M, Cao J, Dunn M, Vaughan R, Ong M, Sung J, Topol E, Liu N. Generative artificial intelligence and ethical considerations in health care: a scoping review and ethics checklist. The Lancet Digital Health 2024;6(11):e848 View
  41. Arora S, Srivastava A. A cross-lingual syntactic investigation of gender bias and stereotyping in GPT-4o: English vs Hindi. AI and Ethics 2025;5(3):2497 View
  42. Jiang L, Lan M, Menke J, Vorland C, Kilicoglu H. Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines. Scientific Reports 2024;14(1) View
  43. Wang L, Wan Z, Ni C, Song Q, Li Y, Clayton E, Malin B, Yin Z. Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review. Journal of Medical Internet Research 2024;26:e22769 View
  44. Kumari K, Pahuja S, Kumar S. A Comprehensive Examination of ChatGPT's Contribution to the Healthcare Sector and Hepatology. Digestive Diseases and Sciences 2024;69(11):4027 View
  45. Zeljkovic I, Novak M, Jordan A, Lisicic A, Nemeth-Blažić T, Pavlovic N, Manola Š. Evaluating ChatGPT-4’s correctness in patient-focused informing and awareness for atrial fibrillation. Heart Rhythm O2 2025;6(1):58 View
  46. Ghanta S, Al’Aref S, Lala-Trinidade A, Nadkarni G, Ganatra S, Dani S, Mehta J. Applications of ChatGPT in Heart Failure Prevention, Diagnosis, Management, and Research: A Narrative Review. Diagnostics 2024;14(21):2393 View
  47. Ermis S, Özal E, Karapapak M, Kumantaş E, Özal S. Assessing the Responses of Large Language Models (ChatGPT-4, Claude 3, Gemini, and Microsoft Copilot) to Frequently Asked Questions in Retinopathy of Prematurity: A Study on Readability and Appropriateness. Journal of Pediatric Ophthalmology & Strabismus 2025;62(2):84 View
  48. Ferrari-Light D, Merritt R, D'Souza D, Ferguson M, Harrison S, Madariaga M, Lee B, Moffatt-Bruce S, Kneuertz P. Evaluating ChatGPT as a patient resource for frequently asked questions about lung cancer surgery—a pilot study. The Journal of Thoracic and Cardiovascular Surgery 2025;169(4):1174 View
  49. Aydin S, Karabacak M, Vlachos V, Margetis K. Large language models in patient education: a scoping review of applications in medicine. Frontiers in Medicine 2024;11 View
  50. Zhang Y, Wang D, Wang G, Xu P, Zhu Y. Data-driven building load prediction and large language models: Comprehensive overview. Energy and Buildings 2025;326:115001 View
  51. Mendel T, Nov O, Wiesenfeld B. Advice from a Doctor or AI? Understanding Willingness to Disclose Information Through Remote Patient Monitoring to Receive Health Advice. Proceedings of the ACM on Human-Computer Interaction 2024;8(CSCW2):1 View
  52. Demir S. Evaluation of Responses to Questions About Keratoconus Using ChatGPT-4.0, Google Gemini and Microsoft Copilot: A Comparative Study of Large Language Models on Keratoconus. Eye & Contact Lens: Science & Clinical Practice 2025;51(3):e107 View
  53. Hussain A. Unlocking the potential of ChatGPT in academic libraries. IP Indian Journal of Library Science and Information Technology 2024;9(2):80 View
  54. Van Meter A, Wheaton M, Cosgrove V, Andreadis K, Robertson R, Laroia G. The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries. PLOS Digital Health 2025;4(1):e0000711 View
  55. Huang A, Chang M, Khanwalkar A, Yan C, Phillips K, Yong M, Nayak J, Hwang P, Patel Z. Utilization of ChatGPT for Rhinology Patient Education: Limitations in a Surgical Sub‐Specialty. OTO Open 2025;9(1) View
  56. Demir S. Investigating the role of large language models on questions about refractive surgery. International Journal of Medical Informatics 2025;195:105787 View
  57. Lin J, Dai X, Xi Y, Liu W, Chen B, Zhang H, Liu Y, Wu C, Li X, Zhu C, Guo H, Yu Y, Tang R, Zhang W. How Can Recommender Systems Benefit from Large Language Models: A Survey. ACM Transactions on Information Systems 2025;43(2):1 View
  58. Dillion D, Mondal D, Tandon N, Gray K. AI language model rivals expert ethicist in perceived moral expertise. Scientific Reports 2025;15(1) View
  59. Mendel T, Singh N, Mann D, Wiesenfeld B, Nov O. Laypeople’s Use of and Attitudes Toward Large Language Models and Search Engines for Health Queries: Survey Study. Journal of Medical Internet Research 2025;27:e64290 View
  60. Kerr W, McFarlane K, Pucci G, Carns D, Israel A, Vighetti L, Pennell P, Stern J, Xia Z, Wang Y. Supervised machine learning compared to large language models for identifying functional seizures from medical records. Epilepsia 2025;66(4):1155 View
  61. Zeljkovic I, Novak A, Lisicic A, Jordan A, Serman A, Jurin I, Pavlovic N, Manola S. Beyond Text: The Impact of Clinical Context on GPT-4’s 12-Lead Electrocardiogram Interpretation Accuracy. Canadian Journal of Cardiology 2025 View
  62. Soon S, Perry B. Paging Dr. ChatGPT: safety, accuracy and readability of ChatGPT in ENT emergencies. Australian Journal of Otolaryngology 2025;8:8 View
  63. Pan R, García-Díaz J, Valencia-García R. Comparing Fine-Tuning, Zero and Few-Shot Strategies with Large Language Models in Hate Speech Detection in English. Computer Modeling in Engineering & Sciences 2024;140(3):2849 View
  64. Chen Z, Xu L, Zheng H, Chen L, Tolba A, Zhao L, Yu K, Feng H. Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models. Computers, Materials & Continua 2024;80(2):1753 View
  65. Zhuge Q, Wang H, Chen X. TwinStar: A Novel Design for Enhanced Test Question Generation Using Dual-LLM Engine. Applied Sciences 2025;15(6):3055 View
  66. Collin H, Tong C, Srinivas A, Pegler A, Allan P, Hagley D. Evaluating the role of AI chatbots in patient education for abdominal aortic aneurysms: a comparison of ChatGPT and conventional resources. ANZ Journal of Surgery 2025;95(4):784 View
  67. Çıracıoğlu A, Dal Erdoğan S. Evaluation of the reliability, usefulness, quality and readability of ChatGPT’s responses on Scoliosis. European Journal of Orthopaedic Surgery & Traumatology 2025;35(1) View
  68. Alanzi T, Arif W, Alotaibi A, Alnafisi A, Alhwaimal R, Altowairqi N, Alnifaie A, Aldossari K, Althumali K, Alanzi N. Impact of ChatGPT on Diabetes Mellitus Self-Management Among Patients in Saudi Arabia. Cureus 2025 View
  69. Suárez A, Arena S, Herranz Calzada A, Castillo Varón A, Diaz-Flores García V, Freire Y. Decoding wisdom: Evaluating ChatGPT's accuracy and reproducibility in analyzing orthopantomographic images for third molar assessment. Computational and Structural Biotechnology Journal 2025;28:141 View
  70. B. Vu C, Y. Park D. Why Would University Students Use ChatGPT for Health Discourses? A Moderated Mediation Analysis. Journal of Consumer Health on the Internet 2025:1 View
  71. Geracitano J, Anderson B, Rosenzweig M, Dorn S, Khairat S, Conklin J. The Accuracy of ChatGPT in Answering FAQs, Making Clinical Recommendations, and Categorizing Patient Symptoms: A Literature Review. Advances in Health Information Science and Practice 2025 View
  72. Zhang Y, Tian J, Deng F. In generative AI we trust: an exploratory study on dimensionality and structure of user trust in ChatGPT. Interacting with Computers 2025 View
  73. Martela F. Artificial intelligence and free will: generative agents utilizing large language models have functional free will. AI and Ethics 2025 View
  74. Chen D, Parsa R, Swanson K, Nunez J, Critch A, Bitterman D, Liu F, Raman S. Large language models in oncology: a review. BMJ Oncology 2025;4(1):e000759 View
  75. Dede B, Oğuz M, Alyanak B, Bağcıer F, Yıldızgören M. Competencies of Large Language Models About Piriformis Syndrome: Quality, Accuracy, Completeness, and Readability Study. HSS Journal®: The Musculoskeletal Journal of Hospital for Special Surgery 2025 View

Conference Proceedings

  1. Kotek H, Dockum R, Sun D. Proceedings of The ACM Collective Intelligence Conference. Gender bias and stereotypes in Large Language Models View
  2. Shi Y, Ma H, Zhong W, Tan Q, Mai G, Li X, Liu T, Huang J. 2023 IEEE International Conference on Data Mining Workshops (ICDMW). ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphs View
  3. Chaddad A, He C, Jiang Y. 2023 IEEE 23rd International Conference on Bioinformatics and Bioengineering (BIBE). ChatGPT: An Artificial Intelligence-Based Approach to Enhance Medical Applications View
  4. Chaddad A, Jiang Y, He C. 2023 IEEE International Conference on E-health Networking, Application & Services (Healthcom). OpenAI ChatGPT: A Potential Medical Application View
  5. Cao Z, Ma Z, Chen M. 2024 IEEE 11th International Conference on Cyber Security and Cloud Computing (CSCloud). An Evaluation System for Large Language Models based on Open-Ended Questions View
  6. Seabrooke T, Schneiders E, Dowthwaite L, Krook J, Leesakul N, Clos J, Maior H, Fischer J. Proceedings of the Second International Symposium on Trustworthy Autonomous Systems. A Survey of Lay People's Willingness to Generate Legal Advice using Large Language Models (LLMs) View
  7. Varanasi R, Wiesenfeld B, Nov O. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. AI Rivalry as a Craft: How Resisting and Embracing Generative AI Are Reshaping the Writing Profession View
  8. de Giorgio A, Matrone G, Maffei A. 2025 IEEE Engineering Education World Conference (EDUNINE). Detecting Large Language Models in Exam Essays View