Artificial intelligence (AI) is a new technology that has changed various industries, including medicine. AI refers to the development of computer systems capable of performing complex tasks that normally require human intelligence, such as understanding conversation, recognizing patterns or images, and making decisions. Traditionally, AI in medicine was used in areas such as medical imaging, diagnostics tests, and prediction tools. However, it evolved and became involved in other aspects of the medical field, for example, helping physicians gather patient data before the visit [1].

One of the most remarkable developments in AI is the advancement of large language models and natural language processing, which aim to facilitate the automatic analysis of language, mimicking human language understanding. ChatGPT is an application built based on large language models, namely, GPT-3.5 or GPT-4. This newly developed AI technology enables users to engage in interactive conversations and receive humanlike responses, thereby creating a more dynamic and engaging user experience [2]. ChatGPT fascinates many people in a variety of fields. In the medical field, it has been used to help write manuscripts [3-5]. However, researchers were still concerned about the contents’ ethical consideration and validity [6]. Many researchers have also evaluated ChatGPT for medical education, such as taking examinations and comparing the results to medical students [7-11]. The use of ChatGPT to help in the patient care process has also been reported [12,13].

The potential of using AI in the medical field, especially orthopedics, is promising. For example, deep learning AI has been used for detecting and classifying many orthopedic conditions, such as degenerative spinal conditions, rotator cuff injury, and implant loosening [14-16]. ChatGPT itself has been tested with the American Board of Orthopaedic Surgery Examination, but it cannot pass the exam [17]. One of the challenges encountered in medical practice is the high volume of patients, which may sometimes prevent physicians from providing detailed information to patients. Given that ChatGPT is a language model focused on communication, it could help provide appropriate treatment plans and patient education.

Therefore, we aim to investigate how medical students and practicing doctors perceive the use of ChatGPT in clinical settings and medical education. Additionally, we will explore whether there are differences in perception between medical students and doctors at various levels of experience regarding ChatGPT’s responses to a clinical question. We hypothesized that different levels of clinical experience would change participants’ perceptions of ChatGPT.

This study reflected how medical students and various levels of physicians felt about medical answers from ChatGPT and its applications. We found that participants with different clinical experience levels had different perceptions toward ChatGPT’s use for clinical practice and medical education. Medical students generally had a positive perception, while practicing physicians were more neutral.

For clinical practice, a higher proportion of attending staff and residents disagreed with using ChatGPT. While medical students were satisfied with responses that followed textbooks and sounded authentic, more experienced physicians could detect the pitfalls of the responses. They had shared their concerns, which had both supporting and conflicting literature.

The first concern was the lack of patient-specific treatment plans. ChatGPT seemed to provide accurate and reproducible advice for general knowledge. For example, bariatric surgeons rated responses of ChatGPT as “comprehensive” for 86.8% of the questions asked [18]. Gastroenterologists also rated ChatGPT’s response to common patient questions with a score of 3.9 (SD 0.8), 3.9 (SD 0.9), and 3.3 (SD 0.9) out of 5 for accuracy, clarity, and efficacy, respectively [12]. It could provide a well-structured and comprehensive response to common breast augmentation surgery questions [19]. The responses to common questions about retinal detachments were rated appropriate in 80%-90% of the questions asked [20]. However, patient-specific conditions should also be included in treatment planning. The most appropriate treatment method selection may need clinical reasoning and experience. Therefore, ChatGPT’s answer could be used as a general outline for treatment, but currently, it could not replace a physician’s clinical reasoning and judgment. If the model is further explicitly trained for some medical conditions, it might be able to provide more specific treatment recommendations.

Another concern about using ChatGPT in clinical practice was its evidence-based element. It seemed that ChatGPT gathered resources from reasonably reliable sources. For example, in responding to public health questions, 91% of the answers given were determined to be based on evidence [21]. However, there were reports of ChatGPT citing nonexistent publications when asked [22]. Data validity was another point of concern. Due to increasing numbers of publications and emerging predatory publishers, ChatGPT might have relied on references that it deemed valid but were, in fact, fraudulent. Therefore, physicians may still have advantages over AI because they can assess and choose the most valid, reliable, and up-to-date knowledge for their clinical practice.

Most participants agreed that ChatGPT could be used for patient education. Some research also supported this opinion. ChatGPT had the potential to be used as a diabetic educator [23]. It could also provide an effective diet plan for people with food allergies, albeit with minor errors [13]. ChatGPT correctly answered 61% of basic public medical consultations, but only 39% of questions asked by health care personnel were correctly answered [24]. It seemed that for general medical questions, ChatGPT could generate appropriate advice. However, for more specific topics, the development of a dedicated chatbot might be more beneficial. For example, the SnehAI chatbot was developed to educate adolescents in India about sexual health and showed promising results [25]. Another chatbot, “VIRA,” was created to communicate and ensure COVID-19 vaccine safety with young adults and minority populations [26].

In medical education, ChatGPT could be used in various aspects [27]. Using ChatGPT for preparing for OSCE and other exams was mentioned by participants and in the literature [28]. For OSCE, it could help by generating example scenarios, suggesting a proper physical examination, and giving appropriate medical advice. Surprisingly, it could score even higher than humans for a virtual OSCE in obstetrics and gynecology [29]. However, it should be noted that ChatGPT responses were compared to only two human candidates and might not represent the whole picture. For multiple-choice question examinations, ChatGPT could answer some questions correctly and give explanations with acceptable insights and reasoning. However, the results of using ChatGPT were quite varied, from passing the exam to failing some [7,8,10,30-33]. When explored in detail, the passing score of ChatGPT in most tests was at average or slightly above minimal passing level. Therefore, it supported the fact that many attending staff and residents felt that the response by ChatGPT was superficial and did not show a deep understanding of the topic. For more advanced examination levels, such as resident-level examinations, ChatGPT performed more poorly [7,34,35]. For example, ChatGPT’s score in the plastic surgery in-training examination was ranked at the 49th percentile compared with first-year residents but significantly worse than fifth- and sixth-year residents at the zeroth percentile [9]. However, more recent research using an updated GPT-4 model capable of advanced reasoning and complex problem-solving showed remarkable results, and the GPT-4 model consistently outperforms GPT-3.5. GPT-4 was able to pass the Peruvian National Licensing Examination, the Japanese Medical Licensing Examination, German medical state examinations, and the Family Medicine Residency Progress Test with exceptional scores [11,36-38].

Our study tried to gather information from different levels of students and physicians and contrasted their results. We found that less experienced medical students might overlook some potential pitfalls of using ChatGPT in clinical practice and medical education. Even though there were many benefits of using ChatGPT, medical teachers needed to be aware of the risks and warn their medical students accordingly.

The limitation of our study was that we used only one scenario of knee osteoarthritis. If there were more scenarios of other diseases, the perception might differ; however, we felt that knee osteoarthritis was a good representation of a condition commonly encountered by various levels of physicians and would generate a diverse response. Moreover, ChatGPT has been known to answer according to the prompt and may change its answer depending on how the question was asked. In our study, the question contained the “General knowledge” word, which might affect how the respondent rates the answer. The participants also came from one center, which could limit the generalizability of the results. Additionally, the response rate of 68.2% might indicate the selection bias toward people who were already interested in AI, therefore, boosting the positive perception toward ChatGPT. Furthermore, besides the limited representativeness of doctors and medical students within the survey setting, the omission of patient perspectives neglected the input of arguably the most crucial stakeholder in health care. Lastly, the latest ChatGPT model is GPT-4, which is more advanced and may be able to provide more detailed responses. However, the superiority of ChatGPT-4 compared to ChatGPT-3.5 has mainly been proven in a scenario of multiple-choice examinations.

In conclusion, medical students generally had a positive perception of using ChatGPT for guiding treatment and medical education, whereas graduated doctors were more cautious in this regard. Nonetheless, both medical students and graduated doctors positively perceived using ChatGPT for creating patient educational materials.