This was a cross-sectional study conducted at a tertiary care teaching hospital. The study was started after approval from the Institutional Ethics Committee of Jawaharlal Nehru Medical College Hospital, Aligarh Muslim University (1018/IEC/23/8/23). All participants were adult MBBS students and provided written informed consent after receiving information about the study purpose and procedures. Data were collected anonymously using coded identifiers. No personal or identifiable information was recorded or published. Participants received no monetary or academic compensation.

Participants included 50 second-year medical students who had just joined the second year and 50 third-year students who had recently passed their second year (ie, N=100). The second-year students, designated as the “naive group,” had no prior formal exposure to pharmacology coursework. The third-year students, designated as the “learned group,” had completed a comprehensive 1-year pharmacology curriculum during their second year. This curriculum consisted of 80 hours of didactic lectures, 150 hours of interactive teaching, including practical laboratory sessions, case-based learning modules, self-directed learning modules, short group teaching, and regular assessments, including sessional and professional examinations. Students had completed this coursework approximately 2‐3 months prior to participation in this study. The curriculum covered topics such as general pharmacology, autonomic pharmacology, cardiovascular drugs, and antimicrobials, aligned with the medical council syllabus for undergraduate medical education. All learned group participants had passed their second-year pharmacology examination with a minimum score of 50%.

Prior to the assessment, all participants received a 15-minute standardized orientation session on using ChatGPT and were provided with an eBook of Pharmacology. This training covered basic prompt formulation and how to ask follow-up questions. Students accessed ChatGPT through computer workstations in a controlled examination hall environment to ensure standardized conditions. For the Google search set, students used desktop computers with standard Google Chrome or Mozilla Firefox browsers. For the e-book set, students were provided with PDF versions of a pharmacology textbook accessible on the same computers. The training session was conducted using a standardized presentation to maintain homogeneity of assistance tool exposure. The self-knowledge set required no external tools. All sessions were proctored to ensure compliance with assigned methods for each question set.

The 80 MCQs were divided into four parallel sets (A, B, C, and D), each containing 20 questions. To ensure equivalence across sets, questions were matched for content area coverage (eg, autonomic, cardiovascular, and chemotherapy) [1], question type (theory, calculation, or image-based) [2], and estimated difficulty level based on faculty assessment [3].

Students had 15 minutes to respond to each set of questions. They were instructed to respond to the questions in a manner that self-knowledge was used to answer questions in set A; e-books were used to answer questions in set B; internet search engine (Google) was used to answer questions in set C; and AI-LLM GPT (ChatGPT) was used to answer questions in set D. Each correct answer was given 1 mark, and no negative marking was given. The methodology flowchart of this study is shown in Figure 1.

Data analysis was done using SPSS (version 23; IBM Corp) and R software package (R Foundation for Statistical Computing). The mean (the average number of correctly answered questions out of the 20 MCQs included in each augmentation set such as self-knowledge, e-book, internet search engine, and GPT-4o) between different groups was assessed using the independent test. Intergroup and intragroup comparisons were conducted using analysis of covariance (ANCOVA), with self-knowledge group (set A) being a covariate. A P value <.05 was considered to indicate statistical significance.

AI is changing the face of medical education, as it suggests some new solutions to the old forms of teaching, increases the educational experience, and improves the results of students [14,15]. AI-based solutions, including LLMs, such as ChatGPT, virtual assistants, and AI-based simulation technologies, can change how medical students learn and apply their knowledge [16,17].

This study found that learned students consistently outperformed naive students across all learning methods. However, the naive students who used ChatGPT-4o achieved higher scores than the learned students who used conventional resources (ie, self-knowledge, e-books, or Google), suggesting that AI-LLM tools may help bridge knowledge gaps more effectively than traditional learning resources.

Our study demonstrates that AI-powered LLMs such as ChatGPT can notably improve students’ performance on pharmacology MCQs, particularly for those without prior subject knowledge. Although our findings show that students achieved higher MCQ test scores when using ChatGPT-4o, we cannot conclude that this represents a genuine improvement in cognitive performance, critical thinking, or clinical decision-making skills. Higher scores may reflect the AI’s ability to rapidly retrieve and present relevant information rather than deeper student understanding. The findings indicated that learned students have outperformed naive students in all 4 sets, and AI-LLM GPT recorded the best performance. The greatest effect size was observed in the AI group (partial η²=0.328), which indicates the tremendous influence of AI-LLM in ensuring the students improve their understanding and problem-solving skills. These results are consistent with those of previous research in that AI tools may enhance critical thinking and support learning under the condition that they are properly used by a student who already has a good knowledge grounding [18].

As opposed to other methods, the e-books set exhibited the lowest performance. This can be explained by the fact that it takes a lot of time to navigate and search relevant information in the e-books. This reveals the difficulties of self-learning with the help of conventional digital sources, especially in the case of naive students who are probably not able to manage their time efficiently. In contrast, the self-knowledge set outperformed the e-books set, which means that dependence on inherent knowledge, although restricted to a certain extent, can be more helpful than the effective utilization of external means.

Although our quantitative data demonstrate that ChatGPT-4o enhanced student performance, particularly for naive learners, the mechanisms underlying this effectiveness warrant deeper exploration. There may be several reasons for AI-LLM GPT (ChatGPT-4o) to outperform other methods of assistance.

First, ChatGPT-4o provides immediate, targeted responses without requiring students to navigate complex search results or textbook indices. Informal feedback from participants suggested that students who used Google often struggled to identify authoritative sources among numerous search results, whereas those who used e-books spent considerable time locating relevant chapters. In contrast, ChatGPT-4o delivered direct answers to questions, reducing the cognitive load related to information seeking.

Second, ChatGPT-4o’s conversational interface may facilitate iterative learning. Students could ask follow-up questions to clarify concepts, request examples, or seek a specific explanation for a specific MCQ—a dynamic interaction not possible with static e-books and more efficient than reformulating multiple Google searches.

Third, the structured, synthesized format of ChatGPT-4o responses may be particularly beneficial for naive learners who lack the prior knowledge framework to evaluate and integrate fragmented information from multiple sources. ChatGPT-4o essentially predigests information, whereas Google and e-books require students to perform this synthesis themselves—a task that may be especially challenging without foundational knowledge.

However, these remain as hypotheses. Future research should use mixed methods approaches, including think-aloud protocols during tool use, posttask interviews exploring student decision-making processes, and screen recordings analyzing search strategies and information evaluation patterns. Such qualitative data would provide richer insights into how students with different knowledge levels interact with various resources and why certain tools prove more effective. The study of recognizing the pattern of wrong responses in the AI-LLM GPTs output may also generate an “array of errors” for further training and improving LLMs.

These findings have substantial implications for both local and global medical education practices. In our institutional context, where students face high patient loads and limited access to senior clinicians for immediate consultation, AI-LLM GPTs could serve as readily available reference tools to support clinical decision-making during training. However, integration must be carefully structured to enhance rather than replace foundational learning.

Globally, these results are particularly relevant for resource-limited settings where access to comprehensive textbooks, updated references, and expert faculty may be constrained. AI-LLM GPTs could help democratize access to medical knowledge, potentially reducing disparities between well-resourced and under-resourced educational institutions. However, this assumes reliable internet connectivity and technological infrastructure, which remain barriers in many settings.

From a pedagogical perspective, our findings suggest a paradigm shift may be necessary in how we structure medical curricula. Rather than focusing exclusively on memorization of facts—information that AI can rapidly retrieve—educational programs should prioritize teaching students how to (1) formulate effective questions and search strategies, (2) critically evaluate AI-generated responses for accuracy and clinical appropriateness, (3) integrate AI-provided information with clinical context and patient-specific factors, and (4) develop metacognitive skills to recognize the limitations of both their own knowledge and AI tools.

Importantly, although AI-LLMs demonstrated effectiveness for MCQ performance, medical practice requires competencies beyond factual knowledge, including physical examination skills, procedural expertise, empathic patient communication, ethical reasoning under uncertainty, and team-based care coordination. Educational programs must ensure that AI integration enhances rather than diminishes these essential human dimensions of medical practice.

An important limitation is our focus on general-purpose ChatGPT-4o rather than specialized medical or pharmacology-specific AI agents. Domain-enriched LLMs that integrate pharmacology textbooks or medical databases may demonstrate different performance characteristics and potentially offer more accurate, context-specific responses. However, our choice reflects the current reality that most students have access to general LLMs such as ChatGPT-4o rather than specialized medical AI tools and makes the study more generalizable than using domain-enhanced LLMs. Second, the cross-sectional, single-day design prevents assessment of long-term learning outcomes, knowledge retention, or the development of clinical reasoning skills. Similarly, the sequential exposure to different resources (sets A through D) may have introduced order effects or learning transfer between sets. To minimize the learning transfer, the questions have been chosen from diverse topics and moderated to remove any repetition. However, the order effect cannot be removed completely in these types of cross-sectional–sequential studies.

AI will be able to deliver personalized learning and adjust to the needs of students. This individualization helps the learners make their progress through their priorities, repeating difficult concepts or working through material as they become familiar with each subject [19,20]. The AI-powered platforms can monitor student performance in real-time and can track the areas where the students might need more education or practice [21]. Such benefits help achieve the speed and the depth of knowledge acquisition with a higher level of efficiency and more targeted learning [22,23].

To conclude, this study shows that the learned group outperformed the naive group in all sets, including AI-LLM GPT, internet search engine, e-books, and self-knowledge in problem solving. Ironically, the AI-LLM GPT naive group outperformed the problem-solving skills of even the learned group augmented with an internet search engine, showcasing the disruptive potential of AI-LLM in medical education. Further longitudinal studies examining knowledge retention, clinical reasoning development, and the ability to solve novel problems without AI assistance are needed to determine whether AI tools genuinely enhance cognitive capabilities or primarily serve as effective reference tools.