Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study

Introduction

The assessment of learning and competencies is a central pedagogical approach in initial health sciences training programs, whether during the preclinical curriculum, in the context of clinical rotations in the undergraduate program, or in the context of rotations in the postgraduate program [1]. Assessments are used, particularly in the preclinical curriculum, to provide an initial checkpoint for student learning, that is, to verify that students possess adequate knowledge and the necessary skills to enter the clinical curriculum [1]. Among the various assessment formats used in medical education, short-answer questions (SAQs) have emerged as a valuable tool for evaluating student knowledge and clinical reasoning capabilities. Indeed, SAQs constitute a widely recognized assessment format for evaluating the knowledge and cognitive skills of medical students [2-4]. In contrast to multiple-choice questions, which primarily assess the recall of discrete facts and may encourage superficial memorization strategies, SAQs offer the opportunity to evaluate a broader spectrum of competencies, including theoretical understanding, analytic reasoning, and reflective thinking [5]. Furthermore, research in cognitive psychology suggests that the retrieval practice inherent in SAQ formats promotes superior long-term retention compared with recognition-based assessments [6]. This capacity to capture higher-order cognitive processes makes SAQs particularly valuable in the context of competency-based medical education.

However, despite these advantages, the large-scale implementation of SAQs in medical school examinations remains constrained by several factors. Chief among these is the significant administrative and faculty burden imposed on institutions with large student cohorts. Specifically, the time-intensive nature of manual grading represents a significant practical constraint, particularly for large student cohorts. Moreover, human grading is inherently susceptible to variability and potential bias, thereby raising concerns about interrater reliability and the objectivity of assessment outcomes [7]. These limitations have historically restricted the use of SAQs in high-stakes examinations, despite their pedagogical advantages.

Recent advances in AI, particularly the development of large language models (LLMs), offer a promising avenue for addressing these challenges. LLMs have demonstrated the capacity to process and evaluate natural language responses at scale, potentially enabling automated grading of SAQs with accuracy comparable to that of human evaluators. If validated, such systems could reduce the logistical and temporal barriers associated with SAQ assessment, thereby facilitating their broader adoption in high-stakes medical examinations [7-9].

Therefore, in this study we aimed to evaluate the performance, of 6 models of 3 LLMs—ChatGPT (GPT-4o, GPT-4.5, GPT-4o3, and GPT-5), Claude Sonnet 4, and Mistral Pro—in the automated grading of SAQs from second-cycle examinations in hepatology and gastroenterology.

Methods

Study Design and Data Selection

We analyzed retrospective data from the SAQs of second-cycle hepatology and gastroenterology examinations conducted in the 2023 to 2024 and 2024 to 2025 academic years at Sorbonne University Medical School, Paris, France.

The dataset included the questions administered to students, the students’ responses, the grading guidelines provided to evaluators, and the scores awarded to students.

Assessment Procedure

Each student response underwent dual grading: human grading and LLM grading. Human grading was performed twice at 3-month intervals by 3 independent evaluators (2 hepato-gastroenterologist (PAS [evaluator 1] and SS [evaluator 2]), and 1 radiologist (QV [evaluator 3]). The 3 evaluators assigned a score of 0, 0.5, or 1 to each student response, based on the grading guidelines set by the teacher who wrote the question. The final score for human grading corresponded to the majority score assigned by at least 2 of the 3 evaluators from their first evaluation. A median score of 0.5 was assigned to 3 different responses.

LLM grading was performed twice by the 6 selected LLMs: 4 versions of ChatGPT (GPT-4o [July 2025 version], GPT-4.5 [July 2025 version], GPT-4o3 [July 2025 version], and GPT-5 with advanced reasoning [“Thinking”; September 2025 version]; OpenAI), Claude Sonnet 4 (September 2025 version; Anthropic), and Mistral Pro (July 2025 version; Mistral AI). Multiple ChatGPT versions were included to assess performance evolution across model iterations, whereas only the most recent versions of Claude Sonnet 4 and Mistral Pro were tested as these represented the state-of-the-art models available from these providers at the time of the study. All models were queried with default temperature settings unless otherwise specified. For each question, the LLMs received the following information: the question prompt, several examples of correct answers, grading guidelines provided to human evaluators, if available, the list of student responses, and the scoring rubric. Multiple responses from the same SAQ were provided within the same prompt rather than graded individually in separate prompts. To reduce redundancy, identical student responses were included only once. Each LLM assigned a score of 0, 0.5, or 1 to each student response. An example of a typical prompt used is provided in Multimedia Appendix 1.

Ethical Considerations

This retrospective study was conducted in accordance with institutional guidelines and approved by Sorbonne University Medical School. The study was approved by the institutional review board under approval number CRM-2607-587. All examination data were fully anonymized by the university prior to analysis to ensure student confidentiality, and no individual student identifiers were accessible to the research team. As this study involved only the secondary analysis of existing anonymized educational data, individual informed consent was not required.

Statistical Analysis

Data are presented as median (IQR) or number (proportion) as appropriate. Comparisons of quantitative and qualitative variables were made using the Mann-Whitney U test and chi-square or Fisher exact test, respectively.

Comparison of Human Grading

Scores assigned by 3 independent evaluators were compared with each other using quadratic weighted kappa (QWK), Cohen κ, intraclass correlation coefficient (ICC; absolute agreement) and mean absolute error (MAE) to assess consistency among human evaluators. Global interrater agreement among the 3 evaluators was assessed using Fleiss κ. To assess intrarater agreement, each evaluator reannotated all responses after a minimum of a 3-month interval, allowing sufficient time to minimize recall bias. Agreement between the first and second annotations was assessed using QWK, Cohen κ, ICC, and MAE. Interevaluator agreement was scored according to the Landis and Koch criteria [10].

Comparison Between LLMs and Human Grading

The performance of each LLM was compared with reference human grading. For each SAQ, the score assigned by each LLM was compared with the score given by the human grader. Agreement between each LLM and the human grader by majority voting was assessed using QWK, Cohen κ coefficient, ICC, and MAE.

Intramodal Agreement

To assess the stability of LLM grading over time, each model was prompted 2 times on the same set of SAQs. Agreement across repeated inferences from the same model was quantified using QWK, Cohen κ, ICC, and MAE, analogous to intrarater agreement in human evaluators.

Correlation Analysis

A correlation analysis (Spearman rank correlation coefficient) was performed to examine the association between QWK and 2 prompt design variables: the number of examples included in the prompt and the minimum number of words required in the response. The minimum threshold was set with the understanding that a correct response may naturally be longer, depending on the use of synonyms or alternative phrasing.

All tests were 2-sided, and P<.05 was considered significant. Data handling and analysis were performed using Python (version 3.11; Python Software Foundation) and scikit-learn (version 1.7) [11].

Results

Overview

A total of 902 student responses from 12 SAQs across 6 second-cycle hepatology and gastroenterology examinations were analyzed. The median number of student responses per SAQ was 53 (IQR 26-116; range 16-208). The median number of examples provided to the LLMs was 3.5 (IQR 2.0-4.5; range 1-10). The median prompt length (excluding spaces) was 1213 (IQR 954-1262; range: 672-1518) characters. The distribution of grades across models and human evaluators is represented in Figure 1.

‎

Comparison of Human Grading

Interrater agreement was high across all evaluator pairs, with QWK values ranging from 0.90 to 0.92, ICCs from 0.90 to 0.92, and MAEs between 0.05 and 0.07. Pairwise Cohen’s kappa values ranged from 0.81 to 0.87, confirming strong consistency among graders. Intrarater agreement was also robust: evaluator 2 demonstrated almost perfect reliability (QWK and ICC 0.98), while evaluators 1 and 3 achieved substantial to almost perfect agreement (QWK 0.85 and 0.90, respectively) according to Landis and Koch criteria [10] (Table 1 and Figures 2-3).

‎

‎

Comparison Between LLMs and Human Grading

A comprehensive comparison of all metrics across models and human evaluators is presented in Table 2. When compared with the majority vote of human evaluators (Table 2 and Figure 2), the best-performing models were Claude Sonnet 4 (QWK 0.88-0.90; ICC 0.88-0.90; MAE 0.06-0.07) and GPT-4o3 (QWK 0.82-0.89; ICC 0.82-0.89; MAE 0.07-0.09), both achieving almost perfect agreement according to the Landis and Koch criteria. These values approached those observed between human evaluator pairs (QWK 0.90-0.92; ICC 0.90-0.92; MAE 0.05-0.07), indicating near-human–level performance. GPT-5 demonstrated substantial agreement (QWK 0.77-0.79; ICC 0.77-0.79; MAE 0.12-0.13), though lower than any human evaluator pair. Mistral Pro showed moderate alignment (QWK 0.56-0.63; ICC 0.56-0.63; MAE 0.22-0.25), while older versions of ChatGPT (GPT-4o and GPT-4.5) exhibited fair to moderate agreement (QWK <0.55 and MAE >0.25) clearly below human consistency levels.

In terms of reproducibility (Table 3 and Figure 3), intramodel consistency was the highest for Claude Sonnet 4 (QWK 0.93; ICC 0.93; MAE 0.04), approaching but not exceeding the best human intrarater agreement (evaluator 2: QWK 0.98; ICC 0.98; MAE 0.009) and surpassing evaluators 1 and 3 (QWK 0.85-0.90; ICC 0.85-0.90; MAE 0.07-0.08). GPT-4o3 and GPT-5 also demonstrated almost perfect stability (QWK 0.85; ICC 0.85; MAE≈0.08), comparable to human evaluators 1 and 3. By contrast, Mistral Pro showed fair intramodel reliability (QWK 0.36; ICC 0.36; MAE 0.28), substantially below any human evaluator’s test-retest performance.

Question-level analysis revealed a strong correlation between interhuman agreement and LLM performance. Among the 5 SAQs with the lowest interhuman agreement, 4 also yielded the poorest performance for GPT-4o3 and Claude Sonnet 4. These questions were neither those with the greatest diversity of student responses nor those requiring the longest average response length (number of words).

Correlation Analysis: Association Between Grading Agreement (QWK) and Prompt Design

Correlations between QWK and the number of examples provided were generally weak and not statistically significant across all models and human evaluators (Figure 4 and Table 4). Most coefficients were close to 0, with wide CIs spanning both negative and positive values. The strongest positive trends were observed for the consensus score of the 3 best-performing LLMs (“LLM (top3)” (ρ [rho]=0.56, 95% CI −0.09 to 0.92; P=.09) and Claude Sonnet 4 at first annotation (“Claude Sonnet 4 (1)” (ρ [rho]=0.51, 95% CI −0.18 to 0.90; P=.13), but neither reached statistical significance.

Correlations between QWK and the minimum number of words required for a full score were also low overall (Table 4). No consistent or significant association was observed across models or evaluators. The strongest negative trend appeared for Mistral Pro at first annotation (“Mistral Pro (1)” ρ [rho]=−0.59, 95% CI −0.89 to 0.02; P=.07), suggesting a possible inverse relationship, though this did not achieve statistical significance. Human evaluators showed comparable weak correlations, with values ranging from −0.16 to 0.50 and wide CIs.

‎

Discussion

In our study, Claude Sonnet 4 and GPT-4o3 showed the closest alignment with human grading, with stable reproducibility across repeated inferences. None of the models surpassed human evaluators in either agreement or reproducibility; the best systems reached human‐level ranges of agreement and reproducibility. Correlation analyses did not reveal clear associations between the prompt design features examined and model performance. These findings suggest that LLM-based grading systems hold promise for enhancing the scalability, efficiency, and potential objectivity of medical education assessment. However, it should be noted that human and LLM biases differ fundamentally: while human graders may be affected by fatigue, familiarity with students, or inconsistent rubric interpretation, LLMs may exhibit systematic biases inherited from pretraining data, sensitivity to prompt framing, or tendencies toward certain response patterns—none of which are yet fully characterized in medical education assessment contexts.

Our results are consistent with existing data. Several studies suggest that LLMs can approximate, but not consistently exceed, human grading in medical education. Seneviratne and Manathunga [12] reported a strong correlation between an AI-assisted SAQ scoring tool and human examiner markings, supporting the feasibility of automated SAQ scoring when aligned with examiner rubrics. Similarly, Rajan et al [13] reported that GPT-4 scoring could be statistically equivalent to faculty grading at the aggregate level, while discrepancies persisted on items requiring more complex reasoning, supporting the feasibility of LLM-assisted scoring without establishing validity across all unseen assessment contexts. Beyond SAQs, Burke et al [14] showed that ChatGPT could score structured free-text clinical notes with fewer scoring errors than human graders, suggesting that LLMs may perform particularly well when evaluation criteria are explicit and fact-based.

At the same time, published findings remain heterogeneous regarding the LLM grading performance in medical education. For example, Grevisse [7] observed moderate agreement between GPT-4 or Gemini and SAQ instructors and noted that GPT-4 tended to be conservative in awarding full credit, highlighting the need for human oversight. Similarly, Bolgova et al [15] showed that expert-expert agreement remained higher than LLM-expert agreement, with performance varying across questions and depending on rubric structure, highlighting the value of expert-expert agreement for identifying settings in which human oversight remains important. In a medical education setting on mechanical ventilation, ChatGPT demonstrated weak agreement with human graders, and the authors cautioned against using LLMs as sole graders in high-stakes contexts [16].

Emerging evidence also suggests that agreement between LLMs and faculty may not be uniform across question types: in a case-based short-answer exam, LLM grading was comparable to faculty overall, but discrepancies increased for harder questions and for higher Bloom’s levels (eg, understanding and evaluating), implying that LLM performance may depend on the cognitive processes targeted by each SAQ rather than reflecting a stable “global” grading ability [13]. In other domains, GPT-4 achieved near-human consistency for lower-stakes short-answer scoring in K-12 education, reinforcing the idea that task difficulty, stakes, and rubric clarity likely influence observed agreement [17].

In our dataset, some questions yielded lower interrater consistency among human graders than among certain LLMs. Rather than indicating that humans “underperformed,” this pattern more plausibly suggests that a subset of SAQs may have been insufficiently precise or supported by incomplete grading guidance, increasing interpretive latitude. This interpretation aligns with works showing that question framing and prompt design can strongly influence LLM outputs [18,19]. Clearer and more specific prompts consistently yield more accurate and relevant outputs because prompt engineering acts as a “steering mechanism” that guides the model’s reasoning. A notable example is chain-of-thought prompting: simply adding a phrase like “Let’s think step by step” can encourage systematic reasoning and dramatically improve accuracy on complex tasks; for instance, boosting GPT-3’s performance on math word problems from about 17% to 78% [20]. Numerous studies likewise show that prompts providing context, explicit instructions, or stepwise guidance enhance LLM outputs across domains, including education, where well-designed prompts have been found to significantly improve the usefulness of responses [21]. Together, these findings emphasize that observed LLM-human agreement is not solely a property of the model but also of assessment and implementation design, including question construction, rubric specificity, and prompting strategy (instruction clarity, context provided, and scoring constraints).

This study has several limitations. First, the limited sample size (12 SAQs from a single institution and specialty) constrained statistical power, generalizability, and the robustness of question-level exploratory analyses. Moreover, strong performance on this specific dataset does not guarantee correct assessment in other medical domains, as LLM outputs depend not only on pretraining data but also on rubric quality, question framing, and contextual factors specific to each assessment setting. Second, grading guidelines were not uniformly available, which may have introduced variability in both human and model scoring. Third, multiple student responses were presented within a single prompt for each SAQ rather than being graded in separate prompts. Although identical responses were included only once to reduce redundancy, potential effects of context or order cannot be excluded. Fourth, we evaluated specific LLM versions (July-September 2025) using default parameters without optimization; rapidly evolving models may demonstrate different capabilities. Specifically, LLMs were not trained using previous questions to improve scoring performance. Fifth, practical implementation challenges, including cost, latency, privacy, and student acceptance, were not examined. Finally, our analysis focused on binary and partial-credit scoring for SAQs; performance may differ for other assessment formats (eg, extended responses, oral exams, and objective structured clinical examination [OSCE] narratives) and more granular scales. In addition, although the majority vote of human graders served as a practical reference standard, high agreement with this standard and strong reproducibility should not be interpreted as equivalent to educational validity, particularly in high-stakes assessment contexts. Multicenter studies across specialties, languages, and SAQ formats are needed to better characterize reliability, reproducibility, and external validity in real-world settings.

In conclusion, Claude Sonnet 4 and GPT-4o3 demonstrated the strongest alignment with human grading in our second-cycle medical education SAQ context, potentially enabling more scalable use of SAQs while preserving near-human consistency. However, models have not yet surpassed human evaluators in reproducibility or accuracy across all question types, and performance appears sensitive to rubric quality and cognitive complexity. Approaches such as retrieval-augmented generation and parameter-efficient fine-tuning may further improve task-specific reliability. At present, LLMs should be considered supportive tools integrated into grading workflows with transparent validation and ongoing monitoring, rather than autonomous replacements for faculty graders, particularly for high-stakes assessments where interpretability, fairness, and educational validity are essential.