This cross-sectional observational study was conducted during the autumn semester of 2025 at Guangzhou Medical University with 22 fourth-year undergraduate students majoring in basic medical sciences, who participated as part of their endocrinology curriculum. The cohort comprised 15 female and 7 male students. Each student interacted with all 8 AI teaching agents sequentially over the course of the semester. Owing to individual absences, the number of completed dialogues per agent ranged from 19 to 22, yielding 167 valid dialogues in total.

All participants were informed of the study purpose, and the study was approved by the institutional review board of Guangzhou Medical University (202607001).

Eight AI teaching agents were constructed and deployed on the Chaoxing e-Learning platform (Beijing Century Chaoxing Information Technology Development Co, Ltd), a widely used commercial platform in Chinese higher education. Each agent covered 1 chapter of the endocrinology curriculum. Two chapters addressed foundational content (a general introduction and organ morphology), and 6 addressed clinical or disease-oriented content (hypothalamic-pituitary diseases, thyroid diseases, adrenal diseases, glucose metabolism disorders, lipid metabolism disorders, and calcium-phosphorus metabolism disorders).

Each agent employed a distinct role-play scenario assigned to 1 of 4 role paradigms: patient (A1, A4, A5), student (A2, A3), expert (A6, A7), and family member (A8); full configurations are provided in Table 1. The 4 paradigms were chosen to represent 4 prototypical interaction scenarios in clinical medicine (doctor and patient, teacher and student, expert consultation, and family communication), thereby sampling a range of communicative and pedagogical demands rather than instantiating a single learning theory a priori. Where relevant, the resulting behaviors could be related post hoc to established educational concepts; for example, the student paradigm elicits scaffolding within the learner’s zone of proximal development [20,21]. We emphasize that because each agent combined 1 role paradigm with 1 content chapter, role and content were confounded by design.

Key implementation differences among the agents lay in the assigned AI persona and the reciprocal role assigned to the student, which together determined the communicative register and the pedagogical stance of each dialogue (Table 1). For example, within the expert paradigm, agent A6 positioned the student as a learner in a mentorship model, whereas agent A7 positioned the student as a patient, providing both a physician and a patient perspective. Agent A8 was deliberately revised after the initial deployment of the first 7 agents, when their empathic engagement was judged to be limited; only its prompt was modified, while all other settings were held constant, to strengthen an empathic family-member persona. For each agent, the research team specified the instructional content, learning objectives, knowledge sources, role-play scenario, and safety and behavioral constraints; the complete per-agent configuration, including prompts, is provided in Multimedia Appendix 1.

Two types of data were collected from each student-agent dialogue. First, the platform’s built-in scoring system generated a single aggregate score per dialogue as an automated assessment of student performance. The platform does not disclose how these scores are generated, and its scoring mechanism could not be independently obtained or verified; only aggregate scores were available to instructors, with no dimensional breakdown. Second, complete dialogue transcripts were exported from the platform for independent external evaluation.

To enable standardized cross-agent comparison of teaching quality, we developed an 8-dimension evaluation rubric (Table 2) informed by existing frameworks for AI conversational agents in health care [14], AI agent evaluation in clinical medicine [16], and educational agent effectiveness research [17], supplemented by expert discussion within the research team. Unlike platform-generated scoring, the rubric was designed to assess agent teaching quality rather than student response quality. The complete scoring prompt is provided in Multimedia Appendix 2.

Dimension weights were assigned by the research team on conceptual grounds: medical knowledge accuracy and pedagogical guidance, as the core teaching functions, received the highest weight (20 points each), whereas student engagement elicitation and feedback quality, as supportive dimensions, received the lowest (5 points each). We note as a limitation that these weights were set without formal student or educator input, and that empirically derived weights are a direction for future work.

All 167 dialogue transcripts were scored across the 8 dimensions using LLMs operating under standardized instructions [15]. To assess the robustness and reliability of the scoring, 3 LLMs from different developers were used as evaluators: Claude Opus 4.8 (Anthropic), representing a leading international model, and DeepSeek (DeepSeek V4 Artificial Intelligence Basic Technology Research Co, Ltd) and Qwen 2.5 (Alibaba Cloud), two widely used Chinese LLMs selected for their strong Chinese-language capability given that all dialogues used Chinese. DeepSeek and Qwen were added at the recommendation of a reviewer and on the basis of our recent study of mainstream domestic and international LLMs for medical assessment tasks [22].

All scoring was performed programmatically through each model’s application programming interface under a fixed request configuration, with an intercall interval of 1.0 second and up to 3 retries per call. To promote deterministic, reproducible outputs, DeepSeek and Qwen were queried with temperatures of 0 and top-p of 0.01; Claude Opus 4.8 was queried at its default setting. Each model independently applied the identical 8-dimension rubric and scored every dialogue 4 times, and the mean of the 4 runs was used as that model’s final score. This repeated-scoring design followed directly from our recent finding that the reliability of LLM-based scoring is better captured by repeated runs and cross-run variance than by single-run performance [22]. Within-model reproducibility across the 4 runs was quantified by the SD of the repeated scores; reproducibility was high, indicating that scores were stable rather than single, unrepeatable judgments [23].

To calibrate the AI-based scoring against human judgment, a medical-education expert independently rescored a stratified subset of 40 dialogues using the identical 8-dimension rubric. The subset was stratified by Claude total score to span high, medium, and low quality and to cover all 8 agents, and the expert was blinded to the AI scores. Agreement between AI and expert ratings was quantified by the Pearson correlation and the intraclass correlation coefficient (ICC), computed at the total-score level and per dimension [24]. Because rescoring was performed by a single expert, expert-to-expert reliability could not be computed; this is addressed in the Limitations section.

Given nonnormal score distributions and heterogeneous variances across agents (Shapiro-Wilk and Levene tests), nonparametric methods were used throughout. Platform-generated scores were compared across agents using the Kruskal-Wallis H test, with post hoc Dunn tests and Bonferroni correction for pairwise comparisons. Effect sizes were reported as η² for omnibus tests and Cohen d for selected contrasts. Between-agent variation in each rubric dimension was summarized using the coefficient of variation (CV).

Comparisons across role paradigms were treated as descriptive, because the independent unit was the agent and each role group comprised only 1 to 3 agents; formal inferential testing was therefore not performed at the agent level. The divergence between platform-based and rubric-based agent rankings was characterized through concrete rank comparisons rather than a single agent-level correlation coefficient, which would be unstable at this sample size. Learner gender differences were examined across dimensions using Mann-Whitney U tests, with results interpreted in light of the small and imbalanced sample. The correspondence between rubric-based agent quality and students’ end-of-course chapter examination performance was examined descriptively. All analyses were conducted in Python (version 3.11), with a 2-sided significance threshold of P<.05.

Chaoxing platform–generated scores from 167 student-agent dialogues differed significantly across the 8 agents (Kruskal-Wallis H₇=57.981; P<.001; η²=0.321; Figure 1), with 9 significant pairwise differences identified by post hoc Dunn tests (Bonferroni-corrected P<.05). Basic science agents (A2, A3) received the highest scores, while clinical disease agents (A4, A5, A7) received the lowest. While these differences indicate that agent design influenced student performance under platform scoring, the platform does not disclose how its scores are generated, and its scoring mechanism could not be independently verified, precluding any interpretation of what drove them or whether they reflect genuine agent teaching quality. Therefore, the 8-dimension rubric was applied as an independent external audit, reported in the following sections.

To evaluate the robustness of the rubric-based scoring, the 8-dimension rubric was independently applied by 3 LLM evaluators from different developers (Claude, DeepSeek, and Qwen), each scoring all 167 dialogues 4 times, with high within-model reproducibility (Table 3; mean within-model SD≤0.36).

The 3 models agreed broadly on the relative ranking of agents (Spearman ρ=0.64 and 0.71 for Claude vs DeepSeek and Qwen) but differed substantially in discrimination. DeepSeek and Qwen showed pronounced leniency, scoring large proportions of dialogues at the ceiling (Figure 2; Qwen: above 90% on medical safety boundary, student engagement elicitation, and assessment and feedback quality; DeepSeek: 60-77% on the same dimensions). This compressed between-agent differences, yielding narrow score ranges (82‐95 and 75‐100) and low discrimination (SD of agent total scores was 4.0 and 7.3). Claude showed a minimal ceiling effects except on the two 5-point dimensions (student engagement elicitation, assessment and feedback quality), with the widest range (61-92) and greatest discrimination (SD 11.6).

Because the analysis aimed to resolve differences in teaching quality among agents, an evaluator able to discriminate between them was required; Claude was therefore used as the primary evaluator, with its agreement against human expert judgment examined separately. The leniency of the other evaluators is itself a relevant finding, indicating that ceiling effects should be anticipated when LLMs serve as evaluators of educational quality.

Applying the 8-dimension rubric, agent total scores ranged widely (60.6‐91.6; Figure 3), with A6 and A3 scoring highest and A7 and A8 lowest. This variation was not uniform across dimensions. It was largest on the 2 knowledge-related dimensions, namely knowledge-point coverage (CV=27.3%) and medical knowledge accuracy (CV=18.5%), indicating that agents differed most in what and how accurately they taught. In contrast, role-play quality was the most uniform dimension (CV=5.7%), with all agents sustaining their assigned personas comparably well. Adaptive difficulty control was low across nearly all agents, emerging as a shared weakness rather than a source of between-agent variation. Thus, the agents were broadly comparable in role enactment but diverged mainly in the knowledge content they delivered.

Grouping agents by role paradigm revealed distinct dimension profiles (Figure 4). At the group level, student-type agents showed the strongest pedagogical guidance (group-mean pedagogical guidance ability was 17.7), whereas expert-type agents showed the lowest medical safety (group-mean medical safety boundary was 6.7). The family-member agent illustrated a notable dissociation between empathy and knowledge: it attained one of the highest role-play scores (role-play quality=14.1) yet the lowest knowledge-point coverage of all agents (knowledge-point coverage efficiency=5.3) and the lowest total score (60.6). Notably, this agent had been deliberately revised after the first 7 agents showed generally limited empathic engagement in student use; only its prompt was adjusted—while all other settings were held constant—to strengthen the empathic, family-member persona. Despite this targeted enhancement of empathy, it remained weakest in knowledge coverage, indicating that strengthening empathy did not translate into broader knowledge delivery. How to balance affective engagement and knowledge coverage within a single agent thus remains an open question. Because each role group comprised only 2 to 3 agents, these between-group comparisons are descriptive; formal inferential testing was not performed at the agent level.

Male (n=7) and female (n=15) students were compared across all 8 dimensions using Mann-Whitney U tests. No significant difference was detected in any dimension or in total score (79.1 vs 80.1; P=.82; all dimension-level P>.05; Cohen d=0.07 for total score). Given the small, imbalanced sample and the very small observed effect size, this analysis was substantially underpowered; the absence of a detectable difference should therefore not be interpreted as evidence of gender-equitable delivery, as a type II error cannot be excluded.

To validate the rubric-based AI scoring, 40 dialogues spanning all 8 agents and the full quality range were independently rescored by a medical-education expert using an identical rubric, blinded to the AI scores (Figure 5A). AI and expert ratings were closely aligned in magnitude (mean totals 79.1 vs 78.5) and were positively correlated at the total-score level (r=0.61; ICC=0.51). Because the two derive from a highly consistent automated rater and a single subjective human expert, close but not exact agreement was expected; the aim was to confirm convergence on the same construct, not identity.

Agreement was strongest on the cognitive-process dimensions, ranging from moderate for knowledge accuracy to highest for adaptive difficulty control. For role-play quality and the two 5-point dimensions, the ICC was near zero or negative, but this reflects a ceiling effect rather than rater disagreement: both AI and expert rated these dimensions uniformly high (for role-play quality, 13.4 vs 13.3 out of 15), leaving minimal between-agent variance (Figure 5B). Because the ICC indexes relative discrimination, it is driven toward zero when there is little variance to rank, and its low value here does not indicate that the raters disagreed; on the contrary, they consistently converged on high scores [24]. The moderate agreement on knowledge accuracy likewise does not undermine the study’s key knowledge-related finding, which rests on a rank-level difference too large to depend on rating precision: A8 scored markedly lowest on knowledge coverage under both AI and expert scoring.

Agent rankings from platform scores and from the 8-dimension rubric diverged substantially (Figure 6A). Two agents illustrate this most clearly: A8 ranked third by platform score but last by rubric quality, and A6 rose from fourth to first. Rather than indicating that either measure is invalid, this divergence reflects that the two capture different constructs—platform scores index student performance during the interaction, whereas the rubric indexes the teaching quality of the agent itself. The two are therefore complementary rather than interchangeable.

There was a similar lack of association between rubric quality and end-of-course chapter examination performance (Figure 6B): higher agent quality did not correspond to higher chapter examination scores (eg, A3 showed high quality but the lowest examination score rate, whereas A7 and A8 showed low overall quality but high examination scores). This absence of a systematic association should be interpreted cautiously, given the small number of agents, the near-ceiling examination scores, and the lack of individual baseline data; it indicates only that agent teaching quality and this particular examination did not track one another, not that better teaching is ineffective.