TY  - JOUR
AU  - Doru, Berin
AU  - Maier, Christoph
AU  - Busse, Johanna Sophie
AU  - Lücke, Thomas
AU  - Schönhoff, Judith
AU  - Enax- Krumova, Elena
AU  - Hessler, Steffen
AU  - Berger, Maria
AU  - Tokic, Marianne
PY  - 2025
DA  - 2025/3/3
TI  - Detecting Artificial Intelligence–Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled Study
JO  - JMIR Med Educ
SP  - e62779
VL  - 11
KW  - artificial intelligence
KW  - ChatGPT
KW  - large language models
KW  - textual analysis
KW  - writing style
KW  - AI
KW  - chatbot
KW  - LLMs
KW  - detection
KW  - authorship
KW  - medical student
KW  - linguistic quality
KW  - decision-making
KW  - logical coherence
AB  - Background: Large language models, exemplified by ChatGPT, have reached a level of sophistication that makes distinguishing between human- and artificial intelligence (AI)–generated texts increasingly challenging. This has raised concerns in academia, particularly in medicine, where the accuracy and authenticity of written work are paramount. Objective: This semirandomized controlled study aims to examine the ability of 2 blinded expert groups with different levels of content familiarity—medical professionals and humanities scholars with expertise in textual analysis—to distinguish between longer scientific texts in German written by medical students and those generated by ChatGPT. Additionally, the study sought to analyze the reasoning behind their identification choices, particularly the role of content familiarity and linguistic features. Methods: Between May and August 2023, a total of 35 experts (medical: n=22; humanities: n=13) were each presented with 2 pairs of texts on different medical topics. Each pair had similar content and structure: 1 text was written by a medical student, and the other was generated by ChatGPT (version 3.5, March 2023). Experts were asked to identify the AI-generated text and justify their choice. These justifications were analyzed through a multistage, interdisciplinary qualitative analysis to identify relevant textual features. Before unblinding, experts rated each text on 6 characteristics: linguistic fluency and spelling/grammatical accuracy, scientific quality, logical coherence, expression of knowledge limitations, formulation of future research questions, and citation quality. Univariate tests and multivariate logistic regression analyses were used to examine associations between participants’ characteristics, their stated reasons for author identification, and the likelihood of correctly determining a text’s authorship. Results: Overall, in 48 out of 69 (70%) decision rounds, participants accurately identified the AI-generated texts, with minimal difference between groups (medical: 31/43, 72%; humanities: 17/26, 65%; odds ratio [OR] 1.37, 95% CI 0.5-3.9). While content errors had little impact on identification accuracy, stylistic features—particularly redundancy (OR 6.90, 95% CI 1.01-47.1), repetition (OR 8.05, 95% CI 1.25-51.7), and thread/coherence (OR 6.62, 95% CI 1.25-35.2)—played a crucial role in participants’ decisions to identify a text as AI-generated. Conclusions: The findings suggest that both medical and humanities experts were able to identify ChatGPT-generated texts in medical contexts, with their decisions largely based on linguistic attributes. The accuracy of identification appears to be independent of experts’ familiarity with the text content. As the decision-making process primarily relies on linguistic attributes—such as stylistic features and text coherence—further quasi-experimental studies using texts from other academic disciplines should be conducted to determine whether instructions based on these features can enhance lecturers’ ability to distinguish between student-authored and AI-generated work. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2025/1/e62779
UR  - https://doi.org/10.2196/62779
UR  - http://www.ncbi.nlm.nih.gov/pubmed/40053752
DO  - 10.2196/62779
ID  - info:doi/10.2196/62779
ER  -