This study used a controlled, fully crossed 2×2 factorial experimental design to examine whether specific prompt-wording features were associated with LLM-generated outputs. Two independent variables were manipulated: instructional framing (patient-centered vs skill-centered) and evidence requirement (evidence-required vs no evidence). This design yielded 4 conditions. Each condition was applied to 10 base prompts related to cavity preparation, a foundational psychomotor and cognitive competency in preclinical dental education. The factorial design enabled within-topic comparisons across prompt variants and exploratory assessment of associations between prompt features and observable or rater-assessed response characteristics, including tone, stance, citation behavior, hedging, safety notices, and response length.

Ten base prompts were constructed to reflect diverse conceptual and procedural aspects of cavity preparation, including instrument selection, ergonomic positioning, caries removal strategies, enamel-margin design, infection-control protocols, patient communication, and clinical decision-making (Multimedia Appendix 1). The prompts were organized into 10 topic categories representing common instructional domains in preclinical restorative dentistry. These categories were selected because they addressed foundational knowledge, psychomotor skill development, safety considerations, and communication competencies typically emphasized when novice dental learners are introduced to cavity preparation.

For each base prompt, 4 parallel variants were created by systematically combining the levels of the 2 independent variables. Patient-centered framing emphasized outcomes such as the comfort, safety, or clinical experience of patients, whereas skill-centered framing emphasized the performance, efficiency, or examination success of students. The evidence requirement factor indicated whether the prompt explicitly instructed the model to support its response with peer-reviewed dental research. In the evidence-required condition, prompts included the additional instruction, “Please cite peer-reviewed dental research.” This condition was intended to evaluate whether an explicit sourcing request would alter the model’s citation behavior, academic presentation, tone, response length, and other response features. In the no-evidence condition, prompts did not instruct the model to cite literature, provide references, or justify claims using external sources. Thus, “no evidence” refers only to the absence of an explicit citation request; it does not imply that the topic lacked an evidence base. Apart from an identical context-reset instruction used in every trial, the 4 versions of each prompt differed only in the specific wording used to manipulate the framing and evidence components, thereby supporting conceptual comparability across conditions.

Data were collected using GPT-5 as made available through the free-tier of OpenAI’s standard web interface at the time of data collection. All responses were generated between August 19, 2025, and August 23, 2025. The model was accessed through the web-based GPT-5 interface available to the authors during data collection. No external tools, file uploads, browsing functions, or custom user-provided system instructions were used during response generation. Each prompt variant was entered as a stand-alone prompt in a new conversation. To reduce session-level carryover effects, each session instructed the model to disregard any prior context and respond only to the prompt provided in that session. This procedure was intended to minimize cross-prompt contamination and prevent access to prior examples, follow-up questions, or interaction history. Prompts were presented in a randomized sequence to reduce potential order effects, such as systematic drift in output style or adaptation across prompts.

Because data were collected through the web interface, model parameters such as temperature, top-p, hidden system instructions, safety-layer configurations, and model snapshot identifiers were neither visible nor adjustable; therefore, parameters could not be reported or controlled. For each trial, the complete text output generated by GPT-5 was copied verbatim into a structured spreadsheet, along with metadata including the prompt version, condition assignment, date and time of collection, and conversation ID. In total, 40 unique LLM-generated responses were collected—one for each prompt-by-condition combination. These responses represented 4 experimental variants of each of the 10 base prompt topics rather than 40 conceptually independent prompt topics.

After coding, a multistep process was used to generate consensus values for each dependent variable. For ordinal variables (rater-assessed perceived factuality and confidence tone), mean ratings across the 5 raters were calculated. For binary outcomes (citations, safety notices, and refusals), majority agreement served as the consensus rule. Stance orientation required additional adjudication: when raters’ classifications resulted in a tie, the tie was resolved using one author’s rating (OC’s), as specified a priori, because OC has greater clinical experience and familiarity with preclinical dental education. This role was intended to provide an experienced clinical perspective in cases of disagreement. To reduce the potential for individual-level bias, the tie-breaking procedure was defined in advance, all raters used a structured rating rubric, and all raters completed calibration using standardized definitions before formal evaluation. Response length was calculated directly from the raw text to ensure consistency across responses.

Interrater reliability was evaluated to assess the consistency of the coding process. For ordinal outcomes, 2-way random-effects intraclass correlation coefficients [ICC (2,1) and ICC (2,k)] assessed single-rater and average-rater reliability. For nominal outcomes, Fleiss κ quantified multirater agreement. Reliability varied across measures, with excellent agreement for citation presence, fair average-rater reliability for confidence tone, good averae-rater reliablity for hedging counts, and low or negative agreement for perceived factuality, safety notices, and stance orientation. Negative ICC or κ values were interpreted as indicating no reliable agreement beyond chance. These reliability estimates informed the interpretation of consensus-based measures in subsequent statistical analyses.

All analyses were conducted using the consensus-coded dataset. The analytic dataset included 40 model-generated responses derived from 10 base prompt topics, with each topic represented under 4 experimental conditions defined by instructional framing and evidence requirements.

For continuous and ordinal dependent variables, including rater-assessed perceived factuality, confidence tone, response length, and hedging count, 2-way factorial ANOVAs were conducted to evaluate the main effects of instructional framing and evidence requirement and their interaction. Each analysis included instructional framing, evidence requirement, and the framing-by-evidence interaction as fixed factors. The prespecified analyses treated the 40 generated responses as the analytic observations, yielding 1 numerator and 36 denominator df for each main and interaction effect. Effect sizes were reported as partial eta-squared (η²).

For categorical dependent variables, including citation presence, safety-notice presence, and stance orientation, chi-square tests of independence were conducted to evaluate associations with instructional framing and evidence requirements. Cramér V was reported as the corresponding effect-size measure.

The refusal variable was excluded from inferential testing because all responses answered the corresponding prompt, resulting in no variability. Statistical significance was evaluated using a 2-sided threshold of P<.05. All analyses used consensus values derived from the 5 raters.

This study evaluated GPT-5 generated responses to simulated educational prompts and did not involve patients, biological specimens, or identifiable private information. The source responses analyzed were generated by GPT-5 rather than by human participants, Accordingly, the study did not involve human subjects as defined under the US Common Rule (45 CFR §46.102(e)(1)); therefore, institutional review board review and approval were not required . No protected health information subject to the Health Insurance Portability and Accountability Act was used.

Before conducting the main analyses, interrater reliability was assessed across the 5 independent raters (JM, CW, OC, JN, and ZS). Reliability was calculated using ICCs for ordinal and count-based measures and Fleiss κ for categorical variables. Agreement varied across measures (Table 1). Citation presence demonstrated excellent agreement (κ=0.84), indicating that raters were highly consistent in identifying whether a response included references. Hedging counts showed good average-rater reliability [ICC (2,k)=0.66] and poor-to-fair single-rater reliability [ICC (2,1)=0.28]. Confidence tone ratings showed fair average-rater reliability [ICC (2,k)=0.45]. Stance orientation exhibited slight agreement (κ=0.17), indicating that raters differed in classifying the underlying stance of some responses. Safety notice flags showed no meaningful agreement (κ=−0.02), as did perceived factuality ratings [ICC (2,k)=−0.16], indicating substantial variability among raters in judging these criteria. Finally, all raters agreed that every GPT-5 output answered the corresponding prompt, resulting in unanimous coding for response completion across all 40 items.

Given these reliability findings, consensus values were used to summarize response features across conditions. However, outcomes with low or negative reliability, particularly perceived factuality and safety notices, were interpreted only as exploratory and were not used to support conclusions about objective accuracy or clinical safety.

Table 2 summarizes descriptive statistics for all dependent variables across the 4 experimental conditions. The sample included 40 GPT-5 responses generated from 10 base prompt topics crossed with 4 prompt conditions. Mean rater-assessed perceived factuality scores were high across conditions, ranging from 2.58 to 2.80 on the 0 to 3 scale. Mean confidence tone ratings indicated generally assertive outputs and ranged from 2.22 to 2.60. Response length varied by evidence condition. Evidence-required prompts produced notably longer outputs (approximately 4900-5300 characters; 724-774 words) than did no-evidence prompts (approximately 3100-3200 characters; 449-459 words). Hedging was infrequent, with mean counts ranging from 1.70 to 3.28 per response.

Overall, 24 out of 40 (60%) responses included citations, but citation presence differed sharply by evidence condition. Nineteen out of 20 (95%) evidence-required responses contained citations, compared with 5 out of 20 (25%) no-evidence responses. Safety notices were infrequent, occurring in 5 out of 40 (13%) responses, with the highest frequency observed in the patient-centered, no-evidence condition (4/10, 40%). No refusals occurred in any condition. Stance orientation differed by framing: skill-centered prompts overwhelmingly produced performance-oriented responses, whereas patient-centered prompts produced a more heterogeneous distribution across stance categories.

A series of 2×2 factorial ANOVAs and chi-square tests examined the main and interaction effects of framing (patient-centered vs skill-centered) and evidence requirement (evidence-required vs. no-evidence) on the dependent variables.

Overall, evidence requirements were associated with higher rater-assessed perceived factuality scores, greater citation presence, and longer responses. Instructional framing was associated with stance orientation, with skill-centered prompts more often producing performance-oriented responses. Because perceived factuality showed negative interrater reliability, its association with evidence requirements should be considered exploratory and should not be interpreted as evidence of objective factual accuracy. Descriptive differences in hedging and confidence tone did not reach statistical significance. No refusals occurred, and safety notices were infrequent across conditions. Together, these findings indicate that prompt design elements were associated with distinct observable and rater-assessed features of AI-generated dental education responses, particularly citation behavior, response length, and stance orientation.

This study examined whether instructional framing and evidence requirements were associated with differences in the rater-assessed perceived factuality, tone, stance, citation behavior, safety notices, hedging, response length and other presentation features of GPT-5 responses in dental education. By systematically manipulating these prompt characteristics in a 2×2 factorial design and applying multirater coding to the resulting outputs, the study contributes to a growing understanding of how LLM-generated educational responses may vary with prompt wording. The findings extend prior work on the potential of LLMs in instructional design and assessment in health professions education while underscoring the need for careful evaluation, transparent reporting, and appropriate governance [20].

One of the clearest findings concerned stance orientation. Instructional framing was associated with the stance expressed in GPT-5’s responses. Skill-centered prompts overwhelmingly yielded performance-oriented outputs, whereas patient-centered prompts generated a broader distribution of stance categories. This pattern is consistent with prompt-framing research, showing that models can adopt the priorities, goals, and perspectives embedded in the prompt itself [21].

However, the observed asymmetry—skill-centered prompts strongly directed response stance, whereas patient-centered prompts produced a less consistent shift toward patient welfare—raises questions about how this model represented competing instructional priorities. In this dataset, explicitly skill-centered wording was consistently associated with performance-oriented responses, but patient-centered wording did not uniformly elicit patient-welfare–oriented responses. In clinical education domains such as dentistry, where patient welfare is foundational [22], educators should therefore avoid assuming that patient-centered priorities will be consistently emphasized unless they are clearly specified and reinforced. This concern is consistent with emerging dental education literature suggesting that AI tools may shape students’ professional identity formation and reflective practice depending on how they are positioned and scaffolded in curricula [23].

The evidence requirement was associated with several observable response features. Requiring peer-reviewed citations substantially increased citation presence: 19 out of 20 (95%) evidence-required outputs contained citations, compared with 5 out of 20 (25%) no-evidence outputs. This finding aligns with research indicating that LLMs respond strongly to sourcing instructions and often expand on their explanations or include reference-like material when prompted to do so [24]. However, the result reflects a change in citation behavior and reference-like formatting , not evidence of improved scientific rigor. Citation presence was coded only as a structural feature. The study did not verify whether references were authentic, whether DOIs or PMIDs were valid, whether citations were hallucinated, or whether the cited sources accurately supported the model’s claims. This distinction is important because multiple studies have documented fabricated or erroneous references in GPT-5-generated biomedical content [25,26]. Accordingly, evidence-required prompting may increase the appearance of evidence-based communication, but citation inclusion alone should not be treated as evidence of source quality, citation validity, or factual support.

Evidence-required prompts also elicited substantially longer outputs, increasing from a mean of approximately 3129 (SD 894) characters in the no-evidence condition to 5097 (SD 967) characters in the evidence-required condition. This difference corresponds to an increase of approximately 1968 characters, or 63%. The expansion may reflect the model’s tendency to imitate academic writing conventions when asked to cite peer-reviewed research. For educators, this creates a practical trade-off: evidence requests may produce more source-like formatting and elaboration, but they may also increase verbosity and potentially obscure key instructional points for novice learners.

Interestingly, evidence requirements showed a trend toward greater hedging, particularly in the patient-centered condition. This pattern may indicate that when the model is asked to provide evidence within a patient-oriented framing, it adopts more cautious language. Such caution may reflect the model’s sensitivity to clinical risk, evidentiary uncertainty, or the constraints of responding without direct access to verified databases [27]. Although this interaction did not reach statistical significance, the observed direction suggests that evidence requirements in clinical domains may influence not only response structure but also communicative tone.

Confidence tone ratings were high across conditions, reflecting the fluent and assertive style characteristic of contemporary LLM outputs. Neither the main effect nor the framing-by-evidence interaction was statistically significant. Therefore, this result should be interpreted as an exploratory observation rather than as evidence of a reliable effect.

Descriptively, confidence tone was lower in the patient-centered, evidence-required condition than in the patient-centered, no-evidence condition, whereas confidence tone was similar across evidence conditions for skill-centered prompts. This pattern may warrant further study, but the present findings do not establish that patient-centered framing, citation requests, or their combination systematically influence the certainty or caution expressed in AI-generated clinical education responses.

Safety notices occurred rarely and inconsistently across conditions, with no statistically significant differences between evidence-required and no-evidence prompts. The cavity-preparation topics may not have elicited explicit safety disclaimers, or the prompts may have been interpreted as educational rather than as requests for directive clinical advice. Because safety notice coding showed no meaningful interrater agreement, these findings should not be interpreted as evidence that prompt design did or did not influence clinical safety.

Similarly, no refusals occurred in any condition; GPT-5 answered all prompts, including those requesting citations. Although high response compliance may improve usability, it also raises a potential concern: models may provide reference-like material even when those references are not externally verified. This may result in hallucinated or unsupported citations. Given documented concerns about fabricated citations in biomedical contexts, dental educators may wish to request identifiers such as PMIDs or DOIs and independently verify them in PubMed, Crossref, or publisher’s website as part of AI literacy training [26,28,29].

These findings are most directly relevant to how dental educators design, evaluate, and scaffold prompts when using AI-generated content in instructional settings. Because this study examined model outputs rather than learner outcomes, its implications concern the design of AI-supported educational materials rather than the direct effects on student learning or clinical performance.

First, the association between instructional framing and response stance suggests that educators should craft prompts intentionally to reflect the instructional priorities they wish to emphasize, whether patient-centered reasoning, procedural efficiency, reflective analysis, or examination preparation. Because patient-centered prompts did not uniformly produce patient-welfare–oriented responses, educators should not assume that patient priorities will be consistently emphasized unless those priorities are explicitly stated and reinforced.

Second, the effects of evidence requirements must be balanced against their drawbacks. Although evidence-required prompts were associated with higher rater-assessed perceived factuality scores in exploratory analyses and substantially increased citation presence, they also produced longer responses and may introduce pseudoacademic formatting that could confuse learners or obscure essential concepts. An evidence requirement may also alter the apparent certainty of a response, although this study did not establish such an effect. These complexities highlight the need for explicit instructions on how students should interpret, verify, and critically evaluate AI-generated citations.

Third, the low or variable interrater reliability observed for several measures, including perceived factuality and safety notices, underscores the difficulty of evaluating AI outputs consistently, even among trained raters. Consensus scoring does not eliminate this measurement uncertainty. Educators should consider scaffolded training to prepare students and faculty to assess the rigor, clinical appropriateness, and limitations of AI-generated explanations. As AI becomes more integrated into health professions education, the ability to critically appraise model outputs is likely to become an essential professional competency.

This study contributes to the prompt engineering literature by showing that framing and evidence instructions may be associated with different dimensions of LLM-generated responses. Framing was primarily associated with response perspective, particularly stance orientation, whereas evidence instructions were associated with response structure and citation behavior; an exploratory analysis also suggested an association with perceived factuality. The pattern suggests that different prompt components may influence distinct aspects of AI-generated communication.

The study did not identify statistically significant interactions for most variables. Instead, 2 prompt factors were associated with different response features: framing was associated with rhetorical orientation, whereas evidence requirements were associated with elaboration and citation inclusion. This distinction may guide future work examining how prompt components interact to shape various dimensions of AI-generated pedagogical content.

Several limitations warrant consideration. First, stance orientation exhibited low reliability across raters, suggesting that coding frameworks for complex discourse characteristics require further refinement. Perceived factuality and safety notices also showed low or negative interrater reliability, indicating that these domains were difficult to apply consistently. Although perceived factuality was rated using a structured rubric by trained raters, it did not represent objectively verified clinical accuracy. Therefore, it should be interpreted as rater-assessed perceived accuracy and completeness rather than as an objective measure of factual truth. Similarly, the presence of a safety notice should not be interpreted as a robust indicator of clinical safety. Findings involving these lower-reliability outcomes should be considered exploratory, and consensus scoring does not remove the underlying measurement uncertainty. Intrarater reliability was also not assessed because the study design did not include a repeat-rating phase in which the raters recoded the same responses after a defined interval. Consequently, this study could not determine whether individual raters applied the rubric consistently over time, particularly for interpretive domains such as perceived factuality, safety notices, and stance orientation.

Second, although the factorial design included 40 generated responses, these responses were derived from 10 base prompt topics crossed with 4 experimental conditions. Thus, the study’s conceptual breadth is better understood as 10 prompt topics rather than 40 fully independent prompts. The statistical analyses treated the 40 responses as analytic observations and did not account for the grouping of 4 prompt variants within each base topic. Because responses derived from the same topic shared underlying subject matter, the observations may not have been independent. Failing to account for this clustering may have produced SEs that did not reflect within-topic dependence and may have overstated the precision of the inferential results. In addition, the small number of base topics and sparse categorical events limited statistical power and generalizability. Accordingly, the findings should be interpreted as exploratory, descriptive patterns within this structured prompt set rather than as definitive evidence of broadly generalizable effects.

Third, reliance on the free-tier GPT-5 flagship model limits generalizability to other models, paid versions, or future system updates, which may differ in safety behavior, formatting tendencies, citation behavior, or response style. Because all outputs were generated by a single model through one interface, the findings reflect GPT-5 behavior at the time of data collection rather than prompt effects across all LLM systems. Although each prompt was administered in a new conversation with instructions to disregard prior context, this procedure reduced session-level carryover but did not make the outputs independent of the underlying model system.

Relatedly, the study used the publicly accessible GPT-5 web interface rather than an application programming interface with fixed, reportable generation parameters. Consequently, technical details such as temperature, top-p, hidden system prompts, safety-layer settings, alignment updates, and model snapshot identifiers were unavailable and could not be controlled. Because LLM systems are updated over time, the exact outputs generated in this study may not be fully reproducible in future sessions. To improve transparency, the study reports the data collection procedure in detail and provides sample prompts and responses in Multimedia Appendix 3.

Fourth, citation presence was coded as a structural feature but was not equivalent to citation validity, evidence quality, or the correct interpretation of the cited literature. The study did not verify DOIs or PMIDs, confirm whether each cited reference corresponded to an authentic publication, assess hallucinated references, or evaluate whether the cited literature supported the model’s claims.

Finally, although cavity preparation is a foundational topic in dental training, the study’s narrow clinical-procedural scope limits the generalizability. Because the study examined a single domain, the observed prompt effects may not extend to other areas of dental education, including diagnosis, treatment planning, prevention, patient communication, ethics, or other restorative procedures.

Future research should investigate how students and faculty use and interpret AI-generated explanations. Studies could examine whether learners mistake unverified or fabricated citations as genuine evidence, how confidence tone influences trust and study behavior, and whether stance orientation affects learning outcomes such as conceptual understanding or clinical reasoning. Future studies should also use larger and more diverse prompt sets across multiple dental education domains, including topics with different levels of evidentiary support. Incorporating validated reference answers, expert-consensus standards, or evidence summaries would allow for more rigorous assessment of factual accuracy and clinical appropriateness.

Additional work should refine and validate coding rubrics for evaluating LLM-generated dental education content. Future studies should assess both interrater and intrarater reliabilities, with intrarater reliability evaluated by having raters recode a subset of responses after a defined interval. This approach would help determine whether individual raters apply the rubric consistently over time, particularly for interpretive domains such as rater-assessed perceived factuality, stance orientation, safety notices, and hedging.

Experiments involving interactive, multiturn dialogue may also reveal more dynamic framing effects as the model adapts to user responses. More sophisticated prompt engineering strategies, such as structured reasoning scaffolds, self-check prompts, reference-verification prompts, and citation-validity checks, could also be tested to determine whether they better align LLM outputs with educational and ethical goals.

This study found that prompt framing and evidence instructions were associated with systematic differences in the discourse characteristics of GPT-5 responses related to cavity preparation. Evidence requirements were associated with greater citation inclusion, longer responses, and higher rater-assessed perceived factuality scores; instructional framing was associated primarily with response orientation. These findings indicate that prompt wording can influence how LLM-generated educational responses are presented and perceived. As generative AI use expands in dental education, deliberate prompt design may help educators better align AI-generated content with intended instructional objectives. However, objective accuracy, citation validity, clinical safety, and effects on learning require direct evaluation.