Developing a Large Language Model–Based Feedback System for Case Report Writing in Rehabilitation Education: Tutorial

Introduction

Background

In medical education, writing is not merely a record of clinical events but a vital learning activity that supports reflection and growth. Reflective writing (RW), including the reflective elements embedded in case report writing, fosters core competencies such as clinical reasoning and professional identity formation [1,2]. However, writing becomes educationally meaningful only when feedback effectively guides learners toward explicit goals [3]. Previous studies confirm that supportive, structured, and longitudinal feedback maximizes the impact of RW [2,3].

Delivering such feedback at scale faces persistent challenges in clinical settings. Time constraints and limited instructional skills often prevent adequate feedback [1-3]. Consequently, this feedback loop frequently breaks down, hindering the professional development of novice staff.

Large language models (LLMs) have demonstrated utility for feedback, clinical reasoning, and research assistance, including health care contexts [4-6]. A recent review suggests that artificial intelligence (AI) reduces workload and improves efficiency, particularly in medical writing and administrative support [7]. These developments have increased interest in applying LLMs to educational feedback for RW.

However, realizing this potential requires purpose-built systems that go beyond general-purpose chatbot interactions, along with careful consideration of data privacy. This tutorial addresses that gap.

Aims and Target Audience

This tutorial provides a practical guide for educators without programming experience to develop an AI-based feedback system for case report writing in clinical education. Specifically, it offers (1) a conceptual framework distinguishing custom AI systems from standard chatbot use, (2) a step-by-step implementation guide using 3 accessible tools (Dify [LangGenius, Inc]; Slack [Slack Technologies, LLC]; and Google Apps Script [GAS; Google LLC]), (3) ethical and data privacy guidelines for deploying AI in medical education settings, and (4) empirical evaluation data from a pilot implementation at a community care hospital.

The primary audience is clinical educators who supervise novice staff writing case reports, experience time pressure when providing feedback, and lack programming expertise. Although the examples and prompts presented here focus on rehabilitation settings as a working illustration, educators in other medical fields can adapt this framework by modifying the prompts to suit their specific clinical domains and educational objectives.

Rationale for a Customized System

General-purpose chatbots (eg, a standard ChatGPT interface) pose substantial risks when used without customization in medical education. The medical domain requires specialized terminology and linguistic precision that general models may fail to process accurately [8]. Effective AI use in medical education depends on strategic prompt design; poorly constructed instructions can yield inaccurate or unsafe outputs [9]. Mastering such prompt design techniques requires technical expertise that most clinical educators do not possess [9], making accessibility a core design requirement. These considerations led us to develop a customized system rather than rely on off-the-shelf chatbot interactions.

Three requirements distinguish a custom educational system from standard chatbot use. First, standardization ensures fairness. In direct chatbot interaction, response quality depends entirely on the user’s prompting skills. Novice staff may receive suboptimal feedback simply due to limited prompting skills. A custom system uses preprogrammed instructions, known as meta-prompts (Textbox 1), to evaluate every draft against the same rubric, ensuring consistent feedback for all learners. Second, integration enables visibility and progress management. When novice staff use personal chatbot accounts, educators cannot monitor their interactions. Integrating the system into a shared workspace enables educators to transparently track progress and intervene when necessary. Third, centralized logging supports educational auditing. To improve the curriculum, educators need to identify common errors among novice staff. A custom system automatically records all interactions, generating valuable data for analyzing trends in learner performance.

Methods

Highlights

This section is organized as follows. First, we describe the 3-component system architecture (Dify, Slack, and GAS) and the dataflow connecting them. Second, we present the feedback design principles and the specifications of 4 educational chatbots. Third, we provide a step-by-step implementation guide for reproducing the system. Fourth, we address ethical considerations and data privacy safeguards. Finally, we describe a pilot implementation conducted at a community care hospital to assess system usability and feasibility.

System Architecture and Dataflow

Figure 1 illustrates the operational workflow. From the novice staff’s perspective, the system provides a straightforward interaction: the novice staff tags the relevant bot in Slack (eg, @Discussion_Bot) and submits a draft text. The system processes the request and provides specific feedback within the same thread. The novice staff then revises the text and replies to the bot again. This iterative conversation continues until the draft meets the required standards. This workflow off-loads the time-consuming revision cycle to the AI, reserving human feedback for clinical validity and professional judgment.

The system architecture consists of 3 components selected to enable implementation without advanced programming knowledge (Figure 2). Programming is a well-documented barrier for health care professionals seeking to develop AI applications [10]. We therefore adopted an approach that does not require specialized programming expertise and uses the following 3 components: Dify, Slack, and GAS.

Dify [11] is an open-source LLM application development platform that serves as the system’s central processing engine. Hosted in the cloud, it receives user inputs, processes them through a predesigned workflow, and instructs the AI how to respond. Educators can build the AI’s reasoning process visually, without writing code. We selected Dify because it offered the most accessible visual interface among available open-source LLM workflow platforms, combined with built-in application programming interface (API) publishing and conversation logging. The free tier is suitable for initial testing but has limited message credits; a paid plan may be required for sustained use with multiple learners. Data sovereignty considerations are discussed in the Ethical Considerations and Data Privacy section.

Slack [12] serves as the interactive chat interface for novice staff. Its private channel structure enables educators to monitor all novice staff-AI interactions in real time and to communicate with novice staff within the same workspace. The free tier limits message history retention, which we addressed through automatic logging to Google Sheets via GAS.

GAS functions as the technical bridge linking Slack and Dify, securely relaying messages between the 2 platforms. We chose GAS for its native integration with Google Workspace, enabling automatic logging to Google Sheets and administrator error notifications without additional infrastructure. Although Dify provides a built-in Slack plug-in for direct integration, it does not support the logging and oversight functions considered essential for educational management. When a novice staff sends a message on Slack, the data travels through GAS to reach Dify, and the response returns along the same path. Every interaction is automatically logged, visible to educators, and monitored for errors.

A key design feature is the iterative pass or revise loop implemented within Dify’s Chatflow functionality (Figure 3). Unlike a simple chatbot that answers a question once, effective education requires iterative engagement [3,13]. A recent Association for Medical Education in Europe guide indicates that continuous, formative feedback aligned with explicit learning objectives develops competencies more effectively than one-time summative evaluation [14]. The system, therefore, implements a conditional workflow:

• Analyze: the system evaluates the novice staff’s draft against the defined rubric.
• Decide the following:
  • If the draft meets the criteria, the system outputs “pass” and allows the novice staff to proceed.
  • If not, the system outputs “revise,” provides targeted comments, and awaits the corrected draft.
• Repeat: this mechanism prevents novice staff from overlooking feedback and promotes iterative learning.

‎

‎

‎

Feedback Design and Bot Specifications

Overview

The feedback design rests on 2 complementary principles: comment-based feedback and the human-in-the-loop approach.

Comment-based feedback prioritizes written comments over direct correction. Research suggests that comments produce greater gains in writing proficiency than simple error location or direct correction [13] because comments compel learners to actively process information and revise their own work [14]. The system thus functions as an always-available virtual instructor, serving as a strict gatekeeper prior to human review. By off-loading the repetitive task of checking logical consistency and structure to the AI, educators can focus on the final clinical assessment.

The human-in-the-loop approach ensures that educators retain final decision-making authority; AI serves only as a support tool. The AI focuses on overall structure, logical consistency, and writing style, while avoiding specific clinical judgments to mitigate the risk of hallucinations (ie, plausible but incorrect medical advice). This system does not evaluate clinical validity or refine medical discussions. Qualified educators must provide professional feedback on these aspects, and novice staff should treat AI feedback as suggestions for improving writing structure and logic, not as clinical guidance.

On the basis of these principles, the system provides 4 distinct chatbots tailored to the report writing process (Table 1; refer to Multimedia Appendix 1 for detailed meta-prompts.) The system uses 2 distinct feedback approaches: a “loop-based” approach for iterative revision and a “single-shot” approach for efficient one-time processing.

Loop-Based Chatbots

The 2 loop-based chatbots promote clinical reasoning through iterative, comment-based feedback:

• Integration and interpretation bot: verifies clinical reasoning by checking the logical consistency between assessment results, the problem list, and goal setting. The bot prompts revisions until all logical checkpoints are met.
• Discussion bot: evaluates whether the argument aligns with the interpretation and logically supports the conclusion. The bot prompts revisions until all checkpoints are satisfied.

Single-Shot Chatbots

The 2 single-shot chatbots perform efficient, single-pass tasks:

• Proofreading bot: corrects grammatical errors and improves clarity without altering the clinical meaning. It generates a polished draft in a single interaction.
• Abstract summarization bot: produces a concise summary of the completed case report in a format suitable for conference or journal submission.

Step-By-Step Implementation

Overview

This guide enables educators to build the system by following step-by-step instructions and copying the provided resources; no complex coding is required. The following subsections outline the role of each component and the key actions involved. Detailed instructions, including specific platform operations, configuration values, and troubleshooting guidance, are provided in Multimedia Appendix 2.

Prerequisites: Accounts and API Keys

Ensure the following are ready before starting:

• Dify account: (cloud version or self-hosted)
• Slack workspace: (educators need permission to create apps and add bots)
• Google account: (to access Google Drive and Sheets)
• API keys: an API key from an LLM provider (eg, OpenAI API key or Anthropic API key; Textbox 2).

Setting Up Dify

Dify serves as the AI processing engine. Implementation involves creating a new Chatflow-type application, importing the feedback workflow via the domain-specific language file (Multimedia Appendix 3), registering the LLM API key, and adjusting the meta-prompts to suit the target clinical domain. Once published, Dify generates an API key for use in subsequent integration steps.

Creating the Slack Application

After configuring Dify, the next step is to create the bot account in Slack. This involves registering a new app in the Slack API console, assigning the minimum necessary permissions (eg, reading messages, sending replies), and installing the bot in the target workspace. The resulting bot token is required for the integration step.

Configuring GAS

Dify handles the AI logic, and Slack provides the chat interface, but the 2 platforms cannot communicate directly. GAS serves as the bridge connecting them. Implementation involves creating a GAS project, deploying the relay script (Multimedia Appendix 4), and securely storing the API credentials from the previous steps as script properties. The script is then deployed as a web application so that Slack can send messages to it.

Final Configuration and Verification

The final step links all 3 components and verifies system function. Configuring Slack’s event subscriptions feature to forward messages to the GAS end point completes the communication loop. A test message sent in Slack should trigger the full pipeline: Slack receives the message, GAS relays it to Dify, the LLM generates feedback, and the response returns to Slack while being logged in Google Sheets.

Data Privacy

Because this system transmits case report content through multiple external platforms, safeguarding protected health information (PHI) is essential. This section describes the data protection measures organized into 3 categories: administrative, technical, and supervisory safeguards.

Administrative Safeguards

Before submitting any draft to the system, novice staff must deidentify all patient information following the anonymization standards recommended by the International Committee of Medical Journal Editors [15]. The deidentification protocol involves 2 steps:

1. Remove all direct identifiers: patient names, medical record numbers, dates of birth, addresses, and contact information.
2. Generalize quasi-identifiers to prevent reidentification through combination: convert specific dates to relative timelines (eg, “Day 1 of admission” rather than a calendar date), express exact ages in decade ranges (eg, “a patient in their 80s”), and replace facility names with generic descriptors (eg, “a community rehabilitation hospital”).

In the pilot implementation, we provided deidentification training to all participants prior to system use (refer to the Ethical Considerations section). Each novice staff member received individual instruction on the protocol, supplemented by a short instructional video demonstrating the 2-step process with concrete examples from rehabilitation case reports. We reinforced compliance through ongoing supervision during the pilot period.

Technical Safeguards

All communication with LLMs in this system occurs exclusively through the API, not through consumer-facing chat interfaces such as ChatGPT (OpenAI) or Claude.ai (Anthropic). This architectural choice provides a contractual safeguard: both OpenAI and Anthropic explicitly state in their API terms of service that data submitted through the API are not used to train or improve their models [16,17]. Both OpenAI and Anthropic retain API data for up to 30 days solely for abuse monitoring purposes, after which it is automatically deleted. Because data submitted to LLM providers cannot be recalled during this retention window, deidentification prior to submission constitutes the critical safeguard.

To provide a transparent risk assessment, we trace the complete data path (Figure 4). When a novice staff submits a message in Slack, the data traverses 5 external nodes before a response is returned:

1. Slack servers: Slack stores messages with encryption; access is restricted to invited channel members.
2. GAS: GAS processes data transiently during relay without persistent storage.
3. Google Sheets: conversation logs are persistently stored for educational analysis, with access restricted to designated administrators.
4. Dify Cloud: conversation logs are retained within the platform and are accessible only to the system administrator.
5. LLM API end point (OpenAI or Anthropic): the LLM provider processes data and retains it for up to 30 days under the nontraining policies described above.

For institutions requiring full data sovereignty, Dify offers a self-hosted deployment option that retains all data within the institution’s own infrastructure, though this requires server infrastructure and technical expertise.

‎

Real-Time Oversight and Incident Response

The system architecture supports real-time oversight. Because all interactions occur in shared Slack channels, educators can continuously monitor both novice staff submissions and AI responses, enabling immediate detection of any inadvertent PHI disclosure. The Google Sheets log provides a persistent, searchable record for periodic auditing. If PHI is inadvertently submitted, educators can take the following remediation steps:

• Delete the PHI-containing message from Slack.
• Remove the corresponding row from the Google Sheets log.
• Purge the conversation log from the Dify platform.
• The LLM provider automatically deletes data within the 30-day retention window.

The current system does not include automated PHI detection prior to submission; deidentification relies on the administrative safeguards described above. We identify the development of automated presubmission screening as a priority for future implementations (refer to the limitations and future directions section).

Transparency, Consent, and Scope of AI Feedback

Transparency is foundational to ethical AI adoption. Before introducing the system, educators should clearly explain to novice staff that they are communicating with an AI system and how their data will be processed. Educators should also define the system as an educational support tool for facilitating the drafting process, rather than a replacement for human supervision or a formal assessment of clinical capabilities.

To manage the risk of hallucinations, the system uses 2 strategies. First, the AI’s feedback includes a disclaimer: “AI can make mistakes. Always verify medical facts with your supervisor.” Second, the AI’s role is limited to structural and logical feedback rather than clinical diagnosis, reducing the risk of harmful errors.

Ethical Considerations

The ethics committee of Kyoto Min-iren Asukai Hospital approved this pilot evaluation (2024-0502). All participating staff members received verbal and written explanations of the study’s purpose and data handling procedures, and provided written informed consent. We explained deidentification requirements to all novice staff members individually and reinforced them through an instructional video prior to system use. For past case reports used in meta-prompt tuning, we implemented an opt-out procedure. Participants received no financial compensation.

Iterative Development Process

The current Dify-based system evolved through iterative refinement of an earlier prototype. The initial version used a single-prompt chatbot approach. Early testing revealed 2 challenges: a ceiling effect in which most novice staff passed all checkpoints on their first submission, indicating that the meta-prompts were too lenient for meaningful learning, and declining instruction adherence as prompt length increased. These issues motivated the transition to the Dify-based Chatflow architecture, which evaluates checkpoints sequentially and allows educators to adjust passing criteria through its visual interface.

Some instructors also expressed concern that overreliance on AI feedback could diminish clinical reasoning skills. We therefore designed the system to limit AI feedback to structural and logical aspects and to prompt learners to reconsider their own reasoning rather than providing corrected example text.

Pilot Implementation

To inform the design and iterative refinement of this tutorial, we conducted a single-center pilot implementation at Kyoto Min-iren Asukai Hospital. This pilot used a prototype version with direct Slack-GAS-AI integration prior to the introduction of Dify. Because the user interface (Slack) and the fundamental feedback logic (iterative revisions) remained identical, the usability data are directly applicable to the Dify-based system.

Five novice rehabilitation staff members (3 physiotherapists and 2 occupational therapists) who joined in April 2024 and 5 educational instructors participated. The pilot ran from April to June 2024. Prior to this pilot, the facility had not used Slack for official communication; thus, this pilot represented the participants’ first experience with the platform. During the pilot, participants tagged the specific bot (eg, “@Integration_Bot”) in Slack to initiate iterative feedback on structure and logic. Once the content was finalized, they used the “@Proofreading_Bot” for grammatical corrections before submitting the final draft to their human supervisors.

We collected data through 2 methods. First, postimplementation surveys via Google Forms assessed the system’s usability and acceptance. Novice staff completed the System Usability Scale [18] and rated the appropriateness of the AI feedback content. The System Usability Scale is a standardized 10-item questionnaire for measuring system usability. Scores above 68 indicate above-average usability, and scores above 85 indicate excellent usability. Instructors assessed changes in instructional efficiency and feedback appropriateness. Second, we analyzed use logs from the Slack-linked spreadsheet, including use frequency, revision counts per task, and the content of feedback.

Results

This section describes the pilot implementation experience, which informed the iterative refinement and final design of this tutorial.

The postimplementation survey response rate was 100% (5/5) for both instructors and novice staff. Detailed survey results are provided in Multimedia Appendix 5.

Novice staff found the system accessible, with a median System Usability Scale score of 90 (range 70-95), confirming that the approach lowered the barrier to adoption. All 5 staff members reported that the AI feedback was helpful for their learning (median 5, range 4-5). Instructor feedback identified areas where the tutorial framework requires further customization: while 80% (4/5) of staff members anticipated a reduction in future instructional burden (median 4, range 1-5), ratings for the appropriateness of AI feedback content were moderate (median 3, range 1-4), suggesting that the balance between structural feedback and clinical depth needs institution-specific calibration.

Log analysis confirmed that all 5 novice staff members actively used the system. Typically, they used the integration and interpretation bot 1 to 2 times and the proofreading bot once per report.

Qualitative feedback from novice staff highlighted the psychological benefits of the system, with comments such as “My mental burden is reduced because I can get feedback before showing it to a human”. Instructors acknowledged efficiency gains, noting the ability to “focus on specialized feedback”. However, some instructors expressed concern that reliance on the system could lead to a decline in writing skills and clinical reasoning abilities.

Discussion

Principal Findings

This tutorial presents a practical framework for building an LLM-based feedback system for case report writing in clinical education. The principal contribution is demonstrating that educators can construct a customized AI feedback system by combining 3 accessible tools (Dify, Slack, and GAS) while maintaining educational quality and data privacy.

The system embodies 2 key design principles. First, comment-based iterative feedback encourages active revision rather than passive correction, preserving the cognitive effort essential for developing clinical reasoning skills [3,13,14]. Second, the human-in-the-loop principle limits AI to structural and logical evaluation, reserving clinical validity assessment for qualified educators and mitigating the risk of unwarranted clinical authority [19].

The pilot implementation experience supported the design decisions underlying this tutorial. In particular, the transition from the prototype to the Dify-based architecture was directly motivated by 2 challenges observed during early testing: a ceiling effect indicating that initial meta-prompts were insufficiently stringent and declining instruction adherence as prompt complexity increased. These observations illustrate a core advantage of the framework presented here: educators can iteratively adjust passing criteria and feedback logic through Dify’s visual interface, enabling continuous adaptation without programming expertise.

Comparison With Prior Work

To our knowledge, few studies have provided practical, reproducible guidance for clinical educators to build customized LLM-based feedback systems. Systematic reviews of LLMs in medical education have predominantly identified studies evaluating LLM performance on examinations and clinical knowledge assessments, with very few reporting empirical implementation research [20,21]. Practical programming-free AI development guides in health care have focused on other domains, such as image classification [10], and exploratory studies in adjacent fields have examined similar accessible tools for providing formative feedback on student writing [22]. A step-by-step guide for building an LLM-based formative feedback system for clinical writing education has not been described. This tutorial addresses that gap.

The design principle of delegating structural and logical review to AI while reserving clinical judgment for educators reflects an emerging model in which AI supports, rather than replaces, expert judgment [19,20]. Prior studies on the use of AI in education have reported reductions in teachers’ workload and stress, along with improvements in instructional efficiency [23,24]. The pilot findings were consistent: instructors reported the ability to “focus on specialized feedback” rather than routine text review. This division of labor may offer a scalable approach to feedback delivery in resource-constrained educational settings.

Accessibility was a key design consideration. Prior research indicates that AI-based educational tools positively influence learning efficiency and cognitive motivation [24,25]. The pilot results supported this, with high usability ratings achieved even among staff with no prior experience using the chat platform.

This tutorial demonstrates that rubric-based LLM evaluation can be customized for domain-specific educational purposes without programming expertise. Although the potential of AI as a writing support tool is well documented [26,27] and LLMs can reliably evaluate clinical writing against structured rubrics [28], these capabilities have typically required programming expertise, limiting accessibility for clinical educators.

Overreliance on AI feedback poses recognized risks, including cognitive deskilling and diminished independent reasoning [29,30]. A recent scoping review of ethical challenges in AI-assisted medical education underscores the need for hybrid learning models that balance AI assistance with traditional teaching [19]. This tutorial addresses these concerns through the human-in-the-loop principle: the system limits AI feedback to structural and logical aspects, and all AI-generated comments include disclaimers directing novice staff to verify content with their supervisors.

Limitations and Future Directions

This tutorial and its pilot evaluation have several limitations. First, the small sample size (5 novice staff and 5 instructors) limits causal inference. Second, as a single-center pilot, future research should involve multicenter collaborative studies. Third, the outcomes relied on subjective questionnaires. Future studies should analyze quantitative data such as the duration of instructor guidance and use independent reviewers to evaluate report quality objectively. Fourth, although we provided deidentification training through individual instruction and an instructional video, the pilot did not include an automated presubmission PHI detection mechanism. Recent studies have demonstrated that LLM-based systems can analyze conversation data while preserving privacy [31].

No incidents of PHI disclosure were identified during the pilot period, suggesting that the current approach, which aligns with standard security practices for cloud-based services, was sufficient within this controlled setting. However, future implementations should strengthen compliance through additional safeguards: supervisor-led presubmission review, prioritization of hypothetical cases during early adoption, and development of automated PHI detection systems. In contexts requiring stricter data governance, locally hosted LLMs may also be considered.

Two broader considerations apply to the adoption of this approach. The system relies on cloud-based services, which may not meet the data sovereignty requirements of all institutions. Additionally, because LLM capabilities and API policies evolve rapidly, educators should monitor LLM provider updates regarding data handling practices and model performance. Furthermore, while this tutorial provides detailed implementation instructions and reproducible resources, the step-by-step guide has not yet been independently tested by educators outside the development team.

Future directions include broader validation of the tutorial’s reproducibility across diverse clinical settings, development of automated PHI detection as a presubmission safeguard, and investigation of the system’s long-term impact on writing proficiency and clinical reasoning development.

Conclusions

This tutorial provides a reproducible guide for building a case report feedback system using Dify and Slack without programming expertise. The architecture enables educators to fine-tune AI logic through visual interfaces, separating AI reasoning from software code. The tutorial also addresses data privacy considerations for deploying AI systems that handle clinical educational content. By emphasizing human oversight at every stage and iterative refinement of feedback criteria, this framework supports educators in adapting the system to their institutional context and evolving educational needs.