This paper is in the following e-collection/theme issue:

Mobile Approaches to Medical Education (53) e-Learning and Medical Education (645) Human Factors and Usability Case Studies (739) Participatory Medicine & E-Patients (758) Innovations and Technology in Cancer Care (400) Patient Education for Cancer (138) Supporting Partners and Informal Caregiver of Cancer Patients (37) Supporting Informal Care and Caregivers (272)

Published on in Vol 10 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/50118, first published .
Development of Web-Based Education Modules to Improve Carer Engagement in Cancer Care: Design and User Experience Evaluation of the e-Triadic Oncology (eTRIO) Modules for Clinicians, Patients, and Carers

Development of Web-Based Education Modules to Improve Carer Engagement in Cancer Care: Design and User Experience Evaluation of the e-Triadic Oncology (eTRIO) Modules for Clinicians, Patients, and Carers

Development of Web-Based Education Modules to Improve Carer Engagement in Cancer Care: Design and User Experience Evaluation of the e-Triadic Oncology (eTRIO) Modules for Clinicians, Patients, and Carers

Authors of this article:

Rebekah Laidsaar-Powell1, 2 Author Orcid Image ;   Sarah Giunta1 Author Orcid Image ;   Phyllis Butow1, 2 Author Orcid Image ;   Rachael Keast1 Author Orcid Image ;   Bogda Koczwara3, 4 Author Orcid Image ;   Judy Kay5 Author Orcid Image ;   Michael Jefford6 Author Orcid Image ;   Sandra Turner7, 8 Author Orcid Image ;   Christobel Saunders9 Author Orcid Image ;   Penelope Schofield6, 10, 11 Author Orcid Image ;   Frances Boyle8, 12 Author Orcid Image ;   Patsy Yates13 Author Orcid Image ;   Kate White14 Author Orcid Image ;   Annie Miller15 Author Orcid Image ;   Zoe Butt1 Author Orcid Image ;   Melanie Bonnaudet5, 16 Author Orcid Image ;   Ilona Juraskova1, 2 Author Orcid Image
Article Authors Cited by Tweetations (2) Metrics

    Original Paper

    1Centre for Medical Psychology & Evidence-based Decision-making, School of Psychology, The University of Sydney, Sydney, Australia

    2Psycho-Oncology Co-operative Research Group, The University of Sydney, Sydney, Australia

    3Flinders Medical Centre, Adelaide, Australia

    4College of Medicine and Public Health, Flinders University, Adelaide, Australia

    5School of Computer Science, The University of Sydney, Sydney, Australia

    6Health Services Research and Implementation Science, Peter MacCallum Cancer Centre, Melbourne, Australia

    7Department of Radiation Oncology, Westmead Hospital, Westmead, Australia

    8Faculty of Medicine and Health, The University of Sydney, Sydney, Australia

    9Department of Surgery, Royal Melbourne Hospital, University of Melbourne, Melbourne, Australia

    10Department of Psychology and Iverson Health Innovation Research Institute, Swinburne University, Melbourne, Australia

    11Sir Peter MacCallum Department of Oncology, The University of Melbourne, Melbourne, Australia

    12Patricia Ritchie Centre for Cancer Care & Research, Mater Hospital, Sydney, Australia

    13Faculty of Health, Queensland University of Technology, Brisbane, Australia

    14Susan Wakil School of Nursing, The Daffodil Centre, The University of Sydney, a joint venture with Cancer Council New South Wales, Sydney, Australia

    15Cancer Council New South Wales, Sydney, Australia

    16School of Electrical Engineering and Computer Science, Kungliga Tekniska högskolan Royal Institute of Technology, Stockholm, Sweden

    Corresponding Author:

    Rebekah Laidsaar-Powell, PhD

    Centre for Medical Psychology & Evidence-based Decision-making

    School of Psychology

    The University of Sydney

    Room 310, Level 3, Griffith Taylor Building (A19)

    Manning Road

    Sydney, 2006

    Australia

    Phone: 61 2 9351 6811

    Email: rebekah.laidsaar-powell@sydney.edu.au


    Background: Carers often assume key roles in cancer care. However, many carers report feeling disempowered and ill‐equipped to support patients. Our group published evidence-based guidelines (the Triadic Oncology [TRIO] Guidelines) to improve oncology clinician engagement with carers and the management of challenging situations involving carers.

    Objective: To facilitate implementation of the TRIO Guidelines in clinical practice, we aimed to develop, iteratively refine, and conduct user testing of a suite of evidence-based and interactive web-based education modules for oncology clinicians (e-Triadic Oncology [eTRIO]), patients with cancer, and carers (eTRIO for Patients and Carers [eTRIO‐pc]). These were designed to improve carer involvement, communication, and shared decision-making in the cancer management setting.

    Methods: The eTRIO education modules were based on extensive research, including systematic reviews, qualitative interviews, and consultation analyses. Guided by the person-based approach, module content and design were reviewed by an expert advisory group comprising academic and clinical experts (n=13) and consumers (n=5); content and design were continuously and iteratively refined. User experience testing (including “think-aloud” interviews and administration of the System Usability Scale [SUS]) of the modules was completed by additional clinicians (n=5), patients (n=3), and carers (n=3).

    Results: The final clinician module comprises 14 sections, requires approximately 1.5 to 2 hours to complete, and covers topics such as carer-inclusive communication and practices; supporting carer needs; and managing carer dominance, anger, and conflicting patient-carer wishes. The usability of the module was rated by 5 clinicians, with a mean SUS score of 75 (SD 5.3), which is interpreted as good. Clinicians often desired information in a concise format, divided into small “snackable” sections that could be easily recommenced if they were interrupted. The carer module features 11 sections; requires approximately 1.5 hours to complete; and includes topics such as the importance of carers, carer roles during consultations, and advocating for the patient. The patient module is an adaptation of the relevant carer module sections, comprising 7 sections and requiring 1 hour to complete. The average SUS score as rated by 6 patients and carers was 78 (SD 16.2), which is interpreted as good. Interactive activities, clinical vignette videos, and reflective learning exercises are incorporated into all modules. Patient and carer consumer advisers advocated for empathetic content and tone throughout their modules, with an easy-to-read and navigable module interface.

    Conclusions: The eTRIO suite of modules were rigorously developed using a person-based design methodology to meet the unique information needs and learning requirements of clinicians, patients, and carers, with the goal of improving effective and supportive carer involvement in cancer consultations and cancer care.

    JMIR Med Educ 2024;10:e50118

    doi:10.2196/50118

    Keywords

    family carers; patient education; health professional education; web-based intervention; mobile phone 


    Background

    Carers (including but not limited to spouses, partners, adult children, siblings, parents, or friends [1]) of adults with cancer assume many responsibilities in supporting and caring for their loved one [2]. Carers can experience many challenges in this demanding role and often report high distress [3,4], poor physical health, low quality of life, and unmet needs [5,6]. As carer burden increases, carers may neglect their own needs, which can also impact their ability to support and care for their loved one [7,8].

    While issues faced by carers are well recognized by health care professionals [9], many clinicians report that they do not know how to appropriately engage with carers or address their unique challenges [9,10]. Oncologists have reported a lack of education about communicating with carers [10], and suboptimal carer-clinician communication is common [11]. Some carers report being overlooked in medical consultations and feeling disempowered and unprepared in their caregiving role [12]. Clinician inclusion and support of carers have been reported as highly valued by both carers and patients [12].

    Improving carer engagement and support needs to be addressed from multiple perspectives. Not only are clinicians uncertain about how to include carers in consultations [9] but also many carers often lack confidence and skills in caregiving [12,13], and some patients are unsure about what role their carer should assume in medical consultations and decision-making [14]. Therefore, interventions targeting all members of the clinician-patient-carer trio are needed.

    Web-based delivery of education offers efficacy, efficiency, ability to undertake training in discrete periods, lower cost, flexibility, and greater reach than traditional face-to-face formats [15]. A systematic review of web-based health education by George et al [16] found web-based training for health professionals to be as effective as or better than face-to-face formats on outcomes such as knowledge, skills, and attitudes. Web-based communication skills interventions have been found to be effective in improving self-rated clinician confidence, communication skills, and knowledge among cancer clinicians [15]. For example, a web-based module developed by our group to educate nurses about managing conflict involving patients and carers (the Triadic Oncology [TRIO]–Conflict module) was found to improve cancer nurses’ attitudes and confidence in interacting with carers [17].

    Patients and carers can also benefit from web-based resources and educational tools [18]. A systematic review of digital psychosocial interventions for patients with cancer and carers found web-based interventions to be both feasible and acceptable [19]. Digital interventions for carers have been shown to improve carer outcomes, knowledge, and skills, with the additional benefit of being accessible from home, thus minimizing the demands on carers’ time [20]. For example, a web-based psychosocial intervention for patients with cancer, Stress-Aktiv-Mindern (STREAM), has demonstrated beneficial patient outcomes including reduced stress and improved quality of life [21]. Similarly, the psychoeducational platform, Comprehensive Health Enhancement Support System (CHESS), has demonstrated favorable outcomes among carers such as significant reduction in negative mood and carer burden [22]. These beneficial effects were comparable to those of traditional psychoeducation interventions [23,24]. While STREAM and CHESS demonstrate the efficacy of web-based patient and carer support, their focus is on psychosocial support. To date, there have been no web-based education modules dedicated to empowering and upskilling patients and carers in carer-relevantcommunication and engagement with cancer clinicians and in carer participation in cancer treatment decision-making. Therefore, we aimed to develop and evaluate a web-based learning tool to address these needs.

    Interventions to support cancer carers are often difficult to implement in clinical practice and face barriers to implementation including problems with design, feasibility, acceptability, and cost [25]. One way to improve the acceptability and sustainability of an intervention is to use a co-design approach with the target population as stakeholders, to ensure that the program targets user needs and preferences. The person-based approach [26] ensures that intervention development is grounded in the perspectives and psychosocial context of end users via iterative, qualitative research with relevant stakeholders. This approach has been effectively used in the development of web-based health care interventions [27,28].

    Objectives

    This paper describes the development, iterative refinement, and user testing of evidence-based and interactive web-based interventions designed to improve engagement and communication with carers in cancer care. We have published the study protocol for a randomized controlled trial to test the efficacy of the e-Triadic Oncology (eTRIO) modules elsewhere [29]. However, necessary amendments to the planned randomized controlled trial due to the COVID-19 pandemic were made after publication of the protocol. The evaluation approach was revised to hybrid effectiveness and implementation studies using a pre-post, single-arm intervention design.

    In this paper, we have reported about the development of web-based education modules for all 3 relevant stakeholder groups, including oncology health professionals and patients with cancer and carers (eTRIO for patients and carers [eTRIO-pc]).


    Overview

    The person-based co-design approach by Yardley et al [26] underpinned the module design. Development and user experience testing of the clinician (eTRIO) and patient-carer (eTRIO-pc) modules was undertaken in multiple cyclical phases of data collection, analysis, and integration, in a process of iterative refinement [30]. Consistent with the approach by Yardley et al [26], this involved (1) planning: development of module content based on evidence, qualitative interviews with stakeholders, and input from our expert advisory group; (2) design: iterative review and refinement based on advisory group feedback; and (3) development and evaluation of acceptability and feasibility: formal heuristic evaluation, System Usability Scale (SUS) questionnaire, and think-aloud review of the eTRIO modules by stakeholders (Figures 1 and 2). The final phase of implementation and trialing is currently being conducted in a separate pre-post evaluation study, which will be reported elsewhere.

    ‎
    Figure 1. e-Triadic Oncology (eTRIO; clinician) module development process. SUS: System Usability Scale.
    ‎
    Figure 2. e-Triadic Oncology for patients and carers (eTRIO-pc) module development process. CCNSW: Cancer Council New South Wales; SUS: System Usability Scale.

    Phase 1: Development of eTRIO and eTRIO-pc Module Content

    Development of the eTRIO Clinician Module

    The content of the modules was informed by our extensive Triadic Oncology (TRIO) research program, which includes a systematic review of carer involvement in consultations [31]; qualitative interviews with oncology clinicians, patients, and carers [9,14,32]; quantitative and qualitative analyses of audiotaped oncology consultations [11]; a conceptual framework of carer involvement in medical decisions [33]; and carer communication guidelines for clinicians (TRIO Guidelines) developed via a Delphi consensus process [34,35]. Key clinician training needs, strategies, and behaviors relevant to the module were ascertained through this extensive research program.

    On the basis of this prior research, we developed an initial draft of the eTRIO content. The draft module comprised 14 study sections (1 introductory section and 13 strategy areas covered in the TRIO Guidelines [34,35]). A clinician expert advisory group was formed to provide feedback about the module content, comprising medical oncologists (3/13, 23%), oncology nurses (2/13, 15%), psychologists (2/13, 15%), a radiation oncologist (1/13, 8%), an oncology surgeon (1/13, 8%), and the research team comprising psycho-oncologists (4/13, 31%). Each member of the clinician expert advisory group reviewed a text-based draft of the module content and provided written feedback about each module section, including interactive activities, reflective exercises, and wording of strategies. Multiple teleconferences were conducted, where group members provided feedback about the content and structure of each section. Major changes were discussed with the group until consensus was reached. Feedback from the advisory group was collated, and the module content was iteratively refined.

    Development of the eTRIO-pc Patient-Carer Module

    The eTRIO-pc module content was drafted based on a review of current web-based guidance for carers about involvement in medical consultations [18], qualitative studies of patients and carers [9,14,32], and analyses of audiotaped consultations [11]. A meeting with the staff at a leading nongovernment cancer support and advocacy organization (n=5) was also conducted to inform the content of the eTRIO-pc initial draft. The staff members were asked to describe the key content that should be included in the eTRIO-pc module, based on their experience in supporting patients and carers via a telephone information and support service.

    Consumer advisers (3/5, 60% cancer carers and 2/5, 40% patients with cancer) also provided iterative feedback about the module content during a half-day workshop and via email. Consumer advisers were asked to comment about whether the module content was understandable, the relevance of the module content and feasibility of the suggested strategies, the language, and tone of the module. All feedback from the Cancer Council New South Wales support staff and consumer advisers was collated and discussed with the project team until consensus was reached through revisions.

    After the development and iterative revision of the module content was complete, video vignettes modeling key carer communication skills were developed to supplement the written content. Video vignettes have been demonstrated as an effective educational tool for patients and carers and can improve accessibility for those with low literacy [36,37]. We engaged a professional medical education and communication production company to develop a script covering key learning areas for carers, as determined by the consumer advisory groups. The script was iteratively reviewed by the research team, consumer advisers, and a physician to ensure that the videos aligned with the TRIO communication guidelines [34,35] and were clinically relevant and feasible.

    Phase 2: Iterative Design, Review, and Refinement of eTRIO and eTRIO-pc Web-Based Modules

    Design and Refinement of eTRIO Clinician Module

    As shown in Figure 1, phase 2 involved consumer input and refinement of the modules. To translate the text-based content into an interactive web-based educational module, we studied the best practice principles for the delivery of e-learning to health professionals [16,38-40]. This included a review by de Leeuw et al [38] about e-learning features targeted at postgraduate medical students and health professionals completing ongoing professional development, which identified 6 domains of important elements for e-learning quality (preparation, design, communication, content, assessment, and maintenance). Informed by a previous review [38], we developed a base design and catalog of potential design features.

    A prototype web platform was developed by a professional web development company. In 2 sessions conducted via Zoom (Zoom Video Communications), the clinician advisory group completed a walk-through of the module and provided comprehensive feedback. Their verbal and written feedback was collated and integrated into a revised web-based module.

    Design and Refinement of the eTRIO-pc Patient-Carer Module

    Similarly, as displayed in Figure 2, phase 2 involved the conversion of the text-based module content for patients and carers into an interactive web-based platform. We conducted a review of the content and design features of other available evidence-based web-based platforms for carers [18], drew on the evidence base surrounding education for carers [41-43], and received input from the consumer advisory group. To inform the website design, we reviewed the publicly available web-based resources for carers.

    The final design features of eTRIO and eTRIO-pc were implemented by a professional web development company and included interactive activities, video vignettes, and text-based content. The clinician and consumer advisory groups were given access to the draft module, and its content and format were revised based on their extensive feedback. An expert in human-centered IT design was involved in all stages of development of the clinician and patient-carer modules.

    Phase 3: Heuristic Evaluation and “Think Aloud” User Experience Evaluation of eTRIO and eTRIO-pc Web-Based Modules

    As shown in Figures 1 and 2, phase 3 involved usability evaluations of the developed web-based module. We conducted a heuristic evaluation to discover technical and usability issues [44]. The modules were examined by the researchers to identify problems that did not comply with the usability principles recognized by Nielsen [45], which include consistency and standards, error prevention, and aesthetic and minimalist design. The severity and prevalence of the issues were ranked from 1 to 5, with a high rank indicating that the problem was a priority to fix, and the web platform was updated accordingly.

    Usability and user experience testing for the penultimate versions of eTRIO and eTRIO-pc were conducted using think-aloud methodology with 11 participants, including clinicians, patients, and carers, all of whom were naïve to the TRIO Guidelines and modules. Think aloud is an effective evaluation method in which participants are provided with an interface and asked to verbalize their thoughts as they work through it [46,47]. Potential participants were identified through the research team’s professional networks and via social media advertisements.

    The consenting participants completed a demographic questionnaire and a 4-item self-report measure of health literacy [48]. Participants were provided access to the relevant eTRIO module and asked to speak aloud their thoughts and impressions as they were completing the module (think-aloud). These sessions were conducted face to face or via videoconferencing. After working through the module, participants completed the SUS [49]. Think-aloud evaluations were audio recorded and transcribed verbatim. Transcripts were qualitatively analyzed using thematic analysis [50], which involved familiarization with the transcripts, coding of salient initial ideas as codes, identification of patterns in the codes to generate themes and subthemes, and iterative review of the themes and subthemes to ensure a coherent and comprehensive thematic structure. This process was conducted collaboratively and through iterative discussion by RLP, PB, ZB, MB, and IJ. Themes were related to the following: usability and technical issues, positive aspects of design and function, attitudes toward the content of the program, and perspectives about the impact or implementation of the program. All transcripts were analyzed based on the established thematic framework and were grounded in illustrative quotations. Subsequently, the modules were iteratively refined based on this feedback.

    Ethical Considerations

    Ethics approval was obtained from the University of Sydney Human Research Ethics Committee (protocol 2015/468). Participants provided informed consent and were given the opportunity to opt out at any point in time. Participant data were deidentified. Participants were provided a gift card worth Aus $20 (US $13.22) as compensation for their time.


    This section describes the clinician, patient, and carer feedback; iterative revisions made; and lessons learned in the design and development of the eTRIO and eTRIO-pc modules.

    Phase 1: Development of eTRIO and eTRIO-pc Module Content

    eTRIO Clinician Module

    The clinician advisory group members (n=13) emphasized the importance of the module being concise. They suggested more content for the introductory section such as including a broad and inclusive definition for “carers,” content about culturally diverse carers, and more information about the legal and ethical aspects of involving carers. Clinicians also suggested the inclusion of self-reflections about one’s own attitudes and potential biases toward carers. Additional suggestions included addressing the diversity of settings in which family or carer interactions can occur (eg, outside traditional outpatient consultations such as at the patient’s bedside or via the telephone). Several clinicians stressed the importance of including clear learning outcomes and summaries for each of the 14 sections.

    eTRIO-pc Patient-Carer Module

    Cancer support staff (n=5) suggested a clear definition of the role of carers, tailoring based on the cultural backgrounds of patients and carers, and consideration of power imbalances that may exist in patient-carer relationships. They emphasized checking in on patient and carer emotions such as grief and distress, suggested that modules could include opportunities for self-reflection, and highlighted the need to include information about available support for carers.

    The overall impression of the consumer advisory group (n=5) was that the language and tone of the draft module was very formal and academic; they wanted the tone to be more “personal,” “empathetic,” and “softer” and the language to be less prescriptive. They suggested additional strategies for patients with newly diagnosed cancer and carers, such as making notes during medical consultations, and suggested including quotes and stories from actual carers to illustrate examples.

    Phase 2: Iterative Design, Review, and Refinement of eTRIO and eTRIO-pc Web-Based Modules

    Overview

    Table 1 describes the results from phase 2 using the e-learning design features by de Leeuw et al [38] applied to the eTRIO and eTRIO-pc modules.

    Table 1. e-Learning design features identified by de Leeuw et al, as applied to the e-Triadic Oncology (eTRIO) and e-Triadic Oncology for patients and carers (eTRIO-pc) modules.
    Elements of e-learningDescriptionUse in eTRIO and eTRIO-pc
    PreparationIdentifying the needs of the target audience
    • Th research team conducted an extensive program of previous studies on the needs of carers
    • Stakeholder input, feedback, and evaluation
    DesignIncluding elements of accessibility, reliability, user-friendly navigation, and visual appeal
    • Web-based program, simple layout, and designed for easy use
    • Font, color, size, and layout are optimized for accessibility
    • User progress is saved when users log out
    • Website is designed and tested on various software and hardware
    CommunicationCommunication with users and program facilitators
    • Landing page introduces users to the learning objectives and goals (ie, communication skills and strategies, understanding carer roles, and benefits of carer involvement in cancer care)
    • Clear information about program use and navigation is included
    ContentIncluding words, images, videos, interactive activities, summaries, and so on
    • All modules include multimedia content such as several clinical vignette videos, audios, text, images, and interactive features. Interactive activities were designed, including the following:
      • eTRIO (clinician): sorting and drag-and-drop activities, true-or-false exercises, open-text written responses, click-to-expand sections, and identifying behaviors in a vignette video
      • eTRIO-pc (patient-carer): resources that can be individually tailored (eg, assembling a caregiving team, building a question prompt list, and checklist for patients and carers to discuss carer role), click-to-expand sections, and open-text written responses
    • Downloadable summaries are provided to allow access after completing or outside the module
    AssessmentAssessing learning and acquiring feedback
    • Each section of each module contains clear learning objectives, displayed on the first page of each section. For example, section 9 of eTRIO (clinician), related to the use of interpreters, states the following: “In this section you will explore reasons why patients/carers might resist professional language interpreters, and understand strategies to overcome these issues. You will learn practical strategies to engage and use formal interpretation services.”
    • All modules include assessment activities to facilitate learning and reflection:
      • eTRIO (clinician): self-reflection and assessment of own attitudes and practices, true-or-false assessment of content with correct answers and explanations, multiple choice questions asking users to reflect about how they would navigate a clinical scenario, and open-ended responses
      • eTRIO-pc (patient-carer): self-assessment of emotions, opportunities to reflect about own preferences and attitudes and to plan future actions or behaviors, and open-text reflections about video vignettes modeling key skills
    MaintenanceProviding long-term access and updating information and links
    • Website is regularly maintained and updated
    • All users will have access to the program after completion of the training
    eTRIO Clinician Module

    During the transformation of content to a web-based module, features of e-learning [38] were applied as described in Table 1. The design features of other web-based clinician training modules were examined, revealing display, navigation, and interactive activity styles (eg, minimal use of text, prominent navigation buttons, and clickable and expandable content). Our team worked closely with graphic and web designers to develop a consistent color scheme and intuitive navigation system and aimed to minimize visual noise on each page. The refined content and design features were transformed into a web-based web platform.

    All members of the clinician advisory group (13/13, 100%) commented that there was excessive content and that there would not be clinician appetite for web-based training that extended beyond 2 hours in total. The content was subsequently condensed, with the core content displayed with the option of more extensive content, which could be expanded for clinicians interested in deeper learning regarding an issue.

    The final eTRIO clinician module comprises 14 sections (submodules), of which clinicians must complete a minimum of 8. The sections range between 3 and 15 minutes in duration. The following 4 sections were deemed to be mandatory by the clinician advisory group, based on their critical relevance to all clinicians: section 1—introduction, section 4—building rapport with carers, section 7—supporting carers’ emotional and informational needs, and section 10—managing conflicting patient-carer treatment preferences. Clinicians could select additional 4 sections based on their interest and preference. The eTRIO module requires approximately 1.5 to 2 hours to complete, as determined by multiple stakeholders working through the content and documenting the amount of time each section required to complete.

    eTRIO-pc Patient-Carer Module

    Consistent with the principles of computer-based teaching for adult learners by Lau [51], the web-based eTRIO-pc module was created by transforming the written content into interactive, engaging learning activities. Our review of carer resources demonstrated several useful stylistic, formatting, and usability features, for example, the use of bullet points to convey written information, 1-page displays eliminating the need to scroll, and use of simple navigation buttons. These features and principles of web-based education were collated and discussed with the team’s academic IT expert and web developers to select and finalize the most appropriate features to be included. The resultant module prototype included video vignettes that could easily be played and paused, interactive activities such as “drag-and-drop” and “click to reveal” exercises, and type-your-response activities (Multimedia Appendix 1). We maintained consistency in design and formatting across the clinician, patient, and carer modules.

    We sent the prototype to the members of the consumer advisory group (n=5), and they provided written feedback via email and offered additional personal quotes that could be included in the module to personalize the content. They re-emphasized the need for content that was empathetic and offered practical advice. The final eTRIO-pc modules contain 7 sections for patients and 11 sections for carers and requires approximately 1 to 1.5 hours to complete.

    Phase 3: “Think Aloud” Usability Evaluation of eTRIO and eTRIO-pc Web-Based Modules

    Heuristic Evaluation

    Using the heuristic evaluation method [44], we identified 37 usability issues across the draft eTRIO and eTRIO-pc modules, and each was rated for severity. The main areas of the identified problems were as follows: (1) inconsistency of icons and redundancy in buttons (5/37, 14% of the issues; eg, inconsistent use of star and book icons to indicate the bookmark function), (2) buttons and interactions were not working (16/37, 43% of the issues; eg, nothing happens when the print button is clicked), (3) layout problems (6/37, 16% of the issues; eg, text is not aligned with the textbox), and (4) presentation of content (10/37, 27% of the issues; eg, color selection in the bar-slider activity may be confusing) [52]. Following this evaluation, problems with high severity and prevalence were prioritized, and all issues that could be corrected were fixed before conducting the think-aloud user evaluations.

    Think-Aloud User Experience Evaluations

    Overall, 11 individuals (n=5, 45% health professionals; n=3, 27% patients; and n=3, 27% carers) participated in the think-aloud evaluations in individual sessions lasting between 40 and 60 minutes. Participant characteristics are displayed in Table 2.

    Table 2. Characteristics of participants of the think-aloud evaluations.
    Participant category and characteristicsValues
    Health professionals (n=5)

    		Age (y), mean (SD; range)47 (10.3; 35-58)

    		Sex, n (%)


    		Female4 (80)


    		Male1 (20)

    		Profession, n (%)


    		Physician2 (40)


    		Nurse3 (60)

    		Clinical expertise, n (%)


    		Oncology2 (40)


    		Palliative care2 (40)


    		Geriatrics1 (20)

    		Experience (years), mean (SD; range)22 (9.8; 12-37)
    Patients (n=3)

    		Age (y), mean (SD; range)65 (13.7; 50-77)

    		Sex (female), n (%)3 (100)

    		Diagnosis, n (%)


    		Kidney cancer1 (33)


    		Colorectal cancer1 (33)


    		Non-Hodgkins lymphoma1 (33)

    		Cancer stage, n (%)


    		Local2 (67)


    		Advanced1 (33)

    		Health literacy, n (%)


    		Low1 (33)


    		Medium1 (33)


    		High1 (33)
    Carers (n=3)

    		Age (y), mean (SD; range)65 (8.7; 58-75)

    		Sex, n (%)


    		Female2 (67)


    		Male1 (33)

    		Relationship with care recipient, n (%)


    		Spouse or partner2 (67)


    		Mother1 (33)

    		Diagnosis of care recipient, n (%)


    		Lung cancer1 (33)


    		Multiple myeloma1 (33)


    		Non-Hodgkins lymphoma1 (33)

    		Cancer stage of care recipient, n (%)


    		Local1 (33)


    		Advanced2 (67)

    		Health literacy, n (%)


    		Medium1 (33)


    		High2 (67)
    eTRIO Clinician Module

    The usability of the module was rated by 5 clinicians, with a mean SUS score of 75 (range 68-80), which is interpreted as good [49]. All clinicians gave high ratings to their ability to use the module independently without technical assistance. Clinicians identified technical and navigation issues, which were subsequently rectified (such as the side scroll bar not appearing, text appearing outside the text bubble, and a sliding bar not working responsively). For some, the use of specific web browsers corrected these issues. Clinicians described the overall navigation through the module as “straightforward.” Formatting issues with font size and background color were highlighted. Clinicians commented that the ability to easily navigate back to certain sections to “refer back to later” was valued.

    Content analysis of think-aloud evaluations revealed 7 categories related to clinicians’ attitudes toward the design and formatting of eTRIO. Clinicians appreciated that the modules could be completed in small “snackable” periods in any order, that they could keep track of what sections were completed (trackable), and that they were able to refer back to any module at any time. Clinicians enjoyed the “clickable” activities where they interacted with the content. Despite attempts to make the sections as short as possible (average 5-10 min/section), a few clinicians still perceived them as “too long,” with some stating that the videos were “slow” at times. They highlighted a preference for material that is brief, uses simple language, is easy to digest, and “skimmable.” A few clinicians reported “glossing over” or “tuning out” when sections were perceived as very long. They suggested simplifying the language and formatting the text to highlight important information (eg, use of bullet points and bold and italic style). Revisions were made to the text to further improve conciseness, including rephrasing the core content, moving some content to the expandable ‘additional information’ section, and greater use of bullet points and bold text. Where possible, videos were edited to remove nonessential scenes. Most participants appreciated that the content and activities were relevant and “relatable” to them as clinicians, that claims were “supported” by evidence, and that the activities and media were “diverse” and varied to facilitate engagement and interest. Illustrative quotes are provided in Table 3.

    Table 3. Illustrative quotes from think-aloud evaluations by clinicians.
    Usability and content featureDescriptionIllustrative quotes
    SnackableAbility to complete the module in small segments
    • “So, you’re saying you don’t have to do it all in one go...oh, I think that’s really important because you do get called away and the phone is ringing...because I know even with our mandatory online training in the past, you just [had to] forfeit [all progress] if you couldn’t finish.” [Nurse 2]
    TrackableAbility to know what has been completed and refer to the content later
    • “It’s nice to have things you can refer back to because this might trigger things that make you think oh yeah, I did read about that.” [Physician 2]
    ClickableImportance of interactive content
    • “I like this section - it’s really good. I like that activity. I’ve never done one of those before - that’s really good. [Interactive activity clicking points of rapport building throughout a video vignette]. You definitely engage a thousand percent more with the activities.” [Physician 2]
    • “I think [the activities] are quite good because at least you are giving people a little bit more of themselves...I think it’s good to have that interaction rather than just reading...that gets a bit boring. And then, that you ask people to actually write something is good.” [Nurse 2]
    SkimmableImportance of simple, concise language
    • “After reading articles all day I don’t want to read something that has too much jargon in it...Go back and simplify the language...when I read something apart from patients notes, I skim it. So, it’s got to be something that I can get the message with a glance.” [Physician 1]
    • “Uhm why I am I finding it difficult to understand? I think it could be worded more simply.” [Physician 2]
    • “Yeah. I hate the time pressure...It’s so built into our working day, it’s like get on, get it done, that you gloss over so much. I actually didn’t realize before doing this how much I gloss over...I probably would watch [the video] to the end but there’s a part of me thinking yeah it’s going on a little bit.” [Physician 2]
    RelatableRelevance of content to the user
    • “Yeah, I like that there is the suggestions of things to say. That makes it really relatable - I think those are good.” [Physician 2]
    • “I like scenarios...Just sort of triggers you to think a little bit more rather than just reading through something. I think the scenario allows me to put it into practice or put it into place a little bit more.” [Nurse 3]
    SupportedEvidence-based content
    • “I like the use of the quotes. It gives a bit of a supportive evidence to it, as nurse I like that...It has got some stats [statistics] there...When you hover over it...it gives the reference.” [Nurse 3]
    DiverseImportance of variety in media and activities
    • “Oh a video, that’s interesting, it’s sort of mixing it up, it’s nice to have the different things.” [Physician 2]
    eTRIO-pc Patient and Carer Module

    The average SUS score as rated by 6 patients and carers was 78 (SD 16.2; range 55-97.5), which is interpreted as good [49]. Patients and carers were generally happy with the content and usability of the eTRIO-pc module. They commented that the content was relatable and were pleased by the emphasis placed on carers. Overall, they found the web platform easy to navigate and enjoyed the interactive activities; however, 1 (17%) of the 6 patients found the interface to be “overwhelming.” A major critique of the formatting and layout was that the pages were “too busy” and contained excessive information. Illustrative quotes are provided in Table 4.

    The final eTRIO and eTRIO-pc modules were updated based on this feedback. All technical and navigation issues were addressed by the web developers.

    For both modules, the text was condensed and reformatted with the use of bold and italic style to highlight the important points and allow for easier reading and a more streamlined user interface.

    Table 4. Illustrative quotes from think-aloud evaluations by patients and carers.
    Usability and content featureDescriptionIllustrative quotes
    SnackableAbility to complete the module in small segments
    • “Looking at this dashboard I like it that it tells you how long each part is going to take just so you know in advance. You’re busy and maybe you just have time to do half of it and then you can sort of plan how you’re going to tackle it.” [Carer 1]
    ClickableImportance of interactive content
    • “Some of the activities like the questions, I really liked. The ones where you wrote down what you thought the carer might do for you if you then use it as a communication tool, really good as well.” [Patient 1]
    UsableEase of navigation
    • “I think [navigation] is pretty easy and straight forward. I think anybody who’s used to doing online training, modules and so on will probably find it really easy.” [Carer 1]
    RelatableRelevance of content to the user
    • “I think this is a very useful slide. When we went in to our first meeting we were just there, me and my son did this.” [Carer 1]
    Visually simpleCleanness of layout, formatting, and images
    • “I think you are making this page very busy with text and it’s a bit confronting.” [Carer 2]
    • “It is pretty text-heavy and I guess that I am more of a visual learner so it might be nice to have some more pictures, icons, to make it a little bit more visually appealing.” [Patient 1]
    Final Web Platform Design and Content Summary

    The eTRIO modules reflect the reported informational needs of health professionals, patients with cancer, and carers. A full description of the module content has been published elsewhere [29]. The eTRIO modules have been rigorously designed to be easy to use, require minimal time commitment, and be flexible in terms of when and how the platform can be used. The modules are optimized for use on a computer but can also be used on a smartphone or tablet. Some notable features include the following: navigation buttons and a progress bar along the bottom of the page, expandable content for those who want deeper information about a specific topic, and downloadable summaries and lists. Notable interactive activities include the following: testing of knowledge through true-or-false exercises, identifying specific behaviors in a short video vignette, and building a question prompt list. Refer to Table 5 for descriptions and images of key features; full explanations of the interactive activities are provided in Multimedia Appendices 1 and 2.

    Table 5. Key features of the e-Triadic Oncology (eTRIO) modules.
    FeatureDescriptionImages
    Interactive activitiesIncludes self-reflection, knowledge tests, and free-text responses
    Learning outcomesThe eTRIO clinician module features signposted learning outcomes at the beginning of each section
    Downloadable contentIncludes materials and personalized checklists for patients and carers and downloadable summaries for clinicians and
    Video vignettesRealistic scenarios modeling communication skills
    Intuitive navigation featuresNavigation buttons are explained in the module’s introduction

    Principal Findings

    The web-based modules described in this paper represent a crucial step in the development and design of education for clinicians, patients, and carers that is evidence based, practical, and interactive and can be easily disseminated. Drawing on the evidence for best practice web-based learning design [38,51], we sought input from a variety of stakeholders to develop a unique learning experience strongly informed by the needs of the target populations. Rigor was ensured via 3 stages of development in which module content and design were continually revised and refined. Overall, participants were positive about the content and interface. The final prototype was appraised as highly acceptable, relevant, and feasible among the small sample of users; however, more studies are needed to confirm this and to ascertain the effectiveness of the intervention. We are currently conducting a pre-post evaluation of these modules to explore their potential effectiveness in improving communication within the patient-carer-clinician trio.

    Lessons Learned

    Throughout the development and design of these modules, we observed the specific needs and preferences of end users. The person-based approach to developing eTRIO and eTRIO-pc was highly dynamic, and the modules underwent numerous iterations throughout all phases of the design process, which included the involvement of consumers and user-driven evaluations. While there are multiple approaches to developing health interventions, the benefits of the person-based approach include grounding the design in user contexts and lived experiences, integrating feedback based on the actual use of an intervention, and investigating user needs and perspectives beyond just the usability of the intervention [26]. The utility of the person-based approach has been extolled in recent studies [53-55] and is supported by the findings of this study. The eTRIO development process (Figures 1 and 2) provided the necessary building blocks to revise and refine the module for effective use in the real world. Consistent with other studies [56,57], we found that the collaborative co-design process led to positive evaluations of acceptability and usability and high levels of end-user satisfaction.

    As highlighted in the person-based approach, the 3 user groups (clinicians, patient, and carers) demonstrated diverse learning preferences and needs. This was accommodated via tailoring the formatting or content to the strengths and contextual demands of different user groups and differentiating the content based on user needs. We found that clinicians had a strong desire for content that was written in simple, concise, and “sharp” language; could be “skim read”; and could be completed in brief, “snack-sized” sections. For example, clinicians in our advisory group often stressed that they lacked time and that training needed to be short, precise, and able to be stopped and restarted due to interruptions. On the other hand, the structure and time demands of training appeared to be less important to patients and carers. Instead, these groups emphasized the need for the module to be easy to use and navigate and for the content to be more conversational, empathetic, and in plain language (in contrast to the preferences of clinicians). Clinicians in our study valued the integration of academic literature and referencing, whereas some carers advocated for greater inclusion of carer experiences and quotes. The preferences of carers in our study are consistent with previous studies, which have similarly found that carers often prefer web-based education to have an empathetic and supportive tone, the web program to be easy to navigate, and the integration of other carers’ experiences into the content [58-60]. While several differences were identified between the clinician and carer user groups, there were also several similarities across all user groups in how the web-based modules should be structured and delivered. This is reflected in the evidence base, where health professionals, patients, and carers alike report that they prefer flexible, self-paced delivery of web-based programs that are interactive and include a variety of activities across media (visual, written, and auditory) [19,38]. These detailed insights are valuable in designing future training modules to facilitate their acceptability among users in each specific group.

    The final interface used design principles to ensure engaging and interactive content. There is robust empirical evidence suggesting that interactivity in e-learning improves quality, efficacy, and learning outcomes [38,61]. For example, users of a web-based public health program had better learning outcomes when they used a gamified, interactive version featuring responsive design, learning challenges, visible progress, and rapid feedback compared to those using a minimally interactive, survey-based program [62]. Such interactivity was also demonstrated as important for users of the eTRIO modules. For example, in the initial design phases, when content was largely text based, the advisory committee members noted how dense the information appeared. While this was never intended to be the final format of the educational intervention, comments obtained from users in phase 1 highlighted the limitations of passive, didactic, text-heavy information. There is evidence suggesting that people do not learn effectively when information is given without any opportunity to reflect on, test, or demonstrate their knowledge and views [63]. Interactive activities, including assessments of learning and personal reflection activities, offer users the opportunity to reflect and reinforce their learning and become active participants in their education rather than passive consumers of information. Multimedia Appendices 1 and 2 display the engaging interactive activities that were acceptable to eTRIO users, which may be used in other web-based learning interventions and resources.

    For both the clinician and patient-carer modules, we also incorporated a variety of media (text, audio, video, graphics, and images to cater to different learning styles and preferences. There is evidence suggesting that the use of multimedia may increase user satisfaction, acceptability, and engagement [64,65] and thus may improve adherence and broad implementation. The modules were designed such that users could navigate through them at their own pace and read, view, and explore the sections in a self-directed manner based on how they like to engage with and process content. For example, we found that users had mixed responses to the videos embedded in the training module. Some users commented that the videos were very long and that they would mentally switch off or skip them. Others claimed to be “visual learners” and thoroughly enjoyed the opportunity to observe scenarios in this format, especially because the videos included interactive “trigger” questions such as “What would you do next?” where they were required to apply some of their learning to a scenario. This approach has been used in other web-based health interventions [66,67], which include complementary text, images, videos, audios, and interactive content to convey the educational content and cater to these diverse user preferences.

    Strengths and Limitations

    A thoughtful process of iterative design was conducted over a 2-year period, ultimately producing a suite of web-based interventions intended to improve communication between cancer clinicians, patients, and carers. However, important limitations should be noted. While extensive end-user feedback was collected through iterative feedback from clinician and consumer advisory groups, the sample size of participants (patients with cancer and carers) naive to the modules in phase 3 was small, and there was limited diversity among consumer advisers and participants. In addition, we did not measure the computer literacy of the participants, which may have impacted their views about the program’s usability. Thus, the attitudes and preferences of participants may not be reflective of the wider population. For example, we were unable to recruit a carer with low health literacy, and there was an overrepresentation of women.

    Further usability and acceptability testing is currently underway in a larger study with a more diverse sample of patients and carers. Recruitment of participants in phase 3 was conducted through professional networks and social media, and therefore, the participants may have had a strong interest web-based learning or carer communication, which could have biased their views. This study focused only on development and user testing, and therefore, no assessment of the effectiveness or uptake of the modules has been conducted. Larger evaluation studies of the modules are currently being conducted, which will provide insight into the utility of the eTRIO modules in improving carer-related communication and inclusion.

    Finally, while most patients with cancer have a carer or support person, some patients do not. Further studies are required to better understand the needs of people without a carer, which is beyond the scope of this study.

    Future Directions

    The eTRIO and eTRIO-pc modules are now undergoing pre-post evaluation with additional qualitative learner feedback to inform the broad implementation and uptake of these educational resources.

    Conclusions

    By including and being receptive to the needs of our user groups throughout the design process, we were able to create interventions that end users are likely to be more engaged and satisfied with.

    Acknowledgments

    This study was supported by Cancer Australia and Cancer Council New South Wales Grant, through the Priority-Driven Collaborative Cancer Research Scheme (project 1146383).

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    e-Triadic Oncology patient and carer module features.

    DOCX File , 1304 KB
    Multimedia Appendix 2

    e-Triadic Oncology clinician module features.

    DOCX File , 1069 KB
    1. Castro A, Arnaert A, Moffatt K, Kildea J, Bitzas V, Tsimicalis A. "Informal caregiver" in nursing: an evolutionary concept analysis. ANS Adv Nurs Sci. 2023;46(1):E29-E42. [CrossRef] [Medline]
    2. Molassiotis A, Wang M. Understanding and supporting informal cancer caregivers. Curr Treat Options Oncol. Apr 2022;23(4):494-513. [FREE Full text] [CrossRef] [Medline]
    3. Geng HM, Chuang DM, Yang F, Yang Y, Liu WM, Liu LH, et al. Prevalence and determinants of depression in caregivers of cancer patients: a systematic review and meta-analysis. Medicine (Baltimore). Sep 2018;97(39):e11863. [FREE Full text] [CrossRef] [Medline]
    4. Bedaso A, Dejenu G, Duko B. Depression among caregivers of cancer patients: updated systematic review and meta-analysis. Psychooncology. Nov 2022;31(11):1809-1820. [FREE Full text] [CrossRef] [Medline]
    5. Lambert S, Girgis A, Descallar J, Levesque JV, Jones B. Trajectories of mental and physical functioning among spouse caregivers of cancer survivors over the first five years following the diagnosis. Patient Educ Couns. Jun 2017;100(6):1213-1221. [CrossRef] [Medline]
    6. Lambert SD, Girgis A. Unmet supportive care needs among informal caregivers of patients with cancer: opportunities and challenges in informing the development of interventions. Asia Pac J Oncol Nurs. Apr 2017;4(2):136-139. [FREE Full text] [CrossRef] [Medline]
    7. Høeg BL, Frederiksen MH, Andersen EA, Saltbæk L, Friberg AS, Karlsen RV, et al. Is the health literacy of informal caregivers associated with the psychological outcomes of breast cancer survivors? J Cancer Surviv. Oct 2021;15(5):729-737. [CrossRef] [Medline]
    8. Kershaw T, Ellis KR, Yoon H, Schafenacker A, Katapodi M, Northouse L. The interdependence of advanced cancer patients' and their family caregivers' mental health, physical health, and self-efficacy over time. Ann Behav Med. Dec 2015;49(6):901-911. [FREE Full text] [CrossRef] [Medline]
    9. Laidsaar-Powell R, Butow P, Bu S, Fisher A, Juraskova I. Oncologists' and oncology nurses' attitudes and practices towards family involvement in cancer consultations. Eur J Cancer Care (Engl). Jan 01, 2017;26(1):e12470. [CrossRef] [Medline]
    10. Røen I, Stifoss-Hanssen H, Grande G, Kaasa S, Sand K, Knudsen AK. Supporting carers: health care professionals in need of system improvements and education - a qualitative study. BMC Palliat Care. Jul 16, 2019;18(1):58. [FREE Full text] [CrossRef] [Medline]
    11. Laidsaar-Powell R, Butow P, Bu S, Dear R, Fisher A, Coll J, et al. Exploring the communication of oncologists, patients and family members in cancer consultations: development and application of a coding system capturing family-relevant behaviours (KINcode). Psychooncology. Jul 30, 2016;25(7):787-794. [CrossRef] [Medline]
    12. McCarthy B. Family members of patients with cancer: what they know, how they know and what they want to know. Eur J Oncol Nurs. Dec 2011;15(5):428-441. [CrossRef] [Medline]
    13. Morris S, Thomas C. The carer's place in the cancer situation: where does the carer stand in the medical setting? Eur J Cancer Care (Engl). Jun 2001;10(2):87-95. [CrossRef] [Medline]
    14. Laidsaar-Powell R, Butow P, Bu S, Charles C, Gafni A, Fisher A, et al. Family involvement in cancer treatment decision-making: a qualitative study of patient, family, and clinician attitudes and experiences. Patient Educ Couns. Jul 2016;99(7):1146-1155. [CrossRef] [Medline]
    15. Berg M, Ngune I, Schofield P, Grech L, Juraskova I, Strasser M, et al. Effectiveness of online communication skills training for cancer and palliative care health professionals: a systematic review. Psychooncology. Sep 2021;30(9):1405-1419. [FREE Full text] [CrossRef] [Medline]
    16. George PP, Zhabenko O, Kyaw BM, Antoniou P, Posadzki P, Saxena N, et al. Online digital education for postregistration training of medical doctors: systematic review by the digital health education collaboration. J Med Internet Res. Feb 25, 2019;21(2):e13269. [FREE Full text] [CrossRef] [Medline]
    17. Laidsaar-Powell R, Keast R, Butow P, Mahony J, Hagerty F, Townsend J, et al. Improving breast cancer nurses' management of challenging situations involving family carers: pilot evaluation of a brief targeted online education module (TRIO-Conflict). Patient Educ Couns. Dec 2021;104(12):3023-3031. [CrossRef] [Medline]
    18. Keast R, Butow PN, Juraskova I, Laidsaar-Powell R. Online resources for family caregivers of cognitively competent patients: a review of user-driven reputable health website content on caregiver communication with health professionals. Patient Educ Couns. Dec 2020;103(12):2408-2419. [CrossRef]
    19. Heynsbergh N, Heckel L, Botti M, Livingston PM. Feasibility, useability and acceptability of technology-based interventions for informal cancer carers: a systematic review. BMC Cancer. Mar 02, 2018;18(1):244. [FREE Full text] [CrossRef] [Medline]
    20. Zhai S, Chu F, Tan M, Chi NC, Ward T, Yuwen W. Digital health interventions to support family caregivers: an updated systematic review. Digit Health. May 19, 2023;9:20552076231171967. [FREE Full text] [CrossRef] [Medline]
    21. Grossert A, Urech C, Alder J, Gaab J, Berger T, Hess V. Web-based stress management for newly diagnosed cancer patients (STREAM-1): a randomized, wait-list controlled intervention study. BMC Cancer. Nov 03, 2016;16(1):838. [FREE Full text] [CrossRef] [Medline]
    22. DuBenske LL, Gustafson DH, Namkoong K, Hawkins RP, Atwood AK, Brown RL, et al. CHESS improves cancer caregivers' burden and mood: results of an eHealth RCT. Health Psychol. Oct 2014;33(10):1261-1272. [FREE Full text] [CrossRef] [Medline]
    23. Kaltenbaugh D, Klem M, Hu L, Turi E, Haines AJ, Hagerty Lingler J. Using Web-based interventions to support caregivers of patients with cancer: a systematic review. Oncol Nurs Forum. Mar 2015;42(2):156-164. [CrossRef] [Medline]
    24. Northouse LL, Katapodi MC, Song L, Zhang L, Mood DW. Interventions with family caregivers of cancer patients: meta-analysis of randomized trials. CA Cancer J Clin. Aug 16, 2010;60(5):317-339. [FREE Full text] [CrossRef] [Medline]
    25. Ugalde A, Gaskin CJ, Rankin NM, Schofield P, Boltong A, Aranda S, et al. A systematic review of cancer caregiver interventions: appraising the potential for implementation of evidence into practice. Psychooncology. Apr 07, 2019;28(4):687-701. [FREE Full text] [CrossRef] [Medline]
    26. Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res. Jan 30, 2015;17(1):e30. [FREE Full text] [CrossRef] [Medline]
    27. Beatty L, Koczwara B, Butow P, Turner J, Girgis A, Schofield P, et al. Development and usability testing of a web-based psychosocial intervention for women living with metastatic breast cancer: finding my way-advanced. J Cancer Surviv. Jun 15, 2021;15(3):403-409. [CrossRef] [Medline]
    28. Wagner LI, Duffecy J, Begale M, Victorson D, Golden SL, Smith ML, et al. Development and refinement of FoRtitude: an eHealth intervention to reduce fear of recurrence among breast cancer survivors. Psychooncology. Jan 08, 2020;29(1):227-231. [FREE Full text] [CrossRef] [Medline]
    29. Juraskova I, Laidsaar-Powell R, Keast R, Schofield P, Costa DS, Kay J, et al. eTRIO trial: study protocol of a randomised controlled trial of online education modules to facilitate effective family caregiver involvement in oncology. BMJ Open. May 28, 2021;11(5):e043224. [FREE Full text] [CrossRef] [Medline]
    30. Gould JD, Lewis C. Designing for usability: key principles and what designers think. Commun ACM. 1985;28(3):300-311. [CrossRef]
    31. Laidsaar-Powell R, Butow P, Bu S, Charles C, Gafni A, Lam W, et al. Physician-patient-companion communication and decision-making: a systematic review of triadic medical consultations. Patient Educ Couns. Apr 2013;91(1):3-13. [CrossRef] [Medline]
    32. Laidsaar-Powell R, Butow P, Bu S, Fisher A, Juraskova I. Attitudes and experiences of family involvement in cancer consultations: a qualitative exploration of patient and family member perspectives. Support Care Cancer. Oct 30, 2016;24(10):4131-4140. [CrossRef] [Medline]
    33. Laidsaar-Powell R, Butow P, Charles C, Gafni A, Entwistle V, Epstein R, et al. The TRIO framework: conceptual insights into family caregiver involvement and influence throughout cancer treatment decision-making. Patient Educ Couns. Nov 2017;100(11):2035-2046. [CrossRef] [Medline]
    34. Laidsaar-Powell R, Butow P, Boyle F, Juraskova I. Facilitating collaborative and effective family involvement in the cancer setting: guidelines for clinicians (TRIO Guidelines-1). Patient Educ Couns. Jun 2018;101(6):970-982. [CrossRef] [Medline]
    35. Laidsaar-Powell R, Butow P, Boyle F, Juraskova I. Managing challenging interactions with family caregivers in the cancer setting: guidelines for clinicians (TRIO Guidelines-2). Patient Educ Couns. Jun 2018;101(6):983-994. [CrossRef] [Medline]
    36. Cruz-Oliver DM, Pacheco Rueda A, Viera-Ortiz L, Washington KT, Oliver DP. The evidence supporting educational videos for patients and caregivers receiving hospice and palliative care: a systematic review. Patient Educ Couns. Sep 2020;103(9):1677-1691. [FREE Full text] [CrossRef] [Medline]
    37. Gysels M, Higginson IJ. Interactive technologies and videotapes for patient education in cancer care: systematic review and meta-analysis of randomised trials. Support Care Cancer. Jan 23, 2007;15(1):7-20. [CrossRef] [Medline]
    38. de Leeuw RA, Westerman M, Nelson E, Ket JC, Scheele F. Quality specifications in postgraduate medical e-learning: an integrative literature review leading to a postgraduate medical e-learning model. BMC Med Educ. Jul 08, 2016;16(1):168. [FREE Full text] [CrossRef] [Medline]
    39. Hillen MA, van Vliet LM, de Haes HC, Smets EM. Developing and administering scripted video vignettes for experimental research of patient-provider communication. Patient Educ Couns. Jun 2013;91(3):295-309. [CrossRef] [Medline]
    40. Scott KM, Baur L, Barrett J. Evidence-based principles for using technology-enhanced learning in the continuing professional development of health professionals. J Contin Educ Health Prof. 2017;37(1):61-66. [CrossRef] [Medline]
    41. Berry LL, Dalwadi SM, Jacobson JO. Supporting the supporters: what family caregivers need to care for a loved one with cancer. J Oncol Pract. Jan 2017;13(1):35-41. [CrossRef] [Medline]
    42. Sherwood PR, Given BA, Given CW. Caregiver Knowledge and Skills. In: Bellizzi KM, Gosney M, editors. Cancer and Aging Handbook: Research and Practice. Hoboken, NJ. Wiley-Blackwell; 2012;445-458.
    43. Connor KI, Siebens HC, Chodosh J. Person-centered approaches to caregiving. In: Gaugler JE, Kane RL, editors. Family Caregiving in the New Normal. San Diego, CA. Academic Press; 2015;251-268.
    44. Alonso-Ríos D, Mosqueira-Rey E, Moret-Bonillo V. A systematic and generalizable approach to the heuristic evaluation of user interfaces. Int J Hum Comput Interact. Jan 24, 2018;34(12):1169-1182. [CrossRef]
    45. Nielsen J. Enhancing the explanatory power of usability heuristics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1994. Presented at: CHI '94; April 24-28, 1994;152-158; Boston, MA. URL: https://dl.acm.org/doi/10.1145/191666.191729 [CrossRef]
    46. Duncker K, Lees LS. On problem-solving. Psychol Monogr. 1945;58(5):1-113. [CrossRef]
    47. Nielsen J. Usability Engineering. Cambridge, MA. Morgan Kaufmann; 1994.
    48. Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med. Sep 2004;36(8):588-594. [FREE Full text] [Medline]
    49. Brooke J. SUS: A 'Quick and Dirty' Usability Scale. Boca Raton, FL. CRC Press; 1995.
    50. Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qual Res Sport Exerc Health. Jun 13, 2019;11(4):589-597. [CrossRef]
    51. Lau KH. Computer-based teaching module design: principles derived from learning theories. Med Educ. Mar 2014;48(3):247-254. [CrossRef] [Medline]
    52. Bonnaudet M. A usability evaluation of TRIO's e-learning modules enhancing the communication between cancer patients, clinicians and carers. KTH Royal Institute of Technology. 2020. URL: https://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A1466296&dswid=-4538 [accessed 2024-02-29]
    53. Murfield J, Moyle W, O'Donovan A. Planning and designing a self-compassion intervention for family carers of people living with dementia: a person-based and co-design approach. BMC Geriatr. Jan 14, 2022;22(1):53. [FREE Full text] [CrossRef] [Medline]
    54. O'Cathain A, Croot L, Sworn K, Duncan E, Rousseau N, Turner K, et al. Taxonomy of approaches to developing interventions to improve health: a systematic methods overview. Pilot Feasibility Stud. Mar 12, 2019;5(1):41. [FREE Full text] [CrossRef] [Medline]
    55. Band R, Bradbury K, Morton K, May C, Michie S, Mair FS, et al. Intervention planning for a digital intervention for self-management of hypertension: a theory-, evidence- and person-based approach. Implement Sci. Feb 23, 2017;12(1):25. [FREE Full text] [CrossRef] [Medline]
    56. Yardley L, Ainsworth B, Arden-Close E, Muller I. The person-based approach to enhancing the acceptability and feasibility of interventions. Pilot Feasibility Stud. Oct 26, 2015;1(1):37. [FREE Full text] [CrossRef] [Medline]
    57. Bradbury K, Morton K, Band R, van Woezik A, Grist R, McManus RJ, et al. Using the person-based approach to optimise a digital intervention for the management of hypertension. PLoS One. 2018;13(5):e0196868. [FREE Full text] [CrossRef] [Medline]
    58. Köhle N, Drossaert CH, Oosterik S, Schreurs KM, Hagedoorn M, van Uden-Kraan CF, et al. Needs and preferences of partners of cancer patients regarding a web-based psychological intervention: a qualitative study. JMIR Cancer. Dec 29, 2015;1(2):e13. [FREE Full text] [CrossRef] [Medline]
    59. Vaughan C, Trail TE, Mahmud A, Dellva S, Tanielian T, Friedman E. Informal caregivers' experiences and perceptions of a web-based peer support network: mixed-methods study. J Med Internet Res. Aug 28, 2018;20(8):e257. [FREE Full text] [CrossRef] [Medline]
    60. Teles S, Ferreira A, Paúl C. Attitudes and preferences of digitally skilled dementia caregivers towards online psychoeducation: a cross-sectional study. Behav Inf Technol. Jan 09, 2022;42(4):345-359. [CrossRef]
    61. Chang V. Review and discussion: E-learning for academia and industry. Int J Inf Manag. Jun 2016;36(3):476-485. [CrossRef]
    62. Trevors G, Ladhani F. It’s contagious! examining gamified refutation texts, emotions, and knowledge retention in a real-world public health education campaign. Discourse Process. Jun 27, 2022;59(5-6):401-416. [CrossRef]
    63. Karpicke JD, Roediger HL. Repeated retrieval during learning is the key to long-term retention. J Mem Lang. Aug 2007;57(2):151-162. [CrossRef]
    64. Zhang D. Interactive multimedia-based e-learning: a study of effectiveness. Am J Distance Educ. Sep 2005;19(3):149-162. [CrossRef]
    65. Violante MG, Vezzetti E. Virtual interactive e-learning application: an evaluation of the student satisfaction. Comput Appl Eng Educ. Aug 13, 2013;23(1):72-91. [CrossRef]
    66. Scott AF, Ayers S, Pluye P, Grad R, Sztramko R, Marr S, et al. Impact and perceived value of iGeriCare e-learning among dementia care partners and others: pilot evaluation using the IAM4all questionnaire. JMIR Aging. Dec 22, 2022;5(4):e40357. [FREE Full text] [CrossRef] [Medline]
    67. Heynsbergh N, Heckel L, Botti M, Livingston PM. A smartphone app to support carers of people living with cancer: a feasibility and usability study. JMIR Cancer. Jan 31, 2019;5(1):e11779. [FREE Full text] [CrossRef] [Medline]

    ‎
    CHESS: Comprehensive Health Enhancement Support System
    eTRIO: e-Triadic Oncology
    eTRIO-pc: e-Triadic Oncology for patients and carers
    STREAM: Stress-Aktiv-Mindern
    SUS: System Usability Scale
    TRIO: Triadic Oncology

    Edited by T de Azevedo Cardoso; submitted 20.06.23; peer-reviewed by A Castro, Y Asada; comments to author 29.09.23; revised version received 27.11.23; accepted 31.01.24; published 17.04.24.

    Copyright

    ©Rebekah Laidsaar-Powell, Sarah Giunta, Phyllis Butow, Rachael Keast, Bogda Koczwara, Judy Kay, Michael Jefford, Sandra Turner, Christobel Saunders, Penelope Schofield, Frances Boyle, Patsy Yates, Kate White, Annie Miller, Zoe Butt, Melanie Bonnaudet, Ilona Juraskova. Originally published in JMIR Medical Education (https://mededu.jmir.org), 17.04.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on https://mededu.jmir.org/, as well as this copyright and license information must be included.