Health professional programs frequently use simulation-based training as an instructional strategy to introduce new concepts, enrich learning, and provide trainees with feedback [1,2]. Educational simulations are faculty-designed instructional methods that provide learners with realistic, immersive clinical situations to develop decision-making skills [1]. The simulation may include human actors representing specific roles (eg, standardized patients [SPs] and health care professionals) or technologies that mimic people and the environment (eg, augmented reality [AR], virtual reality [VR], AI, and manikins). Simulations are an important bridge between the classroom and the bedside, providing a safe yet realistic environment for trainees to learn material, test strategies, and develop reasoning skills before real-world practice [3]. Educators typically design simulations around scenarios with a clinical challenge and a normative workflow. The scenarios, however, should be flexible and responsive, allowing learners to explore, adapt, and improvise [2,4]. It is also critical that simulations include context-sensitive feedback or guided reflection informed by educational theory and learning objectives [2].

To meet instructional objectives, experts and professional societies have published best-practice recommendations for simulation development [2,5-7]. These recommendations are largely conceptual and lack the granularity needed to guide design, testing, and implementation. Conceptual approaches to design alone do not ensure simulation quality [8,9]. It can be challenging to preserve realism and consistency given the many variables affecting case fidelity and the achievement of learning objectives [4]. Simulationists must ensure the accuracy of the scenario, environment, and digital or practical assets, such as medical equipment and synthetic patient data (ie, ecological validity) [9]. Educators must evaluate learners in ways that align with learning objectives (ie, content validity) [4,10]. Finally, learners need personalized feedback addressing the ability under assessment (ie, construct validity) [2,10]. Optimizing validity often requires educators to limit the number and types of discrepancies between the simulation and real-world clinical practice [11,12].

Discrepancies between the simulation and the real world, including design omissions (eg, incomplete scenarios, data, or objectives) or equipment issues, can limit the simulation’s usability. These problems can consume cognitive resources that learners would otherwise apply to pedagogical tasks [11]. It therefore behooves developers to rigorously test the simulation from each person’s perspective (eg, the learner, the instructor, and SPs) and correct usability flaws before piloting with intended learners [4]. We authored this tutorial to share our methods and demystify the testing process.

Simulation development guidelines recommend rehearsing cases with representative learners and collecting feedback from content experts, colleagues, technical teams, and participants [2,4,8,9]. Yet simulation guidelines rarely include methods for testing design features or collecting performance data before piloting [2,4,8,13]. Recently, human factors researchers have published reports trying to address this methodological gap by applying usability measurement techniques [8,14,15]. The ISO defines usability as the “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [16]. Utility is what a system allows users to do, whereas usability is how easily and efficiently users can access and operate these functionalities. Together, these qualities contribute to the overall user experience (ie, “users’ perceptions and responses resulting from the use and/or anticipated use of a system, product or service”).

Usability testing is an evaluation method in which users interact with a product under realistic conditions to determine how effectively, efficiently, and satisfactorily a set of tasks can be completed [17]. Da Silva and Dubrowski [8], for example, used mixed methods to improve 2 simulations over 5 iterations, demonstrating how usability testing findings can inform revisions. Still, their approach required large numbers of learners and relied, in part, on learner self-reports—an often unreliable proxy for educational effectiveness [18]. We outline here a pragmatic strategy and toolkit to address these limitations.

Our team at the University of Oklahoma designed a series of simulations to teach telemedicine competencies to residents providing online maternal health care. We adapted published usability methods to create a simulation evaluation protocol. Our protocol had 2 aims: to identify technical and content issues requiring immediate correction and to gauge the completeness and difficulty of the clinical tasks. We focused on the following three usability issue types: (1) technology failures that could disrupt workflow; (2) missing case details, patient-script language, or synthetic clinical data crucial for medical reasoning; and (3) faculty scoring guide instructions that might be ambiguous or difficult to apply.

Our high-level aim with this tutorial is to provide medical educators, human factors experts, and simulationists with tools and instructions for usability testing educational simulations. Our learning objectives are to (1) explain how we adapted usability methods for this context, (2) share our data collection instruments with readers to facilitate replication (Multimedia Appendices 1-8), and (3) demonstrate our methods with a use case to illustrate how we applied our results (Textbox 1).

While the objectives for usability testing are straightforward, we recommend articulating a priori research questions to guide protocol design and measure selection. Readers replicating our methods will likely formulate research questions relevant to their simulation. For our study, we had the following 3 questions:

Usability and user experience are not intrinsic properties of a product or a simulation; instead, they are emergent properties of the work system, arising from the interaction of multiple elements: technology, learners, instructors, tasks, environment, and organizational context [12,19]. This holistic perspective emphasizes that evaluating usability requires examining the entire work system, not the interface or equipment in isolation. Consequently, usability evaluation of simulations should aim to replicate realistic conditions regarding participants, task complexity, environmental constraints, and instructor support. Regarding educational simulations, this means evaluating not only the tools and technology but also faculty roles, faculty-learner interactions, and scenario details [20].

It is important to clarify how simulation usability can influence learners’ performance. Human-computer interaction shows that poor user experience increases the likelihood of errors [21-23]. Usability problems add cognitive load, forcing users to dedicate attention, working memory, and problem-solving capacity to compensate for suboptimal design. As a result, learners confronted with unintentional simulation design problems use cognitive resources to manage these problems rather than engaging with the learning objectives. Usability issues can reduce participant engagement, hinder learning, and diminish the perceived credibility of the program [12]. Hence, while simulation guidelines recommend piloting with learners to uncover problems, we believe usability issues should be addressed before piloting or implementation [2,6].

We needed to solve several unique problems in educational simulations that are atypical for usability testing of medical devices or software. First, we needed to calibrate scenario difficulty to participants’ experience level (eg, residents) and expertise (eg, obstetrics and gynecology [OB/GYN], family medicine). Second, faculty observers needed to evaluate learners for specific telehealth competencies to identify skill gaps. Third, we needed to include built-in challenges for the learner. Thus, we needed faculty participants to serve as observers to provide feedback on the scoring guide while gathering tester feedback on the simulation.

The theoretical framework informing our current approach is based on a model of external validity we created for usability evaluations of health information technology (Figure 1) [20]. External validity comprises (1) population validity (ie, the extent to which the sample represents the population) and (2) ecological validity (ie, the extent to which the setting matches the real world). While both types of validity are important, the fidelity level for each dimension depends on the evaluation objectives [12].

When selecting the degree of ecological validity, we needed to consider learner attributes, presimulation didactics, the clinical environment (ie, physical and social), the technology (ie, hardware and software), and the scenario (ie, task characteristics, patient data, patients, and responses). Figure 1 [20] shows how we mapped model constructs to our use case. Poor usability arising from interactions between dimensions could manifest as unclear instructions, confusing interface layouts, incomplete patient scripts, mismatched equipment, or poorly structured task sequences. We chose to test our simulation with faculty rather than with learners. This was an intentional choice to assess case accuracy and to determine whether faculty believed the tasks were appropriately calibrated for resident learners.

During the COVID-19 pandemic, organizations embraced virtual care to enact social distancing precautions without limiting access [24-29]. This expansion highlighted educational gaps; practitioners were uncomfortable providing online care, and few training programs offered telemedicine instruction [30-35]. In response to this emerging need, the Association of American Medical Colleges (AAMC) published a white paper outlining a framework for telemedicine training in the United States [36]. The AAMC identified 20 competencies across six domains: (1) patient safety, (2) access and equity, (3) communication, (4) data collection, (5) technology, and (6) ethical and legal requirements. Since publication, medical schools and residencies have endeavored to teach this material using readings, didactics, and simulations [35,37-39].

At the University of Oklahoma School of Community Medicine (OUSCM), the Department of Medical Informatics partnered with the Department of Obstetrics and Gynecology to develop a telemedicine training program for maternal care [40]. Designed for OB/GYN and family and community medicine (FCM) residents, our simulations combined telemedicine instruction with vignettes managing maternal care conditions [41-43]. By training future physicians to provide online care to expectant mothers, we hoped to close a supply-demand mismatch contributing to local maternal care “deserts” [40,44-49].

Our educational program created the need—and the opportunity—to conduct usability tests of simulations before piloting with residents. The technology-forward nature of telemedicine inspired our team to adapt usability methods typically applied to health information technology. Although we have published preliminary reports describing individual tools, this is the first tutorial comprehensively describing our usability testing methods, findings, and downstream piloting experiences [41-43]. For illustrative purposes, we demonstrate our methods using a simulation of a fictional patient presenting for a telemedicine encounter to manage gestational hypertension.

Our research team conducted all testing at our university’s simulation center in Tulsa, Oklahoma, United States. The team included 2 obstetricians (MR, KB) to provide subject-matter expertise, 4 informaticians (BJL, HM, RM, and JH) and 2 data analysts to design the test protocol (MA and SM), a proctor to supervise each test and code the data (MA or CP), and at least 1 additional observer (typically another member of the research team) to double-code observations. The proctor and observers played the role of faculty during simulation testing. The team also included an SP and a simulationist (KB). The simulationist is an expert in simulation design, SP education, and script writing. The simulation center was equipped with mock clinic rooms, remote monitoring stations, simulation software, and computers with videoconferencing software. Because this was a telemedicine simulation, our SPs connected to the simulation center remotely via videoconference.

To recreate our described approach, we recommend the following resources: a dedicated space with internet access, a trained team member to serve as the SP, videoconferencing software with recording capabilities, and a proctor for the session. We also recommend recruiting team members to observe and double-code findings. We found it helpful to have trained faculty on-site to debrief testers. While the examination rooms in our center increased the ecological validity of our testing, they were not required.

After we completed all usability testing and simulation refinements, we piloted the entire workshop (ie, a telemedicine didactic session, telemedicine simulation with the competency rubric [Multimedia Appendix 9], and postsession questionnaire) with the residents. A detailed protocol for our workshop and simulation pilot falls outside the scope of this tutorial. We included a brief vignette of the pilot herewith (Textbox 2) to illustrate the downstream value of usability testing.

The testing described in this tutorial was reviewed by the University of Oklahoma Institutional Review Board and classified as a quality improvement initiative associated with normal educational activities. Nevertheless, we collected verbal and written consent from each tester to document the testing.

We recruited 12 clinicians for usability testing (first round: 4 OB/GYNs and 2 FCMs; second round: 2 OB/GYNs and 4 FCMs). Table 2 provides demographic details.

Table 3 summarizes all usability measures, with key findings highlighted. We include detailed results for each measure in the following sections. The ATA identified 4 problematic steps in round 1 that we resolved by round 2. Across both rounds, we identified and addressed 15 critical usability issues. The virtual physical examination received the lowest SEQ ratings. During round 2, testers reported a median SUS score of 77.5 (IQR 73.1-80.0) the SUS and a median SSES score of 42 (IQR 41.0-46.75) out of a possible 50. The median NASA-TLX mental demand score in round 2 was 60.00 (IQR 56.3-63.8); a score >50 indicates a high workload. Proctors could not score all testers with the rubric during the first round. After modifying the rubric language during the first refinement phase, proctors successfully scored every tester.

Most testers found the simulation instructions and steps easy to understand (Figure 7 [55]). Testers assigned a median SEQ score of 6 or greater to 4 of the 6 components. Testers said completing a virtual physical examination (median 4.50, IQR 2.75-6.00) was the most challenging step.

Testers reported some challenges with ease of use and learnability (Figure 8 [52]). SUS scores ranged from 65 to 100. For round 1, respondents’ median score was 73.80 (IQR 70.63-82.50); for round 2, the median score was 77.50 (IQR 73.13-80.00). The inclusion of a mock EHR in round 2 may account for the increase in the median SUS score.

The SSES scores indicate that most testers considered the simulation a valuable learning experience (Figure 9 [53]). One tester disagreed with the statement that the simulation tested their clinical abilities.

Testers reported considerable variation in perceived workload, depending on the subscale. Mental demand and effort were consistently the highest and increased from round 1 to round 2 (Figure 10 [69]). Mental workload was highest, with a median score of 37.50 (IQR 21.25-57.50) for round 1 and a median score of 60.00 (IQR 56.3-63.8) for round 2. The median scores for total effort (a composite of mental and physical effort) were 42.50 (IQR 27.50-57.50) and 55.00 (IQR 46.3-63.8) for rounds 1 and 2, respectively.

In the first round of testing, the proctor and observers could not score 3 tasks associated with 2 competencies (ie, “establishes therapeutic relationships and environments” and “supports solutions to ethical problems”). This problem was caused by ambiguities in the scoring guide’s language (Figure 11 [36]). Revising the language improved usability. One persistent problem was identifying when a learner fabricated information (eg, improvising the name of a regional laboratory) to preserve the illusion of the simulation. We therefore modified the SP script before piloting to discourage learner improvisation.

We summed proctor-scored task completion rates to calculate composite telemedicine competency scores (Figure 12 [36]). Testers tended to overlook telemedicine-specific tasks, such as gathering contact information, including the home address (AAMC competency I 2b); describing an emergency escalation plan (AAMC competency I 4b); or checking whether the patient was in a private setting and whether other household members were present (AAMC competency III 2b) [36].

For this tutorial discussion, we first provide an abbreviated review of the most significant usability findings. We then offer a concept-level appraisal of our testing methods and insights for readers planning similar testing protocols. We conclude by comparing with the literature, highlighting gaps, unexplored research opportunities, and future directions.

Our usability evaluation offered valuable insights into the effectiveness of our educational simulation and opportunities for quality improvement. The data gathered using the ATA enabled the development team to track and communicate improvements to the research team and simulation staff. We identified 52 usability issues requiring prioritization and made 15 design revisions before piloting with learners. By the second round of testing, we removed critical chokepoints related to the audiovisual cross-check and added a mock EHR with synthetic patient data and fill-in data fields. The mock EHR provided the learner with visual cues that improved the telemedicine workflow and task completion rates. Our qualitative data clarified the root causes of usability issues and provided an efficient method for prioritizing revisions. We traced our biggest challenges to the fidelity of blood pressure data and the reliability of the evaluation rubric. We subsequently focused revision efforts on clarifying details in patient scripts, adding synthetic patient data—including a blood pressure flowsheet to the mock EHR—and modifying scoring guide language and definitions in the telemedicine rubric.

It was crucial to analyze the simulation in accordance with the workflow steps, expected tasks, and learner competencies. By measuring simulation steps and task performance independently, we could see when testers completed all scripted steps but overlooked clinical tasks. This helped distinguish undesirable technical distractors from scripted challenges. By removing usability issues that diverted testers’ attention from their tasks, testers could more effectively engage with the learning material. This also helped faculty more easily determine when simulation tasks were misaligned with learning objectives.

Testers successfully used the videoconferencing software, resolved technical issues, and established therapeutic rapport. By contrast, most testers neglected tasks necessary to optimize the telemedicine environment or prepare for a medical emergency. This is not unexpected; none of our testers received telemedicine training, whereas we gave a lecture to residents before the simulation.

For educators planning to include grading rubrics with their simulations, we recommend embedding scoring forms or debriefing guides into the testing instruments. In our example, we included the telemedicine competency rubric in the ATA. Our experience suggests a thorough appraisal of the evaluation materials by domain experts and participating faculty can improve construct validity by aligning simulation tasks and learning objectives with rubric terminology and instrument design. We also believe testing can improve instrument learnability and ease of use so that faculty can invest less time training proctors during piloting and implementation.

We used the SEQ, SUS, and NASA-TLX scores to compare our qualitative observations with usability benchmarks (ie, convergent validity). The range of SEQ scores (2 to 7) and SUS scores (65 to 100) helped identify the most challenging simulation components. The SEQ pinpointed issues in the synthetic data collected during the history and examination.

The NASA-TLX data helped estimate overall simulation difficulty. Responses indicated a relatively high score for mental workload compared with the other subscales (round 1: median score 37.50, IQR 21.25-57.50; round 2: median score 60.00, IQR 56.30-63.80). The scores may have increased due to the inclusion of a mock EHR. Considering these findings, we asked faculty whether the simulation should be modified. Obstetrics and family medicine faculty advised against changing the tasks because they aligned with the educational goals and learning objectives. During piloting (illustrative vignette in Textbox 2; key tutorial insights in Textbox 3), residents were indeed able to complete the simulation but struggled with several tasks, including developing a preemptive care escalation plan and instructing SPs on the use of a digital blood pressure cuff. This suggests that our clinical scenario and materials provided sufficient scaffolding for learners to engage effectively with the material while gently challenging them with unfamiliar tasks.

This tutorial advances simulation science in several ways. First, we demonstrated how to leverage usability methods used in technology evaluations to systematically assess educational simulations before piloting with learners. Validated, theory-driven instruments can provide insights into simulation effectiveness, ease of use, and user satisfaction [22,82-84]. These methods are technology-agnostic and, with little modification, are suitable for evaluating simulation constructs such as engagement, cognitive load, skill acquisition, and competence.

While this study offers turnkey methods for formative usability testing, the choice of any single instrument should be driven by the learning objectives, the simulation’s attributes, and the phase of the development life cycle. Not all methods are necessary for every simulation study. Researchers should add or remove components in a modular fashion. For example, user satisfaction with simulation data may be incidental to the educational program objectives; studies show self-assessment is an unreliable estimate of teaching effectiveness [85,86]. We included the SSES in our protocol because we believed fellow educators could provide useful appraisals of our simulation and its educational value.

Second, our approach addressed design challenges unique to educational simulations [22,23,74,87]. We concurrently evaluated simulation design features and learner competencies to align teaching modalities with educational objectives and to balance ecological validity with simulation complexity [4,11,12,20,88,89]. Our instruments were well suited to observing technology use and should be considered for technology-forward educational initiatives that include EHRs, mobile devices, artificial intelligence, and computerized decision support. Unlike traditional usability testing, our goal was not to remove opportunities for user error [17,21,73,80,90-92]. Educational simulations must enable learning through experimentation, discovery, and even failure [4,42,93,94]. We therefore used the ATA to record whether testers had the opportunity to demonstrate their skills during the simulation; the simulation could proceed as intended even if a tester failed a task.

Third, our methods can be used early in the product development life cycle with an array of testers [2,4,9]. While best-practice simulation guidelines recommend piloting simulations with learners, there are practical challenges [2]. Recruiting enough target learners can be impractical. Moreover, testing and piloting serve different but equally important roles. Because simulations can be challenging or stressful for learners, pilots may not be the ideal means of measuring usability dimensions such as learnability, error tolerance, and satisfaction [95-98]. Testing with experts can assess the fidelity of the case, the accuracy of the synthetic data, the appropriateness of the learning objectives, and the difficulty of tasks [41-43,99].

Fourth, we believe our methods are flexible, cost-effective, and scalable. We used low-cost tools that could be adapted to resource-constrained settings, such as smaller academic centers and schools in low- and middle-income countries [73,74,100]. Many of the instruments described herein, including the SUS, NASA-TLX, and SSES, have been validated in languages such as Chinese, Korean, Spanish, French, and Croatian [69,101-107]. These methods do not require large numbers of testers. We recommend multiple short testing rounds with as few as 5 testers; this provides sufficient data to guide refinements [108].

With the expansion of advanced simulation modalities, such as AR, VR, and AI, software developers are increasingly recognizing the role of usability testing in the product development life cycle [109-114]. There are already protocols for a range of tools, including AR, VR, digital twins, and serious games [8,14,15]. While most reports focus on the technology, several have evaluated the broader simulation context. Anton and colleagues described an evaluation of 2 emergency medicine simulations that combined learner eye tracking and faculty ratings on the Non-Technical Skills for Surgeons Scale [14]. Silva et al [15] applied a more comprehensive approach by evaluating a simulation manikin for teaching cardiopulmonary resuscitation. They created a data collection instrument based on a basic life support guide and scored learners using a 16-item Likert-type scale. The researchers recorded time on task and administered the SUS after each encounter. In this manner, they operationalized usability in terms of the dimensions of effectiveness, efficiency, and satisfaction. Together, these reports provide complementary perspectives on the evaluation of simulation-based instructional modules: Silva et al [15] focused on the physical usability of manikins, whereas Anton et al [14] focused on clinicians’ cognitive skills.

Da Silva and Dubrowski [8] described one of the first attempts to incorporate usability methods early in the development life cycle. They conducted a series of design sprints to evaluate a conflict-management scenario between a nursing student and a senior nurse. The team tested 2 scenarios and recruited 20 nursing students. After each simulation, the team conducted exit interviews and administered the Simulation Design Scale (SDS)—a validated 20-item instrument measuring the quality and effectiveness of simulation-based learning experiences [115]. Unlike the SSES, which focuses on learners’ satisfaction with the instructional approach, the SDS focuses on missing or underdeveloped simulation elements. Da Silva and Dubrowski [8] used the SDS as a quality metric and tracked improvement in scores across iterations. We believe this work is an important milestone toward developing a metrics-driven usability strategy.

We believe the application of usability methods to simulation-based education remains underexplored [41,42,99]. To our knowledge, our report and related preliminary work provide the most complete descriptions of mixed methods usability testing for educational simulations. Combining subjective measures (eg, user satisfaction and ease of use) with objective performance data (eg, task completion rates, error tolerance) can maximize insights into technical functionality and educational value [57,116,117].

These observations notwithstanding, several limitations may affect the generalizability of our work. We identified new usability issues after 2 refinement sessions, suggesting we had not reached saturation. Unfortunately, our team faced time and resource constraints; the number of iterations was, in part, dependent on curriculum deadlines and the number of available testers. To our knowledge, there is no universally accepted number of testers or testing rounds that guarantees researchers will find all usability issues. The usability literature, however, describes an issue-discovery curve with diminishing returns; as few as 5 testers may identify between 80% and 90% of critical issues [17]. Experts therefore recommend conducting several small testing cycles with 5 users at a time [17,118-121].

This report describes a single-site, single-SP, quality-improvement design. This raises several validity issues that usability specialists must consider. We conducted all usability testing with a single SP to control for simulation variability. However, our SP’s familiarity with the case may have limited our ability to identify weak prompts, responses, or missing information. In the future, incorporating multiple SPs could reveal variations in script interpretation and behavior. We also observed variation in practice behaviors among subject-matter experts as a function of experience, specialty, and training. This suggests that institutions may inadvertently incorporate provincial practice patterns and local health system norms into their simulation designs. This could create standardization challenges, since usability studies tend to compare testers' performance against an established workflow or an approved set of product design requirements. In our example, we compared usability findings to local workflow and learning objectives. Researchers designing future studies may elect to use an external reference standard, such as a prerecorded best-practice encounter developed by a professional society.

We did not conduct interrater reliability tests for our rubric. In general, calculating agreement statistics falls outside the scope of usability testing during the product design phase. Instead, several members of the research team independently coded each task. We then compared scores and resolved discrepancies through discussion. Task descriptions with vague or ambiguous language were updated during the refinement phases. We believe the next step in our research program is to measure the interrater reliability of our rubric by asking faculty to independently rate learners who have completed the simulation and calculating an agreement statistic (eg, Cohen κ or Fleiss κ) [122].

We qualitatively coded all usability findings using the usability heuristics for health information technology by Zhang et al [80]. While useful for categorization, the codes did not indicate priority or provide insights for remediation. We believe there are opportunities to develop and validate usability heuristics specific to educational simulations. Ideally, codes informed by educational theory would provide design guidance to educators.

Finally, we see future opportunities to test additional usability measurement tools, including the SDS, heuristic inspections, and cognitive walkthroughs [17,22,115]. This could help enrich our logic models that connect usability tools to evaluation dimensions, such as realism, memorability, completeness, and predicted future learner performance. While videoconferencing may not be required for most simulations, other teams might use this capability to link distributed resources, such as testers, researchers, SPs, and software. To support these kinds of remote testing initiatives, researchers may distribute questionnaires digitally and trial wearable technologies, such as eye-tracking lenses and wearable activity trackers, including wristbands and smartwatches.

We believe this tutorial illustrates a practical, reproducible, and resource-sensitive method for usability testing educational simulations before piloting with target learners. While this report describes a single-site, single-SP, quality-improvement design, we strove to include published, validated instruments with claims of generalizability and relatively low-cost testing methods to permit replication or context-appropriate adaptation. We have included appendices with our instruments and encourage readers to adapt and apply these methods to a range of simulations. While we believe this monograph provides simulationists with a turnkey toolkit to conduct similar assessments, further studies using psychometrically validated instruments are needed to clarify the best educational use cases, evaluate additional constructs relevant to teaching and learning, and, when relevant, incorporate additional tests to measure the interrater reliability of learner assessment instruments.