This was a single-center, cross-sectional pilot study of simulation cases on a web-based EPA assessment platform using an exploratory mixed methods design, which entailed a quantitative analysis followed by a qualitative approach. We obtained a convenience sample of 15 incoming first-year interns of a 4-year postgraduate emergency medicine residency program at an academic institution. We purposively sampled these interns and program leadership for subsequent interviews. This study was approved by our institutional review board (protocol 49712).

We used Kolb’s experiential learning model as the framework for developing our assessment [14]. The integration of active experimentation and concrete experience described by Kolb was achieved through web-based simulation [15]. Reflective observation and abstract conceptualization occurred during stakeholder interviews, as well as during the use of the assessment results to design individualized learning plans. The process of developing these learning plans is outside the scope of this study and therefore not reported.

We used the My Sim Cases web platform [16] to customize an assessment for participant skills based on 5 EPAs. We selected EPAs that are well suited for virtual assessment: EPA2: prioritize a Differential Diagnosis Following a Clinical Encounter; EPA3: Recommend and Interpret Common Diagnostic and Screening Tests; EPA4: Enter and Discuss Orders and Prescriptions; EPA5: Document a Clinical Encounter in the Patient Record; and EPA10: Recognize a Patient Requiring Urgent or Emergent Care and Initiate Evaluation and Management [17]. To assess feasibility, participants were asked to access the platform asynchronously anytime between their medical school graduation and the start of internship. Participants viewed a 5-minute tutorial that explained the interface prior to completing 4 virtual clinical cases.

A panel of 6 education assessment experts, residency program faculty, and clerkship leaders convened to design these 4 clinical cases. First, we proposed virtual cases based on chief complaints in common emergency medicine, including chest pain, shortness of breath, vomiting, and altered mental status. One author (CP) then drafted the cases and the corresponding appropriate clinical actions. In order to optimize content and internal structure evidence, the same panel reconvened to review and revise drafts thrice across several months until consensus was achieved. Then, a second panel of trainees (medical students and residents) and faculty members (clinical and nonclinical staff) provided feedback on the cases as well. These responses were cross-checked for consistency, as evidence of response process validity. From that review, a critical actions checklist and corresponding performance report was developed for each case. These reports use “look for” statements that help guide supervisors’ attention to aspects that need to be reassessed in the clinical environment or for which feedback should be provided. A “look for” statement based on expected history and physical examination techniques, missing actions, and harmful actions might read as follows: look for incomplete or missing information in documentation. The cases were implemented in the customized assessment platform. The web-based platform was pilot-tested by the second panel in 2 rounds, to further evaluate the cases for functionality, matching of item construct and content, optimal item phrasing, and overall quality control. Final pilot testing was performed by 6 emergency medicine faculty members with expertise in medical education.

The platform features 2 user interfaces that simulate the electronic medical record (EMR) and bedside evaluation (Figure 1). In the bedside interface, the participants engaged in a clinical encounter in which they clicked through questions to obtain the patient’s history, obtain additional information from prehospital providers, and request bedside clinical actions (eg, ask the nurse to display vital signs on the monitor, insert a peripheral intravenous catheter, and administer oxygen by nasal cannula). In the EMR interface, the participants placed orders for medications, imaging, labs, and requests for consultant advice (Figure 2). Participants chose the documentation of the history and physical exam using a multiple-choice prompt. They utilized an open textbox and a dropdown menu to record a differential diagnosis. The results for labs and imaging were withheld until all orders and documentation were completed. To close the encounter, the participants chose a disposition of “admit to hospital” or “discharge home” and entered a final diagnosis. A composite score report of “look for” statements was generated after completing all 4 cases; reports included between 0 and 4 “look for” statements per case with the desired score being 0.

A postassessment survey (using Qualtrics XM) captured demographic and previous training data. Following the quantitative portion of the study, we individually interviewed a subset of residents and residency program leaders about their experiences and perspectives of the assessment program and resulting data reports.

All 15 eligible subjects consented to participate in the study, and 13 (87%) provided demographic data. Of these, 54% (7/13) were women, with an average age of 28 (range 25-38) years. Only 85% (11/13) of the participants reviewed the tutorial about the interface, and 31% (4/13) of them reported use of a cognitive aid during the assessment (which is defined by any resource outside of the virtual case such as other websites or applications). The mean time to complete all 4 cases was 48.6 (range 16-91) minutes.

Table 1 summarizes the aggregated cohort score reports by EPA. All participants (15/15, 100%) generated the look for statement pertaining to the “ability to enter critical orders and avoid inappropriate or unnecessary orders,” which relates to EPA3 and EPA4. Moreover, all participants (15/15, 100%) had at least one incorrect action or order placed. The average number of missing, but not harmful, actions across all cases was 13.5 (range 8-25), and 40% (6/15) of the participants performed at least one harmful action.

Furthermore, over half (8/15, 54%) of the participants did not include the final diagnosis in their initial differential in at least one of the cases. At the conclusion of the case, 38% (3/8) input at least one incorrect final diagnosis. The participants were most successful in documentation, with only 40% (6/15) generating this “look for” statement (Table 2). The “look for” statement regarding “[the] lack of recognition of most likely urgent/emergent diagnosis and appropriate initial management” was generated most often when the learners indiscriminately applied oxygen to all cases (9/15, 60%).

Overall, participants performed the best on Case D: Shortness of Breath (median of 1 “look for” statement, range 1-2). For all other cases, a median of 2 “look for” statements was reported, with a higher range reported for Case B: Vomiting (range 1-4) and Case C: Altered Mental Status (range 1-4).

We conducted 10 stakeholder interviews, 5 with intern residents (L1-L5) and 5 with residency leadership (L1-L5). The two coders achieved excellent interrater reliability for the resident transcripts (Cohen κ=0.92) and for the residency leadership transcripts (Cohen κ=1.00), with the unit of analysis being responses to questions.

We identified 90 unique content areas for both cohorts. We refined these to 3 major themes in relation to the acceptability of the web interface and our method of asynchronous EPA assessment (Table 3). These themes included (1) psychological safety, (2) assessment for learning, and (3) value of interface fidelity. Topics most commonly discussed by the participants pertained to psychological safety, user experience, and usefulness of formative feedback, whereas the focus of the faculty interviews was individualized learning and potential uses of assessment results.

The use of a web-based simulation platform for EPA assessment is feasible and acceptable to key stakeholders (ie, residents and program leadership). EPA assessment using simulated cases that were customized to common emergency medicine chief complaints may have increased fidelity for incoming interns and relevance of the score reports to their program directors. Trainees found that asynchronous, individual testing provided psychological safety, and residency directors believed that the score reports could guide the development of individualized learning plans early in residency training. Psychological safety and targeted, individualized feedback are desirable outcomes consistent with other EPA-based studies [21-23].

Importantly, no single participant was competent across all 5 EPAs. These findings are somewhat troubling yet promising that our virtual assessment detected such information. Our study also highlights the need for medical schools to better use the EPA framework to guide curriculum decisions and assure the quality of their graduates upon summative entrustment for preparedness to enter into residency training. These findings are neither surprising nor novel, and they are consistent with other EPA literature to date [3,24,25].

Acceptability of this assessment approach was strengthened by the use of “look for” statements rather than numerical scores. Rigorous EPA assessment by medical schools would use multisource feedback and standardized testing to achieve defensible decisions about student competence. “Look for” statements mirror high-quality clinical feedback that is familiar to most students. It also operationalizes the EPA assessment for program directors beyond a construct of competent versus incompetent that might be offered in a summative report at the end of medical school. Moreover, these statements provide trainees and educators with understandable and achievable learning goals and align better with the culture of feedback rather than a punitive approach to learning, while simultaneously highlighting significant educational gaps.

Another important factor to consider with a customized design is the physician order entry interface that could simulate the EMR used in the local clinical environment. Consistency between the simulated platform and the local EMR could result in early adoption of systems at the sponsor institution and reduce cognitive load once interns begin clinical rotations. Outcomes of EMR training have shown a reduction in self-reported medical errors, and similar benefits could be observed with this assessment for learning during orientation [17,26].

The limitations of this study include a small sample size, a single institution pilot, a single specialty cohort, implicit biases noted in the study methods, and the use of a convenience sample of stakeholders. Although all residency directors or associate directors in our department were interviewed, we sought volunteers from the trainee cohort; these could be individuals who may have had a favorable opinion of the pilot study and thus offered to volunteer, which could affect their responses. Similarly, the stakeholders may have inferred that the interviewer had a favorable opinion of the project and therefore softened any potentially negative responses. Our cases are aligned with our specialty to increase response process validity; thus, further testing is required to explore generalization of these results across students entering any field. Finally, this assessment is meant to be formative, not summative; we cannot fully assess student competence for any EPA with a single test and absence of multisource data. Therefore, the results of this assessment are best used by trainees and residency directors, not medical schools seeking a single examination of EPA competence.

Future study of our platform will include the logical expansion of testing to all medical school graduates to ensure this observation remains consistent. Customization of EPA assessments using specialty-relevant cases is also desirable for fidelity; as such, specialty boards might be the logical third parties to oversee development of such interfaces. This would bookend a resident’s interaction with their future boards, with incoming assessment and certification exams at either end. Further investigation should also include a longitudinal evaluation of clinical learning outcomes at various intervals during residency training. Finally, it is critical to examine the design of individualized learning plans based on assessment results. Although the benefits of individualized learning and development plans have been previously demonstrated, their implementation has been difficult in the absence of pragmatic guidelines [2,27].

Asynchronous, individual EPA assessment using this web-based platform is feasible and acceptable to key stakeholders. This offers a psychologically safe and yet practice-relevant way to diagnostically assess incoming interns and will assist with transitions to residency.