This study was designed as a mixed methods approach combining quantitative and qualitative analyses to evaluate residents’ prerounding performance using the Epic EHR system. We invited pediatric residents at a large teaching hospital in the South Atlantic Region to participate in the study as part of an optional professional development event. A convenience sample of all 30 pediatric residents voluntarily participated are reflecting the entire population of pediatric residents in the hospital. The residents’ level of training ranged from 1 to 3 years of postgraduate medical education, and all residents had more than a month of direct patient care in the pediatric wards. All residents had prior experience using the EHR system (Epic Systems) for prerounding as part of their work routine. To ensure completeness and transparency of reporting, we followed the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) reporting guidelines [33].

Several days prior to the professional development event, participants were asked to complete a web-based questionnaire asking about their prerounding experience. At the start of the professional development event, participants were also requested to complete another demographic and EHR usability questionnaire (see Multimedia Appendix 1 for more details on the questionnaires).

For the experimental portion of the study, residents were instructed to perform their prerounding routine on 2 pediatric inpatients. All residents who participated in this study prerounded on the same 2 patients. Both cases were of medium complexity and representative of the types of patients that residents routinely care for in the acute care wards (for more details, see Multimedia Appendix 2).

Each resident was provided a 17.3″ Lenovo workstation laptop with a wireless mouse that had Epic EHR system preinstalled. Upon logging in the system, the workstation displayed the same EHR layout that the residents typically use to preround with any customizations of the EHR system they have created. Morae video analysis software was also installed to record residents’ speech and video capture all user interactions with the system. Residents were also provided paper so that they could write down any information they normally write down during prerounding to serve as their notes during rounds. Residents were seated in proximity to each other, similar to the environment in which residents typically preround in.

At the beginning of the study, residents were given the names of the 2 case-study patients and were asked to log into their accounts in the EHR system and initiate the video-capturing software. The video-capturing software would then prompt the residents to complete a small questionnaire related to their experience and the EHR system’s usability. After completing the questionnaire, residents began prerounding on the 2 case-study patients using the think-aloud protocol to verbalize their internal thought processes while completing the tasks.

The study involved 2 groups of 15 residents who alternated prerounding on the patients. Each group was allotted a maximum of 20 minutes to complete prerounding on both patients. This time limit was determined by pediatric experts, based on the relative complexity of the cases and questionnaire responses, where the majority of residents indicated that they usually need less than 10 minutes for prerounding a patient. After residents prerounded both patients, they were asked to fill out a debriefing questionnaire on their experience during the study, their concerns about the time constraints, and any difficulty they encountered while completing the prerounding tasks.

A team of 5 researchers used a standardized spreadsheet to systematically categorize the data collected from the Morae video analysis software during the recordings of the prerounding process. To ensure consistency in video analysis, prerounding data collection was categorized into the following six tasks based on literature [34-36] and recommendations of pediatric experts who assisted in conducting the study. These tasks included (1) reviewing patient vital signs (vitals), (2) checking prior and upcoming feeding and lab orders (orders), (3) reviewing recent lab results (labs), (4) checking patient intakes and outputs (IOs), (5) reviewing clinicians’ and nurses’ notes (notes), and (6) reviewing current medications and medication changes (meds). These tasks served as a benchmark for evaluating residents’ performance, as they are expected to complete all 6 tasks for each patient. We analyzed the video recordings to determine whether each task was completed or omitted, the time taken to complete each task, and any participant comments related to the task being performed, including any difficulties or challenges encountered. To facilitate the analysis process, standardized drop-down menus were used to populate the spreadsheet with 5 events, including the start or end of prerounding of the patient, start or end of a task, page access, information or data collection, and participant comments. The video reviewer created an entry for each event by recording the timestamp of the event and using the drop-down menu to populate the entry with the relevant event type, prerounding task being performed, task, and the page being viewed, alongside any comments made by the resident (see Multimedia Appendix 3).

To ensure the reliability of our data, we used a rigorous 2-reviewer approach, where each video recording was independently analyzed and coded. The level of agreement among reviewers was very high, with less than 5% (80/1926) of entries showing discrepancies between reviewers. A third reviewer was assigned to reconcile any discrepancies and consolidate similar entries, and all proposed changes or modifications were mutually agreed upon by all reviewers before proceeding to the analysis phase of the study.

To assess the effectiveness of the prerounding process, several outcome variables were analyzed:

These outcome variables provide valuable insights into the effectiveness of the prerounding process and the performance of residents.

We performed statistical analysis using Excel (Microsoft Corporation) for data entry and SPSS (IBM Corp) for data analysis. Categorical variables were presented as frequencies and percentages. To investigate the association between variables, we used the independent sample t test and ANOVA. A P value less than .05 was considered statistically significant.

Ethics approval for this study was obtained from the institutional review board for Social and Behavioral Sciences (IRB-SBS) at the University of Virginia (IRB protocol number is 3480). All participants provided informed consent before taking part in the study.

This study had no external funding to declare. All aspects of the research, including design, data collection, analysis, and publication, were independently managed by the authors.

A total of 30 pediatric residents participated in our study, but due to technical issues related to data extraction (specifically, corrupted recording files), only 20 residents (16 females and 4 males) had video recordings that could be analyzed. The analyzed video recordings were evenly distributed across residents of different pediatric department experience levels, with 7 PGY-1 (postgraduate year) residents, 6 PGY-2 residents, and 7 PGY-3 residents.

Based on the debriefing survey presented at the conclusion of the study, only 2 residents (10%) reported not having enough time to preround, and only 1 participant (5%) reported not being able to find all the information they searched for. However, based on the video analysis we found that 16 residents (80%) did not complete at least 1 task. Table 1 shows the tasks that were omitted and whether they were omitted for 1 or both patients. The task “meds” (ie, reviewing medications and medication changes) was the most overlooked task; 7 residents omitted the task for both patients, and 4 residents omitted it for 1 patient. For the task “orders” (ie, reviewing feeding and laboratory orders), 5 residents omitted this task for both patients, and another 5 residents omitted it for 1 patient. Finally, only 1 participant omitted checking “IOs” (ie, checking intakes and outputs) for 1 patient. The 3 remaining tasks—that is, “labs,” “notes,” and “vitals”—were completed by all residents.

We noted that multiple residents forgot to complete a task but went back to it while prerounding on the same patient or after prerounding on the other patient. These instances are not reflected in Tables 1 and 2 since residents eventually performed the task.

To examine the effect of experience level on the task omission, we calculated the percentage of tasks that were omitted by residents, categorized by their level of experience. Table 2 shows that residents with more experience had lower task omission rates. However, more than half (4/7) of the residents with the most experience (PGY-3) still omitted at least 1 task while prerounding.

Using chi-square tests for independence, we found no significant difference in both the proportion of omitted tasks among experience levels (χ²₂=1.8; P=.41) and the proportion of residents who did not complete all tasks (χ²₂=4.1; P=.13).

When responding to the questionnaires prior to participating in the experimental portion of the study, residents cited having to access numerous pages to collect the relevant patient data. Therefore, we wanted to see whether prerounding became more effective and efficient with more experience.

From the video analysis, we noted all pages that were accessed in the EHR when collecting data during prerounding. Pages that were accessed by mistake (ie, mis-clicking on a page then quickly exiting it) or were used mainly to access another page were not included in the analysis since they serve no purpose in data collection. Across all 20 residents, 58 distinct pages were accessed while collecting data on the 2 patients during prerounding. Table 3 shows that the total number of distinct pages accessed by each experience group ranged from 35 to 41 pages and did not seem to vary by level of experience.

The mean number of pages accessed by each participant while prerounding was also tabulated. On average, residents accessed 21 pages when prerounding on both patients. Table 3 shows the mean number of pages accessed by residents when categorized by experience level. There was no significant difference in the mean number of pages visited as a function of years in residency (F_2,17=0.08; P=.92), suggesting that the mean number of pages visited does not decrease with experience.

We also wanted to see whether EHR system use efficiency improves with experience. While the mean prerounding duration for third-year residents was about 45 seconds faster than first- and second-year residents, it was not statistically significant (F_2,19=0.75; P=.48; see Table 4).

The video analysis revealed that regardless of experience level, residents spent the most time on the task of reviewing notes. This task was especially time-consuming given residents had to read through the free-form text inputs that varied depending on who inputted the notes.

Another task residents spent a lot of time on was reviewing lab results. The video analysis showed that residents had to frequently scroll both vertically and horizontally during this task, which was noted to be difficult and disorienting based on the residents’ oral comments and questionnaire responses.

From the video analysis, we observed that pages that provided aggregated data for multiple tasks were already implemented within the EHR system. The use of aggregated data pages could potentially reduce the time spent navigating between pages (ie, “Summary/Ped Rounding” page); however, only 3 residents made use of these pages. Of the 3 residents who accessed the aggregated data pages, 2 were in their first year of residency (PGY-1), while 1 was in the second year (PGY-2).

Although the sample size is too small to draw conclusions, it is worth noting that the mean number of pages accessed by the 3 residents was 19 pages, which was slightly lower than the average of 21 pages, but with no statistical significance (t₁₈=0.80; P=.43, 2-tailed). In contrast to the expectations, residents who used this aggregating page had an average prerounding time of 7:29 minutes, which was higher than the sample average of 6:27, but the difference was not statistically significant as well (t₁₈=−1.60; P=.12, 2-tailed).

The goal of this study was to examine the effect of experience level on EHR use during prerounding. Our study revealed that while most residents reported having enough time and being able to find the information they needed during prerounding, video analysis showed that 80% (16/20) of residents did not complete at least 1 key task. This finding was applicable regardless of experience as over 50% (4/7) of the most experienced residents (PGY-3) still omitted some tasks.

Specifically, our study found that in the specialty focused on in our research (ie, pediatrics acute care wards), the tasks most frequently overlooked were reviewing medications and orders. This finding differs from the results reported in the literature for other specialties. The variations in task omission patterns between our study and those found in the literature suggest that specialty-specific workflow and EHR system design could influence task omission patterns and the quality of pre-rounding. The findings here highlight the importance of identifying workflow-specific solutions that could prevent the omission of tasks and the need for strategies to improve the efficiency of EHR use during prerounding.

One major challenge residents faced during prerounding was the time spent navigating between pages, which contributed to the inefficiency of the process. On average, residents accessed approximately 21 pages during prerounding, with the number of unique pages accessed amounting to 58 distinct pages. This finding demonstrates an inefficient prerounding process. While summary data pages that consolidate patient data for multiple tasks on a single page were available, most residents chose to gather raw data from different pages instead. It is unclear why residents did not use these summary pages, but it may be due to the lack of training and integration of these pages into the prerounding process or the fact that residents find them confusing or incomplete. This is supported by the fact that residents who did use the summary data pages did not preround any more efficiently than those who did not use them in terms of per-rounding time or number of pages visited.

To improve the efficiency of prerounding, it may be necessary to streamline the process of data collection, such as improving the design and usage of summary pages by tailoring to user needs, providing targeted training on their use, and encouraging residents to use them. EHR providers should also consider other EHR design changes and technological assistance such as artificial intelligence–assistive tools that can facilitate efficient data gathering if summary pages are not providing adequate assistance.

This study revealed a significant variation in data omission rates across tasks, where only labs and meds showed significant omissions among residents. This finding contrasts with a previous study [27], which used the same EHR system to examine omissions among residents in nonpediatric ICU settings that found medication data were almost never omitted (~3%), whereas fluid balance (IOs) was frequently omitted (~37%). Similarly, studies in respiratory ward [37] and general medical ward [38] indicate that fluid balance was often omitted. This disparity suggests that factors such as specialty and care setting may influence data omission rates. For instance, IOs are often more critical to monitor for pediatric than for nonpediatric patients, while medication infusions are more critical in ICU settings than in non-ICU settings, which are supported in the literature. These variations in omission patterns highlight the need to consider contextual factors when designing interventions aimed at improving EHR use efficiency and reducing omission rates.

Our study contributes to the growing body of literature on EHR use in medical settings, specifically regarding prerounding practices in pediatrics as mentioned in the “Specialty and Task Omission” section. Previous studies have shown that there are significant gaps in identifying dangerous medical management issues within EHRs, despite high levels of medical training [30]. These findings are consistent with our own, which revealed that even the most experienced residents still omitted some prerounding subtasks. However, our study adds to the existing literature by specifically examining the completion of prerounding tasks in the context of pediatrics. Furthermore, prior research has also shown that residents often omit collecting some information during prerounding [27]. However, our study expands upon this by revealing that entire tasks were not completed, and more than half of the most experienced residents still omitted some prerounding tasks.

We believe the lack of improvement in prerounding speed and accuracy with increased experience could be attributed to inadequate EHR training as well as poor EHR design [39]. Based on our findings, interventions to improve the efficiency and effectiveness of prerounding could include checklists within the EHR system or in paper forms to ensure all tasks are completed. Previous work has shown that supporting knowledge in the world versus knowledge in the head—that is, reducing recall and memory—is effective in reducing omission [40]. We recommend the use of checklists that include prompts that remind residents of what information is needed, instead of relying on the residents’ memory each time they preround.

A more comprehensive solution could involve designing the EHR system with case-specific semiautomated workflows for prerounding, which would suggest relevant pages to residents that can help them complete the required tasks. This would ensure that each prerounding task is not only completed but also done in the intended manner. This would necessitate the need to conduct a hierarchical task analysis [41] to decompose the overall prerounding task into goals, subgoals, operations, and plans to determine how the EHR could best support the residents at each level.

Studies have shown that the use of automatically generated templates had a positive impact on residents’ performance during rounding, including omission rates [35,42,43]; however, the use of such automation techniques could impact the residents’ situational awareness and cause overreliance on the automation [44]. Therefore, the impact of introducing artificial intelligence automations should be studied more before implementing them within EHR systems.

Furthermore, we recommend implementing training programs for residents that are tailored for specific tasks such as prerounding to standardize the process and introduce the residents to system features that might be useful and time-saving when prerounding. For example, training programs could recommend structured sequential procedures for completing tasks and introduce residents to the different functionalities of pages and new dashboards that allow for faster and more centralized information access [45]. Such training programs could be implemented as training sessions, system walkthroughs, or web-based videos that are accessible when needed [46]. However, the efficacy of the training program and its added work burden on the residents should be considered before implementation.

The design of the EHR system should also be reconsidered to better support the work of the residents [36]. Information access cost should be reduced, and features should be made clearly visible to users in ways that eliminate the need for training, and instead, users can explore system features on their own.

This study has several notable strengths that contribute to the understanding of EHR use during the prerounding process. First, our mixed methods approach, which combines self-reported data with video analysis, is allowing for a comparison between residents’ perceived performance and their actual performance and is enabling a more accurate evaluation of EHR use.

Second, the focus on the pediatric specialty provides valuable insights into the unique challenges faced by pediatricians and allows comparison of the EHR usage patterns to other specialties studied in the literature. Third, the varying experience levels among participants allow for a broader perspective on the impact of experience on EHR usage and performance.

However, this study is not without limitations. First, the study was limited to a single setting, a single medical center, one department, and using a single EHR system, which may limit the generalizability of our findings, and additionally, the use of EHR for prerounding may have unique considerations for pediatricians when compared to other specialties. Second, the small sample size of this study may have limited the statistical power of our analyses. However, the combination of data collected was among the few of its kind, and we performed time-intensive analyses that revealed new trends and supported existing work. We also acknowledge the need for caution in generalizing our results due to the majority of the residents being females, which may have introduced potential gender bias into our findings.

For future work, building on the strengths of our study, larger-scale studies across multiple settings and specialties could be conducted to confirm the generalizability of our findings. This would help to establish the validity of our conclusions and allow for broader insights into EHR use during prerounding across different clinical contexts.

Moreover, given the identified tasks that were frequently omitted, future research could focus on exploring the underlying reasons behind this discrepancy. Specifically, research could study how different clinical roles or specialties may affect task omission rates and how interventions such as checklists and workflow automations could be tailored to address these differences.

Overall, our findings reveal that residents often omitted completing tasks while prerounding and the process was largely inefficient due to the EHR design, lack of proper training, and an unstandardized prerounding process. To improve EHR use efficiency and prevent omissions, interventions such as checklists, training programs, and customized EHR interfaces are suggested. Despite its limitations, our study provides important insights about specialty-specific EHR challenges and those associated with EHR use during prerounding in general.