Impact of a Workflow-Integrated Web Tool on Resource Utilization and Information-Seeking Behavior in an Academic Anesthesiology Department: Longitudinal Cohort Survey Study

Introduction

Barriers to Resource Utilization

Starting in 1997 with a survey of family medicine residents in internal medicine, physical medicine, and rehabilitation programs, who were followed up with in 2004, several attempts have been made to quantify and evaluate trainees’ reading behaviors [1-3]. In alignment with adult learning theory, a study conducted by Cassidy suggests that residents’ desire to learn is primarily motivated by personal interests and clinical relevance to patient care rather than by training requirements [4]. Johnson and colleagues’ investigation showed that residents read for approximately 3.7 hours per week [1]. They also showed that while trainees desired to read more, they were limited by fatigue and time constraints due to personal obligations. The groups concluded that improved delivery of educational materials, particularly via new databases and technologies, could improve the breadth and depth of resident reading. While this was true in 1997 and 2004, as the body of medical literature grows exponentially, it is even truer today.

With the extensive volume of medical literature available at the click of a mouse, one might wonder why any further learning technology intervention is needed. This expansion of electronic data, as well as fragmentation of resources across multiple websites and forums, has created a barrier to information-seeking behaviors [5,6]. A qualitative study conducted through five focus group interviews about a French general medicine program in 2015 found that both residents and general practitioners understand the importance of utilizing evidence-based medicine (EBM) and the need for unbiased information. However, study participants who generally used a limited number of online sources were not confident with their ability to assess the quality of information found, and they generally sought information in concordance with their existing knowledge [7]. Barriers to resource utilization are a problem that has been well-enough identified to have incited the development of the BARRIERS (Barriers to Research Utilization Scale) scale in 1991, which has been utilized in some 63 studies [8].

Training Tools for Graduate Medical Education

Learning technologies in graduate medical education and anesthesiology are in their early stages but continue to grow. In a broad review of the 21st-century learner, Chandrasoma and Chu confirmed that millennial learners overwhelmingly had smartphones, so they preferred to learn via electronic methods on a variety of platforms as passive consumers as opposed to content creators [9]. Directed reading programs are one learning intervention that can be implemented as a learning technology. de Virgilio and colleagues implemented a nontechnology-based directed reading program, which, along with textbook readings, included weekly exams [3]. This resulted in increased reading and improved examination scores in a surgical residency program. A directed reading program as a learning technology, in which readings were targeted toward in-training exam objectives, has also been successfully implemented in internal medicine as well as obstetrics and gynecology residency programs, resulting in improved board pass rates [10,11].

Use of Web Tools as a Novel Strategy for Learning

With the increased use of digital learning tools in medical education, web tools remain the most frequently utilized digital resource among medical students and residents [12]. A recent survey study of an inpatient medicine team found that web-based learning interventions improved self-directed learning, communication goals, and learning environment among medical students and residents [13].

With consideration to Thomas et al’s conceptual framework for curriculum development [14], as well as O’Brien’s conceptual framework for learning technology implementation [15], we have sought to build a learning technology into the daily workflow of trainees and faculty. These frameworks, in particular, guided us in attempting to evaluate our learners’ needs as well as the effectiveness of our learning technology in an iterative process. This learning technology consisted of an online teaching file of EBM resources as well as local and national policies and protocols that were on the same web tool as the daily operating room schedules and staff assignments. We were guided by a constructivist learning theory in our attempt to provide primary resources to trainees, which allowed them to build constructs in direct connection with their clinical experiences.

Firstly, we hypothesized that implementation of this web tool into the workflow would increase utilization of the provided resources by reducing barriers to access, including time constraints. Secondly, we hypothesized that the web tool would improve the trainees’ satisfaction with the educational environment, improve resident-faculty educational interactions, and improve faculty evaluation of trainee resource utilization.

Methods

Setting and Participants

Institutional Review Board approval was obtained on February 16, 2017, to conduct a longitudinal cohort survey study of trainees (ie, anesthesiology resident physicians and student registered nurse anesthetists [SRNAs]) and faculty (ie, physician anesthesiologists and certified registered nurse anesthetists [CRNAs]) at MedStar Georgetown University Hospital, a tertiary academic medical center in Washington, DC. Nonrandom sampling included the full accessible population of trainees and faculty. The primary outcome of this study was to determine if an educational web tool (Multimedia Appendix 1) for trainees designed and built into the daily workflow of an academic anesthesia department could significantly increase the trainees’ utilization of provided resources. The secondary outcome was to determine if this web tool could improve the perceptions of the educational environment, faculty evaluation of trainee recourses, and resident-faculty educational interactions.

Survey Development

Surveys attempted to elicit information regarding the 3 months prior to, and the 3 months after, implementation of the web tool. The trainee survey (Multimedia Appendix 2) queried the number of journal articles read or referenced, the number of local and national policies referenced, trainees’ self-perceived efficiency for accessing these resources, and their overall satisfaction with their education and educational resources within the department. The faculty survey (Multimedia Appendix 3) queried the faculty members’ perception of their trainees’ use of journal articles as well as local and national policies and their perception that the department provided effective educational resources. Evaluative queries were rated on a 10-point Likert scale, ranging from 1 (strongly disagree) to 10 (strongly agree). All surveys were completed on paper and administered by two medical students working with the research team.

Introduction of the Online Web Tool

Following the collection of the preintervention surveys, the online web tool—Departmental Intranet—was introduced, with operating room schedules and assignments published daily. Resources on the web tool were compiled and indexed by the research team. The web tool consisted of 121 journal articles, indexed in a variety of subject matter; 156 local policies and protocols; and 38 national society policies. The web tool was introduced with a brief oral presentation at the monthly faculty meeting, the resident morning lecture, and at grand rounds. A total of 3 months following the introduction of the web tool, identical postimplementation surveys were passed out to both trainee and faculty groups.

Data Analysis

Data were compiled in a deidentified manner and analyzed as trainee and faculty composite data, as well as in subgroups. Data were analyzed unpaired using the Student t test for continuous data and the chi-square test for ordinal data.

Results

Survey Responses

Survey response rates were greater than 50% in all groups: of the 43 trainees, we obtained 27 (63%) preimplementation surveys and 22 (51%) postimplementation surveys; of the 46 faculty members, we obtained 25 (54%) preimplementation surveys and 23 (50%) postimplementation surveys (Table 1).

Trainee Resource Utilization

Postimplementation trainee survey results showed a significant increase in the utilization of all resource categories, including journal articles (preimplementation mean 8.67, SD 6.45; postimplementation mean 18.27, SD 12.23; P=.02), national guidelines (preimplementation mean 2.3, SD 2.40; postimplementation mean 6.14, SD 5.01; P<.001), and local policies (preimplementation mean 2.23, SD 2.72; postimplementation mean 6.95, SD 6.09; P=.02).  There was also significant improvement in the resources that residents referenced, specifically for their clinical cases (preimplementation mean 4.63, SD 3.75; postimplementation mean 16.09, SD 20.07; P=.005), as well as faculty-trainee discussions of journal articles (preimplementation mean 1.67, SD 1.33; postimplementation mean 6.05, SD 8.74; P=.01) (Figure 1).

Trainees also self-reported feeling that it was more efficient to identify EBM resources in their clinical practice after implementation of the web tool (preimplementation mean 5.81; postimplementation mean 7.36; P=.03) (Figure 2).

‎View this figure

‎View this figure

Trainee Subgroup Analysis

Subgroup analysis showed that implementation of the web tool led to broad increases in resource utilization for both SRNAs and resident physicians as well as satisfaction among both groups with the educational resources, though not all results were statistically significant (Table 2).

Faculty Evaluation

Faculty were asked to assess the trainees’ ability to efficiently utilize evidence-based articles, local policies, and national policies. Faculty reported that trainees showed an improved ability to locate resources and reported increased satisfaction with resources provided by the department (Figure 3).

‎View this figure

Faculty Subgroup Analysis

Subgroup analysis showed that both CRNAs and attending physicians reported similar improvements in trainee abilities to identify and utilize resources (Table 3).

Discussion

Principal Findings

Following the implementation of the web tool, our trainees showed a significant improvement in the utilization of peer-reviewed articles, national guidelines, and local policies and protocols. Postimplementation surveys demonstrated a significant improvement in faculty-trainee educational interactions as well as faculty assessment of trainee resource utilization.

The development of an online web tool, more specifically, a centralized repository of academic articles, local policies and protocols, and national policies, was considered a first step in the implementation of improved educational technology at our institution. While there is certainly an abundance of materials available to faculty and trainees alike, searching and indexing through an ever-growing volume of information can be daunting. The goal of our web tool was to build these resources into the daily workflow, decrease barriers to utilization, and create a base for further educational technology interventions.

Adult Learning Theory for Millennials

The implementation of a departmental educational web tool is an example of applying adult learning theory for the millennial learner. Adult learning theory tells us that our residents are not oriented toward classical methods of lecture-based education with a postponed application of knowledge. Alternatively, they are oriented toward educational materials that are integrated into their current clinical experience and the immediacy of application. For this reason, modern e-learning in a professional environment is asynchronous, personalized, and just-in-time [16]. Millennial learners in the digital age prefer modalities that are online and self-paced. Meeting these needs requires us to build relevant educational tools into the learning environment [17]. Our survey study shows that creating such tools is an effective way to connect with millennial adult learners via their preferred methods. By meeting them where they are, we are able to encourage our learners to read, to improve their satisfaction with the educational environment, and to better connect trainees and faculty.

Integration Into the Workflow

We focus on the concept of integrating our web tool into the daily workflow of our learners because this method approximates a passive clinical decision support (CDS) system. By providing resources to providers based on the patients they are seeing, CDS systems ease the barriers to information-seeking behaviors, as discussed earlier [18]. CDS systems have been used in our field of anesthesiology to encourage protocol utilization, remind providers to administer antibiotics, and to improve compliance and billing [19-21]. Active CDS systems send notifications to providers, while passive systems like ours require providers to click links to access information [22]. Our web tool is not a full CDS system. The web tool lacks the data acquisition and rules modules to be considered an independent CDS system. However, by taking the relevant resources, including local and national protocols, and linking them directly into the workflow, we lessen the barriers to utilization of these resources and approximate a CDS system with many of the benefits. CDS systems have been shown to be very effective in increasing protocol utilization, with findings similar to our web tool implementation [21,22].

Limitations

The primary use of self-reported resource utilization via surveys introduces certain limitations and the potential for bias. Surveys were administered in person, preventing false respondents. Surveys were administered by medical students and were deidentified to minimize social desirability bias; however, this cannot be eliminated and may be particularly prevalent among trainees. Given that survey response rates were 50%, there was a possibility of self-selection bias in which trainees and faculty that did not find the intervention helpful may have elected not to respond, which would skew results in the positive direction. Additionally, recall bias must be considered when asking respondents about their past behaviors, even over a 3-month period.

Utilizing data such as web tool click rates or log-in data would have provided more “hard” data regarding resource utilization. Survey response rates were acceptable at 50% but could have been higher; in addition, due to the variance in pre- and postsurvey respondents, and the desire to deidentify data, the analysis was performed using an unpaired method. Additionally, this survey study looked at only one pre- and postimplementation time point and, therefore, says nothing about the long-term use of resources, and certainly says nothing about an improvement in knowledge gain or educational achievement. An underlying assumption is that more resource utilization is better, though this may be unproven.

Future Studies

As we continue to build electronic resources to assist in anesthesiology training, further investigations should ascertain what the ideal volume and type of resources would be for such a web tool, and what delivery mechanisms are ideal to introduce these resources to trainees and encourage utilization. Additionally, determinations can be made for what sorts of systems will decrease barriers to faculty-trainee teaching. This group has begun to undergo further investigations to deliver resources from this web tool to providers in a more targeted method utilizing a CDS system. We have begun evaluation of how such targeted delivery methods increase utilization of policies, improve resident reading, and improve trainee knowledge acquisition and retention.

Conclusions

As we hypothesized, the implementation of our web tool did increase the volume of journal articles, local policies, and national policies that our trainees utilized. The web tool also subjectively increased the self-rated comfort of trainees in their utilization of resources, as well as the faculty evaluation of trainee resource utilization. Encouragingly, this common framework for access to shared resources also increased the volume of faculty-trainee discussions about evidence-based resources. We are encouraged with the belief that building from this, such a framework can ease barriers to faculty-trainee educational interactions.