First-year students in postgraduate general practice (GP) training at the University of Leuven in Belgium were invited to participate in this study on a voluntary basis. The postgraduate GP training is for 2 years and starts after successful completion of the 6-year basic medical course. Students were eligible to participate if they were involved in an internship at a GP clinic between January and May 2016 and performed consultations on patients with acute lower back pain during that period. Recruitment was conducted via an electronic mailing list and face-to-face announcements. At this stage of their training, students participate in an integrated curriculum in which they spend approximately 23 weeks at the university, supplemented by a 6-week internship at a GP and a 14-week internship covering various disciplines at the hospital. The workplace supervisors of the participating students were informed about the study via a telephone conversation, an email, and a letter.

A mixed-methods approach was used, consisting of a quantitative section followed by a qualitative section (see Figure 1) [25]. This study was approved by the Social and Societal Ethics Committee of the University of Leuven (reference number: G-2016 01 437/G-2016 01 438).

The effect of the EHR-embedded learning aids on the transfer of learning was assessed by the score difference between the performance on a pretest (a few days before the intervention) and a posttest (5 weeks after the intervention). During both tests, students performed one consultation with a standardized patient suffering from acute lower back pain. Each consultation contained three parts: (1) history taking, (2) clinical examination, and (3) management. There were three scenarios for the pretest and three different scenarios for the posttest. Students were randomly allocated to one of these scenarios. Three medical teachers validated the content and the degree of difficulty of all six scenarios. The standardized patients had more than 5 years’ experience performing such roles and were trained to participate in each consultation in a standardized way. The students could not access the EHR-embedded learning aids during the pretest and posttest. In both tests, the consultation took place in a GP practice, and the students’ performances were video recorded. Two observers (JH and JC), who were not aware of the students’ assignment to one of the two conditions, scored the video recordings with a checklist derived from an objective structured clinical examination (OSCE) checklist (see Multimedia Appendix 2). The checklist consisted of 29 items and covered aspects of the whole consultation. The checklist was assessed for face validity and tested by three medical teachers and an educational researcher. The reliability analysis indicated a Cronbach alpha of .84 for the pretest and .72 for the posttest (McDonald omega of 0.84 and 0.75, respectively).

Both observers had many years of experience with scoring OSCEs. All the video recordings were analyzed independently by the two observers and scores were agreed on during a discussion. Moreover, demographic information of the participants and information about the internship (eg, How many acute lower back pain consultations did each perform during the study?) was retrieved via a short questionnaire to the students and the workplace supervisors. Two additional outcome measures were completed by the participants in the aids plus feedback condition. Firstly, the frequency of students’ use of the EHR-embedded learning aids was recorded via log files. Individual student’s use of the aids was linked to their performance on the pretest and posttest. Secondly, students’ perceptions about the usefulness, perceived ease of use, and the intention for continued use of the aids were measured during the posttest via the technology acceptance model (TAM) 6-point rating scale [21,31]. Each item of the TAM rating scale was translated into Dutch and then revised by two researchers using the translation/back-translation method to avoid semantic problems [32].

An unpaired t test on the difference between groups in pretest to posttest difference was performed using IBM SPSS Statistics for Windows version 24.0 (IBM Corp, Armonk, NY, USA). Additionally, a Bayesian unpaired t test was performed because conventional t tests (eg, statistical significance tests) may provide some evidence against a null hypothesis (eg, in the case of a statistically significant outcome), but cannot provide evidence in favor of a null hypothesis [33]. The Bayes factor hypothesis testing approach allows for comparing the likelihood of a finding under the null hypothesis with that under an alternative hypothesis [33,34]. The Bayesian analysis was performed in JASP version 0.8.1.1 [35].

Semistructured interviews were conducted with a voluntary sample of the students in the aids plus feedback condition. An attempt was made to gather a wide range of participants, both in terms of their demographics and their experience with the learning aids at the workplace.

The open-ended interview questions were based on three general questions: (1) What are your perceptions about using these EHR-embedded learning aids during clinical practice? (2) Why and how did you use these aids? and (3) What is the feasibility of using such aids at the workplace? The interview questions were revised by three researchers, and two pilot interviews were conducted to test the interview questions and to practice interview techniques. The interviews were recorded and transcribed verbatim. An iterative process of data collection and analysis took place [36]. The principle of data saturation was applied. The interviews were coded and analyzed independently by two researchers (AR and SP), using the software program QSR International’s NVIVO version 11. Following the thematic analysis approach, the initial coding phase focused on small units of the transcripts to ensure that the most prominent ideas were identified. In the second coding phase, broader categories containing a number of conceptually related ideas were developed [37]. Discrepancies in coding were discussed until consensus was reached.

Twenty-two students participated in the feedback condition. Two of those participants dropped out before the posttest. There were 28 participants in the aids plus feedback condition, of whom four dropped out before the posttest. The data from the participants who dropped out were not taken into account for the data analysis. Descriptive statistics are reported in Table 1.

The checklist to assess students’ performances on the pretest and posttest contained 29 items (Multimedia Appendix 2). Items 28 and 29 were not included in the data analysis because the researchers realized that these two items were too general and difficult to score.

The unpaired t test on the difference between groups in pretest to posttest showed that participants in the aids plus feedback condition scored slightly higher than the students in the feedback condition, but this difference was not statistically significant (t₄₂=–0.511, P=.61, 95% CI –4.86 to 2.90). The Bayesian unpaired t test revealed a Bayes factor for the null versus the alternative hypothesis of 3.015, indicating some evidence in favor of the null hypothesis of no difference between the two conditions. Without the data of the student in the aids plus feedback condition who did not access the EHR-embedded learning aids, the conventional t test was t₄₁=–0.600 (P=.55, 95% CI –5.12 to 2.77) and the Bayes factor for the null versus the alternative hypothesis was 2.877.

The students’ log files in the aids plus feedback condition showed that the EHR-embedded learning aids were used 451 times by students. Students in the feedback condition reported that they did not frequently access the aids through the course notes while on an internship (never: n=13; sometimes: n=5; usually: n=1; always: n=1).

The students’ perceptions about the learning aids are shown in Table 2. Students found the aids generally useful, easy to use, and they had the intention to continue using them. Yet, the standard deviation is rather large.

Although the learning aids were developed for the students, 12 of 24 workplace supervisors in the aids plus feedback condition reported that they also used the EHR-embedded learning aids.

Data saturation was reached after 16 interviews with students in the aids plus feedback condition. Table 3 shows a detailed summary of the qualitative findings. The data analysis indicated that all students reported having used the learning aids, but only a small fraction of them had done so during the consultation. The following reasons were given: low confidence or resistance to accessing the EHR during patient encounters, supervisor’s influence (eg, time pressure), patient’s influence (eg, patient’s individual situation), practical limitations, small number of consultations performed with patients suffering acute lower back pain, and in some cases students knew the content of the aids already.

Students who used the learning aids during consultations mentioned that the aids were helpful to check if anything had been forgotten or to clarify uncertainties. Some participants explained that the aids helped them to feel a bit more self-confident. Students described that it was feasible to use the aids during consultations because they were familiar with them as they reflected what was learned before, contained brief and practice-based information with step-by-step instructions, and were presented in an easily accessible format. The beneficial characteristics of the learning aids were perceived to be missing in the general evidence-based guidelines available on the EHR. Although the learning aids were developed for use during consultations, the majority of students used them before and after consultations. Participants mentioned that they used the aids to refresh acquired competences before consultations because they were anxious about making mistakes or perceived weakness in this area of their practice. Some students explained that these aids helped them to be more aware of the learning content and to apply this in the workplace. Students also found it useful to check afterward whether they had performed the consultation correctly. Some students explained that they also used the aids to discuss their performance with their supervisor or to reflect on their supervisor’s performance. The aids were even perceived as tools for supervisors’ professional development.

Students thought it was very useful and feasible to use the EHR-embedded learning aids because they were familiar with them and they contained brief, practice-based, step-by-step instructions presented in an easily accessible format. Moreover, the aids were frequently used by students. This suggests that they addressed a need for this kind of support. Students described that the aids facilitated transfer of learning because they allowed them to refresh and check classroom-acquired competences at the workplace. Additionally, the aids appeared to give students more self-confidence, supported students’ reflections, and stimulated interactions with supervisors. Yet, no significant difference was found between offering support for feedback with the additional access to EHR-embedded learning aids and solely providing support for feedback without the additional aids.

Participants in this study described that the learning aids often could not be directly applied in practice because they required tailoring to a patient’s circumstances, as pointed out in previous research [8]. Transfer research emphasizes that transfer is not a simple “store and retrieve process” but involves active interpreting, modifying, and reconstructing the competences to be transferred [12]. To help students better understand how classroom-acquired competences are transformed into clinical decisions for a particular patient, feedback from the supervisor is essential [24,26]. Some participants mentioned that the feedback sheet stimulated their supervisors to give better feedback.

Additionally, students in this study mentioned that their use of the learning aids also depended on their supervisor. This is in line with previous research that shows that evidence-based medicine (EBM) decision-making processes are influenced by patient factors as well as general practitioner factors (eg, time pressure, attitude, and experience) [13]. Also, the GP supervisor’s role as a model for students is important because their attitude toward the use of EBM and learning aids may be a barrier for students [38,39]. This is especially the case for GP students and trainees because they generally have just one supervisor [40].

Students in this study found the way in which the aids reflected previously acquired competences was beneficial for transfer of learning. This is in line with previous research that showed that spaced repetition of what was learned before promotes students’ memory, problem solving, and transfer of learning [41]. Moreover, research indicated that familiarity with learning aids is one of the crucial elements for medical doctors and students to use these aids [19]. Among other reasons, familiar resources might help with finding answers to point-of-care questions more quickly than unfamiliar resources [19,42]. This is especially relevant for students because it allows them to focus on the application of classroom-acquired competences rather than losing time searching for the desired information [43]. Yet, the learning aids in this study may not be up-to-date after a period of time. Nevertheless, the aids were not intended as a replacement of evidence-based guidelines but rather as a supplement.

Participants in this study explained the added value of learning aids alongside evidence-based guidelines. The latter often consist of long passages of text and are perceived to be too complex to be useful, often requiring a lot of time for searching and reading [13]. Some advantages of the learning aids in this study were their brevity, practice-orientation, and step-by-step design in an easily accessible format.

Previous research indicates that practical tools can encourage transfer of learning [8,15]. Moreover, these practical tools or aids can be adjusted to students’ individual needs or extended with students’ personal documents so that they become “personalized learning aids” [29]. This allows richer and deeper personalized learning experiences [22,44]. Additionally, students can become actors to develop classroom-based learning aids themselves, which may be shared with other students [22,28,44,45]. Additionally, students and supervisors could work together on such projects to bring classroom and workplace learning closer together. Previous research indicated that collaboration between supervisors and students can take place by involving students in a shared project [12,42]. Moreover, learning aids could also be based on existing sources available at the workplace, such as evidence-based guidelines. Bringing such sources into the classroom more frequently might allow students to familiarize themselves with them.

The EHR-embedded learning aids have a lot in common with electronic performance support systems. These are electronic systems that offer support during performances at the workplace [29]. They are mainly used during workplace performances but can also be used prior to and after performance [29]. Rather than using the learning aids during patient encounters, as was the intention of the researchers, most participants of this study used the learning aids before and/or after patient encounters. This is in contrast with previous research that showed that students or trainees primarily seek answers to clinical questions during patient consultations [46,47]. Yet, it indicates that students’ individual perceptions influence how they use educational support, as shown by previous research [21,22].

A limitation of this study was the small number of participants and the limited intervention period. Nevertheless, the use of both a quantitative and a qualitative approach in the study design was a strength. This allowed investigation of students’ competences and perceptions as well as more insight into students’ use of the learning aids. Yet, interviews were only conducted with the aids plus feedback condition. Interviews with the feedback condition could have been informative too. Another limitation was that pretests were used although it is known that these are not learning neutral. They may stimulate learning to the test and affect performance on an identical posttest [48], which could have diminished the effect of the learning aids. Another limitation was that the pretest and posttest only contained one OSCE. Yet, it was a strength that the OSCE in this study was based on a whole consultation rather than short stations assessing clinical competences in isolation. The use of whole consultations and integrated competence assessments more closely reflect the real patient encounter [49]. Additionally, given that the OSCEs in this study were videotaped, it allowed the two observers to review elements that they were unsure of on the first viewing [50]. Regrettably, individual student use of the learning aids could not be linked to their performance on the pretest and posttest. Although it was the researchers’ intention to do so, it was impossible due to a failure of the logging system. Another limitation was that there was no information regarding the duration of time in which students accessed the learning aids. This study focused on the use of EHR-embedded learning aids, but “use” does not automatically translate to the application in practice. Previous research indicated that using EBM (or learning aids) may be important, but alone it is insufficient to improve clinical practices or patient care [51]. It was a limitation of the study that the quality of the received feedback was not assessed. Yet, materials and instructions were provided to generalize its format. The study results showed that students in the aids plus feedback condition were able to perform more consultations and they received more feedback than students in the feedback condition, which may have influenced the results. It is possible that the extra reminder via email and postal mail for supervisors in the aids plus feedback condition plays a role, although this remains unclear. It might, for example, also be possible that students took more initiative because they had learning aids available in the workplace. Furthermore, random assignment to the conditions was not feasible in this study. Given that the students ended up in two groups by a mechanism (time of registration for the postgraduate GP course) unrelated to the two conditions, it was assumed to be sufficient. It was a strength of this study that participants were unaware of their assignment to one of the two conditions.

During patient encounters, there is often not much time to search for information in evidence-based guidelines [13]. This study showed that information that is easily accessible, user-friendly, and familiar to students facilitates the information search. This raises the question whether changing the format of evidence-based guidelines and implementing a design more in line with the characteristics of the learning aids in this study (practice-based, step-by-step instructions and easily accessible format) may be helpful. This study also indicated that students often consulted the learning aids before or after patient encounters, provided they were familiar with the aids. This implies that access should be possible at any time and available aids should be made familiar by using them in classroom-based learning activities. This study also showed that students felt restrained when using learning aids during patient encounters, although they perceived the aids as useful, easy to use, and an added value above the existing evidence-based guidelines. This study provides insight into the interfering factors in the transfer of learning, such as the supervisors’ role model. Future research should explore how these interfering factors could be overcome (eg, the influence of supervisors’ educational training).

Access to EHR-embedded learning aids, in addition to feedback, did not seem to enhance transfer of learning based on the results of the pretest and posttest. However, students perceived the aids as helpful in their transition from medical school to the workplace. They expressed that the learning aids offered additional support during, but also before and after, patient encounters. The learning aids can be easily implemented into workplace-based learning to assist both students and supervisors.