Preclerkship medical students were recruited from four Canadian medical schools over 2 years from April 2014 to December 2016. Convenience sampling was used with informal recruitment through word of mouth, social media, and email by 2 authors (RK and DK). The primary inclusion criterion was that participants must be medical students in preclerkship training (ie, in their first or second year of medical school) at a Canadian medical school. After recruitment, participants were anonymized with a unique identifier and randomized in an allocation ratio of 1:1:1 to one of the three groups: (1) Wikipedia, (2) UpToDate, and (3) digital textbook. The random allocation sequence was created by one author (MAS) using a Web-based random number generator. Another author (CW) assigned participants to each of the three groups using sequentially numbered assignments. Participants were blinded to group assignment until they were required to use their intervention. Once they began using their Web-based resource, blinding was not possible. Data analysts were blinded to group assignment. Participants were not told which Web-based resource was the intervention of interest.

The study methodology is summarized in Figure 1 . All participants completed a questionnaire of their demographics. Participants then had 30 min to complete a multiple-choice questionnaire (MCQ) pretest of 25 questions, wherein they could take written notes of questions they wanted to research further. After the pretest, participants had 30 min to research the questions using their assigned Web-based resources, during which they were allowed to create written notes. Finally, participants completed the same MCQ as a posttest, wherein they could use their written notes and general knowledge acquired from the intervention. They also completed a follow-up survey on the Web-based resources.

Data collection was done using two formats: in-person and online. Online administration was used to connect with remotely located participants and was conducted the same way as in-person collection. The two data collection methods differed only in degree of interactivity with the study coordinator. During in-person data collection, a study coordinator was present throughout the entire administration, who assigned participants to individual computers and only interacted with participants for consent, initial test setup, and notification of time remaining on each section. During online collection, screen-sharing software was used to track participant completion of the tests and to indicate time remaining on each section. In both scenarios, participants completed surveys, tests, and intervention using a standard Web browser and Google Forms software.

Study coordinators only answered questions regarding logistics (eg, remaining time) and did not advise participants on test content or Web-based resource navigation. Coordinators also ensured that participants only used their assigned intervention through direct observation or screen-sharing.

Participants completed a MCQ of 25 questions that emulated questions on the Medical Council of Canada Evaluation Exam (MCCEE). The MCCEE is a standardized examination administered by the Medical Council of Canada (Ottawa, Canada) to assess basic medical knowledge and readiness for postgraduate medical training in Canada. The content of the MCCEE is aimed toward graduating Canadian medical students to ensure that participants (ie, preclerkship medical students) would not have considerable prior knowledge. Test questions were retrieved from an MCCEE site, which is freely available online [20]. These questions were imported into a Google Forms questionnaire and delivered online. Questions were reviewed by 2 academic physicians (SG and JH) to ensure broad coverage of topics and appropriateness.

After recruitment, participants were randomized to one of three Web-based resources: (1) Wikipedia, (2) UpToDate, and (3) digital textbook. During the testing, participants each had 30 min to access the Web-based resource and could make notes using pencil and paper on any topics or questions on the test to research using the assigned intervention. Wikipedia and UpToDate were accessed using an Internet browser, with the participants logging into the latter using institutional accounts. The digital textbook, Harrison’s Principles of Internal Medicine, 18th edition, was accessed through institutional accounts.

Participants were limited to only their assigned interventions and were not allowed to search for additional information online. Moreover, participants were not provided with guidelines or strategies on how to access information. Coordinators tracked participant progress to ensure adherence with the assigned interventions. Within the allotted time, participants used a self-directed approach to research topics relevant to the MCQ.

To test for knowledge acquisition, participants completed the same MCQ administered at the beginning of the study. During this iteration, participants could refer to their written notes as a reference. After completing the test, participants answered a follow-up survey regarding five electronic resource factors: search functions, hyperlinks to other pages within the resource, references, open access editing, and expert editing. In the first section of the survey, participants rated the importance of each of the five factors in their learning. In the second section, participants then rated the availability of each of the factors within their assigned resource. For perceptions of importance, participants rated the five factors with respect to their general importance when using Web-based resources on a Likert-type scale of 1 to 5, where 1 represented not important at all and 5 represented very important. For perceptions of availability, participants rated the five factors with respect to only their assigned resource (Wikipedia, UpToDate, and digital textbook) on a Likert-type scale of 1 to 5, where 1 represented not at all available and 5 represented very easy to access.

The primary outcome of the study was the difference in knowledge acquisition between the three groups as indicated by percentage scores on the MCQ. The tests were graded using a scoring key on a scale of 0 to 25. Each correct answer was awarded one point; incorrect answers or omissions were not penalized. Secondary outcome measures were the participants’ perceptions on availability of the five following factors: search functions, hyperlinks to other pages, references, open access editing, and expert editing.

On the basis of previous research on knowledge acquisition using Web-based resources among medical trainees, 28 participants per group have been sufficient to detect significant differences between four groups [21]. To account for potential dropout, 116 participants were recruited.

Data were analyzed using Statistical Package for the Social Sciences (SPSS) version 20 (IBM Corp). Demographic variables were represented using descriptive statistics. All quantitative data were represented using means with standard deviations or medians with interquartile range, where appropriate. Categorical data were represented by count with frequency.

Primary analysis was intention-to-treat. To determine a difference in the MCQ scores across the three groups, a two-way mixed-factorial analysis of variance (ANOVA) was completed with one within-group factor (test: pretest and posttest) and one between-group factor (group: Wikipedia, UpToDate, and digital textbook). To determine whether there were any differences in participants’ perceptions of the importance and availability of the five resource factors for the resources (Wikipedia, UpToDate, and digital textbook), a Kruskal-Wallis test was used. Any significant effects on ANOVA or Kruskal-Wallis tests were further analyzed using Tukey honestly significant difference (HSD) and Mann Whitney U post hoc tests, respectively. Additionally, an exploratory analysis was conducted; two-way ANOVA was performed for posttest MCQ scores using highest level of education before medical school (group: masters, PhD, other professional degree) and assigned resource (group: Wikipedia, UpToDate, and digital textbook). The assumptions for the mixed ANOVA and two-way ANOVA were assessed and the appropriate corrections were applied for any violations [22]. Following statistical reporting recommendations, effect size was calculated using generalized eta squared (η_g²) and Cohen measure (d) for ANOVA and Tukey HSD post hoc tests, respectively [23]. Alpha was set at .05 for all statistical tests.

A total of 116 preclerkship medical students were randomized and completed the study from April 2014 to December 2016. No participants were lost to follow-up. Participant demographics, prior resource use, and data collection format are provided in Table 1. Participants’ perceptions of the importance of several resource factors with respect to general Web-based resources are shown in Table 2. There were no significant differences between the groups on any of the five factors (P>.05).

MCQ responses for each group are shown in Figure 2. There were no significant differences between the three groups at pretest (Table 3). The ANOVA of the MCQ scores indicated a significant interaction between time and group effects (F_2,113=10.07, P<.001, η_g²=0.03). Post hoc analysis indicated that the Wikipedia group scored significantly higher on the posttest compared with the textbook (P=.01, d=0.86). There were no other significant post hoc pairwise comparisons between the other two groups. On the two-way ANOVA, there was no significant interaction between group assignment and highest education before medical school for posttest MCQ scores (F_4,106=171.85, P=.51).

Participants’ perceptions of the availability of resource factors within their assigned resource are shown in Table 4. There were significant differences between the groups on all the five factors (P<.001).