We developed a web-based survey to assess student information-seeking behavior, formal instruction on information searches, and evaluation of sources of health information. The survey was developed based on a review of relevant published literature and pretested with a convenience sample of ten medical students. The final survey contained 20 questions, predominantly requiring yes/no or rating scale responses; one question required a numeric estimate, one asked students to list their five most-used resources, and two were open-ended. Questions regarding resource use were based on recall of the previous seven days. The list of survey questions is included in Multimedia Appendix 1.

All students registered in the Dalhousie University 4-year medical doctorate program, at either the Halifax or the Saint John site, were considered eligible for participation in the survey. At the time of the survey, there were 462 registered students. We considered students in years 1 and 2 preclinical and those in years 3 and 4 clinical because the first two years are classroom-based while the third and fourth years take place predominantly within clinical settings.

The survey was published online using the Opinio 6 survey platform (ObjectPlanet, Inc) and remained open for two weeks. Two of the study authors (AO and EW) made in-class announcements to each student cohort, and invitation and reminder emails were sent to eligible students.

The survey responses were exported from the survey platform into Excel (Microsoft Corp). Data were analyzed using descriptive statistics, computed by hand. Frequencies were reported as percentages. Where appropriate, 95% confidence intervals were calculated using the .cii command with exact binomial confidence intervals using STATA version 12.1 (StataCorp LP). No correction was made for multiple statistical testing.

One question asked respondents to list their five most-used resources from the previous seven days. We assessed response frequency, and used Wordle [26] to generate a graphical representation of this data in which type size reflects frequency of occurrence.

Selected data were subsequently graphed using R (The R Foundation). In keeping with a paper presented at the 2011 Joint Statistical Meeting [27], these Likert data were presented graphically, using diverging stacked bar charts across information sources.

Open-ended questions were analyzed qualitatively for themes by AO. Responses were read and common themes identified; on second reading, responses were categorized into thematic groups. Responses that reflected more than one theme were included in each relevant thematic group.

The study was formally reviewed and approved by the IWK Health Centre Research Ethics Board. We also received approval from the Dalhousie Undergraduate Medical Education Curriculum Committee.

Of 462 students invited to respond, 213 (46.1%) provided evaluable responses. Two-thirds of closed-ended questions received response rates of 42.0% or above (193 or more students), and all but one question had responses from at least 39.0% (180 students). Ten students accessed the survey but did not provide any responses; these were considered to be nonrespondents and were not included in the response rate figures.

Of the respondents, 56.6% (120/212) were female and 42.0% (89/212) were male; 3 preferred not to answer this question. Preclinical and clinical training levels received comparable representation (110/212, 51.9%, and 102/212, 48.1%, respectively). Roughly half (109/207, 52.7%) had prior experience as contributors to peer-reviewed literature, while relatively few (35/207, 16.9%) had posted information online for the public.

While 80.9% (165/204; 95% CI 74.8%-86%) of respondents recalled receiving formal instruction on searching for health information, 67.1% (139/207; 95% CI 60.3%-73.5%) also recalled being discouraged from using certain resources. Education regarding bibliographic databases (eg, PubMed) was common (162/206, 78.6%; 95% CI 72.4%-84.0%), whereas respondents seldom had instruction regarding general purpose Internet sources. Only 11.1% (23/207; 95% CI 7.2%-16.2%) reported teaching regarding the general search engine Google, and fewer had education on Wikipedia (a free, online, open-content encyclopedia) or social media. Although self-rated competence in finding information showed some variability, a strong majority (184/207, 88.9%; 95% CI 83.8%-92.8%) felt their skills were good or better.

We considered a resource to be heavily used if a student reported use 4 or more times in the previous 7 days (Figure 1, Table 1). Most respondents use Google on a daily basis (154/202, 76.2%; 95% CI 69.8%-81.9%), and very few reported using the search engine on fewer than 4 of the prior 7 days (13/202, 6.4%; 95% CI 3.5%-10.8%). Wikipedia, UpToDate (a subscription-only, evidence-based summary source), and personal/provided notes were also heavily used. In contrast, few respondents (28/199, 14.1%; 95% CI, 9.6%-19.7%) had used bibliographic databases on at least 4 days, and just over one-third (68/199, 34.2%; 95% CI, 27.6%-41.2%) had not used such sources at all in the previous week.

Comparison of preclinical- and clinical-level respondents revealed two prominent differences: clinical-level respondents reported less frequent use of bibliographic databases (P<.001) and more frequent use of UpToDate (P<.001) (Table 1).

Of note, these results were somewhat different from respondents’ self-generated lists of most used resources. Here, UpToDate was the most-listed single reference, while Google and Wikipedia were the next most commonly cited resources (Figure 2).

Whereas students valued the general purpose resources Google and Wikipedia highly for their accessibility, understandability, and usefulness, they gave PubMed and other bibliographic databases stronger ratings for accuracy and trustworthiness. As a representative EBM summary, UpToDate appeared to bridge this gap: it received high ratings across values although many students noted elsewhere that access to this resource was limited by cost and lack of an institutional subscription (Figure 3).

Students identified key factors in their impressions of a source’s trustworthiness: recognition of factual errors in a reference and reputation among mentors and peers, as well as being specifically counseled to use or avoid a reference.

Students offered valuable insights in their responses to open-ended questions, and several themes emerged. Many spoke of being overwhelmed by the number of available and/or suggested resources as well as by the density of information in these sources; respondents repeatedly reported a need for more basic information and for managing information/source overload. They requested faculty- and/or peer-generated resource recommendations, although some noted that current lists added to a sense of overload. Perhaps in response to this challenge, several requested increased and longitudinally integrated instruction on searching for information and often highlighted a desire to learn how to approach nontraditional, or general purpose, Internet sources. Finally, resource accessibility presented an additional, practical barrier to finding useful information. Respondents commonly identified a need for freer access, including at the point of care; they cited lack of universal Internet access, inconvenience of multiple sign-ins, and subscription-based access as barriers.

Our survey assessed patterns of information resource use among students in the undergraduate MD program at Dalhousie University. Students in the program must have an undergraduate degree; some have additional study or work experience. The school has two campuses (Halifax, Nova Scotia, and Saint John, New Brunswick); the sites share a uniform curriculum, and many lectures are videoconferenced. All students have remote access to Dalhousie University library electronic resources throughout their training.

The medical school follows a 4-year curriculum. Two classroom-based preclerkship years aim to provide a foundation of knowledge and basic skills. A two-year clerkship follows, spent mostly in clinical settings.

A concentrated series of lectures on EBM are delivered at the beginning of year one. On a recent syllabus, seven hours during that unit were devoted to aspects of EBM, including five hours on question formulation and searching for evidence. Lecture notes refer to a standard hierarchy of preappraised evidence and discuss how and when to use various resources [Robin Parker, Sources of evidence, 10 September 2012, Dalhousie University, Canada].

Information management and EBM continue to receive attention at various points after this introductory unit. Weekly sessions during the first two years address evidence search and appraisal and other topics (ethics, law, population health, and professionalism). The formal curriculum is more limited during the clinical years, and much of the planned teaching time addresses clinical skills; training objectives continue to refer to critical use and appraisal of evidence.

Students have access to an array of resources and resource guides. They receive reference recommendations for each unit, and notes and links are posted on general and course-specific websites. The library website houses subject guides for each course and clinical rotation, including guides to search strategies for PubMed and Cochrane Library, a subscription-based set of medical and healthcare databases.

Our study has limitations. We conducted the survey at a single medical school. The response rate was lower than desired. Our response rate appears comparable, however, to that achieved by previous online surveys addressing technology and information management in medicine [20-22,42].

Our survey relied on self-reported behaviors, which are known to be subject to social desirability bias. External observation of behavior, however, is not feasible for the current inquiry. A daily diary approach might have offered greater accuracy but at a further cost to the response rate. We asked respondents to report practices from the preceding 7 days as a strategy to obtain more accurate self-reported data than a more general inquiry. Despite this, the difference seen between the frequencies of use reported for different resources and respondents' self-generated lists of most-used resources suggested the possibility of bias in some questions. We would, however, have expected social desirability bias to skew results away from non-traditional, general-purpose sources—instead we saw quite a dramatic favoring of such resources over bibliographic databases.

Our survey obtained a cross-sectional assessment, with reference to behavior during the preceding seven days. It is possible that student behavior differed from usual practice during the week prior to the survey, rendering nonrepresentative results. In an effort to limit this possibility, we discussed survey timing with the Dalhousie Undergraduate Medical Education Curriculum Committee and student leaders and selected survey dates that fell during the routine session, avoiding examination or recess periods.

No tool is optimal for every purpose. Students need to gain skills and familiarity assessing primary literature, but they also must learn to find useful, practical information efficiently. They need to recognize what tool will best serve a given purpose—whether this is getting an overview of a common condition or physiologic process or assessing effectiveness of alternate medical therapies—and to appreciate the limitations inherent to that tool. Thus, whereas current formal instruction on information searching neither reflects nor appears to alter,student behavior, consideration should be given to instruction on information management and appraisal in general. Medical education that includes use and appraisal of primary literature, summary sources, and even general purpose resources might more effectively equip students in their pursuit of lifelong learning.