Meeting the Shared Goals of a Student-Selected Component: Pilot Evaluation of a Collaborative Systematic Review

Introduction

In Outcomes for Graduates [1], the General Medical Council states that medical graduates should be able to apply the scientific method and understanding of medical research when making decisions regarding patient care. Opportunities for medical students to be involved in research are now required by all medical schools in the United Kingdom. This commonly takes the form of student-selected components (SSCs), a dedicated period in the medical course where medical students can engage in a diverse range of research opportunities [1,2].

Medical students are not always able to seize the full potential of SSCs due to several factors. First, teaching in research methodology is inconsistent among undergraduate medical curricula [3]. Second, the duration of an SSC is relatively short for a project to be completed [4,5]. Limited prior research training and difficulty identifying a manageable project with good mentors provide further challenges for those with little prior research experience [3,6]. Together, these factors can leave medical students feeling poorly prepared, overwhelmed, and insufficiently supported, which can ultimately lead to a poor experience of research and eventually disengagement [7,8].

Review articles are the most common article type published by medical students [9,10]. Systematic reviews (SRs) combine a high likelihood of publication with the ability to actively contribute to research, allowing students to acquire fundamental research and evidence-based medicine skills [11,12]. As part of a quality improvement initiative, we hypothesized that a collaborative approach to SR may offer a solution to these problems. We aimed to explore whether offering a small team of students the opportunity to take part in the title and abstract screening phase of SRs while being led by an experienced postgraduate team could enhance the learning experience, overcome the barriers to successful research engagement, and deliver published output.

Methods

SR Conception

In all, 2 SR articles were devised by postgraduate researchers based on the current research interests of the Degenerative Cervical Myelopathy (DCM) Research Group in Cambridge, United Kingdom. Both SRs were in due reference to the priorities of patients with DCM, expressed through forums including Myelopathy.org, an international myelopathy charity, and the Research Objectives and Common Data Elements for DCM process, an international consensus process to define the research priorities for DCM [13-15]. The topics of the reviews were (1) the impact of phosphodiesterase 3 and 4 inhibition on neurobehavioral outcomes in preclinical models of traumatic and nontraumatic spinal cord injury and (2) the role of cannabinoids on modulating neurobehavioral outcomes in preclinical models of traumatic and nontraumatic spinal cord injury [16]. Both reviews were registered on PROSPERO (University of York, United Kingdom; CRD42019150639 and CRD42019149671, respectively). Search strategy and protocol development was led by the 2 lead students, with reference to previous SRs conducted by our group, followed by review, discussion, and feedback from postgraduate researchers [15-21].

Recruitment

A national advertisement was disseminated by the national network of the Myelopathy.org Student Society to recruit medical student and junior doctors interested in participating in the title and abstract screening phase of the SRs. A total of 14 students applied to be involved. All 14 students were invited to participate to promote inclusivity given the flexibility in the number of students that could be involved.

An undergraduate medical student was selected to lead each review under the supervision of postgraduate researchers and a medical librarian at the University of Cambridge.

Collaborative Process

Postgraduate researchers provided the 14 students with training, including written guidance, on the process of title and abstract screening, in addition to search strategy and inclusion and exclusion criteria formulation. All students were given the opportunity to email questions, and explanations were provided. Rayyan software (Rayyan Systems) was used to enable a collaborative multiresearcher approach to the screening of titles and abstracts, ensuring that each article was independently reviewed by 2 students [22]. Initially, a Rayyan sandbox containing a pilot sample of 100 titles and abstracts was created. All 14 students screened the 100 titles and abstracts. The student pilot-screening results were then compared to those of the postgraduate researchers. Subsequently, definitions were clarified and explanatory statements for the inclusion and exclusion criteria were revised to ensure strong interstudent reliability.

The 14 students were then equally involved in completing title and abstract screening for the 2 SRs. A total of 10,251 titles and abstracts were allocated (8714 and 1537 articles from the 2 SRs) such that each title and abstract was screened by 2 students. This resulted in each student screening 1464 articles. Following the completion of screening, the 2 leading undergraduate students then completed the remainder of the SRs. As a pilot evaluation of this approach, this was a pragmatic decision, given the uncertainty of the effectiveness of the collaborative approach. The remaining 12 students were updated on project progress and provided with written materials on the key stages of SR, in addition to specific examples from the present SRs.

Survey Design

To enable the assessment of the effectiveness of this methodology, participating students completed 3 surveys throughout the process. The first survey was conducted prior to the pilot screening of 100 articles, the second after the completion of pilot screening and during screening of the titles and abstracts for the 2 SRs, and the third after the completion of all title and abstract screening and the provision of the written summary of the remaining SR methodology. Figure 1 illustrates the timings of the surveys. All 3 surveys assessed students’ perceptions of research; experience of this collaborative initiative; and their “knowledge,” “confidence,” and “experience” of SR methodology. SR methodology was divided into 12 components: question formulation, development of a search strategy, development of inclusion and exclusion criteria, title and abstract screening, full-text screening, risk of bias assessment, development of an extraction template, data extraction, data synthesis, data interpretation, manuscript writing, and presentation skills. In addition, the first survey captured information such as the stage of training, prior research experience, and the amount of research methodology teaching received.

In total, there were 85 questions across the 3 surveys. Of these, 67 questions were close ended in Likert-scale format with a scale from 0 to 10, with 0 being “strongly disagree,” 5 being “neither agree or disagree,” and 10 being “strongly agree.” The full list of questions in each survey is available in Multimedia Appendix 1. The questionnaires were hosted using the SurveyMonkey platform (Momentive Inc). Each student created a unique identifier that was entered each time they completed a survey to allow changes in perceptions to be anonymously measured over time. Reminders for survey completion were sent to students throughout the process; however, survey completion remained voluntary.

‎

Data Analysis

Survey results were exported into Microsoft Excel, where responses were collated. Descriptive statistics, including means and SDs, were calculated where appropriate. Inferential statistical analysis was not appropriate given the small sample size of students (N=14).

Ethical Considerations

Ethical approval was not obtained as this project was considered an initial part of a quality improvement process looking to improve student experience of SSCs. The findings are intended to inform the optimization of a teaching program that would still need subsequent evaluation. This was checked with the Human Research Authority, using their decision aid [23] to arrive at this conclusion.

Results

Response Rates

All 14 students responded to each of the 3 surveys, answering all the questions apart from 2 questions where 1 student did not respond (questions assessing the experience of full-text screening and experience of manuscript writing).

Student Demographics and Prior Research Experience

Demographics and previous research experience are summarized in Table 1 (see Multimedia Appendix 1 for additional information). When asked what specialties they were interested in, 10 (71%) out of 14 students expressed interest in neurology or neurosurgery, and 10 (71%) considered research to be necessary to secure a training post in their desired specialty.

Research Methodology Teaching Received

A summary of the amount and form of research methodology teaching students received and their perceptions are provided in Table 2. The most common form of teaching was lectures (6/14, 43%). Of the 14 students, 4 (29%) agreed with the statement, “I have had sufficient training in research methodology at medical school”; whereas 2 (14%) students strongly agreed and 5 (36%) students agreed with the statement, “I have had sufficient opportunity to participate in research at medical school.”

How Did Perceptions of Research Change Throughout the Process?

Table 3 summarizes how perceptions of research changed during the collaborative SR training process. There were increases in the responses to “I am good at research,” “I am confident at research,” “I am experienced at research,” “I have experience conducting systematic reviews,” “I am confident with the theory of a systematic review,” and “I am confident with the practicalities of conducting a systematic review.” There was otherwise little change in the perceptions of the other statements. The average response to “I enjoy research” and “Research is interesting” in the prepilot survey was 8.07 (SD 1.59) and 8.21 (SD 1.88), respectively. Similarly, the average response to “I would consider being involved in research in the future” was greater than or equal to 9 in all 3 surveys.

How Did Knowledge, Confidence, Experience of SR Methodology Change Throughout the Process?

Tables 4-6 and Figures 2-4 illustrate how knowledge, confidence, and experience of the 12 components of SR methodology changed before, during, and after title and abstract screening. An increase in mean scores of knowledge, confidence, and experience of all 12 components was noted in the postscreening survey compared to the prepilot survey. The largest increases in knowledge (before: mean 5.57, SD 3.32 vs after: mean 8.50, SD 1.45), confidence (before: mean 5.07, SD 2.89 vs after: mean 8.14, SD 1.75), and experience (before: mean 4.00, SD 3.46 vs after: mean 7.93, SD 1.69) across the process were noted for title and abstract screening.

‎

‎

‎

Evaluation of the Process

Figure 5 highlights student students’ evaluation of the collaborative process across the 3 time points. Additional questions were asked in the final survey, and the mean responses to these statements are as follows: “This collaborative SR improved my research experience” (mean 8.50, SD 1.56), “My understanding of research methodology improved as a result of being part of this review” (mean 7.64, SD 1.86), “Being involved in this research has made me more likely to do research in the future” (mean 8.57, SD 1.50), and “Being involved in this research has made me more likely to do myelopathy research in the future” (mean 7.2, SD 2.39).

When asked whether the “Overall experience was worthwhile,” all 14 (100%) students responded “yes.” When asked, “Would you have preferred to be involved in all stages of the review?” 11 (79%) of the 14 students responded “yes.”

‎

Research Output

As of the time of writing, 1 of the SRs has been published and the other is being prepared for submission [16].

Discussion

Principal Findings

Our study provides insight into the perspectives of medical students involved in a trial of a collaborative approach to SR, in which students were given the opportunity to be involved in research while being closely supported by experienced postgraduate clinical researchers. Within the practical limitations of students primarily being involved in title and abstract screening, the responses to our questionnaires suggest the approach was well received by those involved.

With regard to prior understanding of research methodology, the questionnaire identified that the teaching of research methodology received by students varied in format and quantity. All students involved received at least some form of teaching on research methodology at university; however, only 29% of students agreed that the teaching they received was of sufficient quantity. This finding is in alignment with a larger questionnaire of medical students, which found that 43% of respondents felt their medical school provided adequate research training [3]. SSCs present students with a learning opportunity to gain insight into research that may not have been provided through medical school lectures, seminars, or tutorials.

The collaborative approach appeared to be useful in introducing students to research. A negative trend in the perceived difficulty of research was observed across the 3 questionnaires, which could suggest that a collaborative approach, such as this one, may be helpful in making research more accessible for medical students. Positive trends in self-reported knowledge, confidence, and experience of SR methodology were also noted. The biggest changes in knowledge, confidence, and experience were for the process of title and abstract screening. This was the process that the medical students were actively involved in and gained hands-on experience of. Active learning in which students have opportunities to participate and engage with their learning is supported by adult learning theory and is being increasingly used in medical education [24,25]. Furthermore, students reported that the understanding of research they obtained from being involved in this program could not have been obtained from “passive learning e.g., textbook or lecture.” Given that this was an initial trial of this collaborative approach to SRs, students were primarily involved in title and abstract screening. Future projects involving greater student participation, for example, in data extraction, may prove useful in further elucidating the efficacy of collaborative approaches to SRs.

It has previously been shown that poor initial experiences with research can lead to disengagement [7,8]. On the other hand, positive experiences of research with good mentorship are associated with increased interest in research and future research participation [26,27]. The benefits of successful research engagement are not limited to research and academia [5]. Research placements provide an opportunity for medical students to gain deeper insight into a specialty of their choosing, thus placing them in a position to make informed career choices [26]. Students have been shown to be 2.7 times more likely to pursue the same clinical specialty that they undertook a research project in while at medical school [5,28]. These factors emphasize the significance of the initial exposure to research that medical students experience, both in terms of their future clinical practice and scientific output. Throughout the collaborative process, levels of enjoyment and satisfaction with the level of guidance were consistently high. Additionally, students responded positively to the statement, “being involved in this research has made me more likely to do research in the future,” with a mean response of 8.57 (SD 1.50) out of 10. Although the students in this study were primarily only involved in title and abstract screening, a collaborative approach may be an enjoyable method of involving students in research.

Limitations

First, as this was a pilot evaluation of the collaborative approach to SR, this study was conducted by 1 research group and involved a small number of medical students (N=14) working on title and abstract screening for 2 SRs. This was a pragmatic decision given the uncertainty regarding the efficacy of the approach. Due to this small sample size, inferential statistics were not considered appropriate. Following this pilot, future studies should involve multiple research groups, with larger numbers of students, and student participation in a greater proportion of the SR process to better evaluate the collaborative approach.

Second, students were recruited from the student network of Myelopathy.org, as this was the most practical option for reaching out to medical students. This approach may have selected for students more interested in an academic career in neuroscience, which may have skewed perceptions toward research. Third, several students involved had previous degrees and research experience, potentially impacting scores of knowledge, confidence, and experience of SR methodology throughout the process. This student group is therefore unlikely to represent all medical students, and further exploration of this collaborative approach with subgroup analysis between those with and without prior research experience would be insightful.

This was ultimately an initial, small-scale exploration of whether real-world experience of a SR was advantageous to medical students. The findings of this study should therefore inform further optimization, including consideration of the aforementioned limitations, and subsequent formal evaluation.

Conclusions

Within the limitations of the study, this collaborative and educational approach to SR was well received by medical students, allowing them to gain insight into research methodology while contributing to publishable research. This potentially represents a useful technique for SSC projects; however, it requires further formal evaluation.