The SDM training program has changed twice over 3 years. It was introduced in the family medicine residency program in 2011 as a multicomponent program: a Web-based tutorial entitled “Shared decision making to treat ARI,” a 2-hour workshop in the form of a classroom course, and a decision aid. However, the 2-hour workshop was withdrawn after the 2012-2013 residency period, leaving only the Web-based tutorial and the decision aid. The rationale for this removal was the time constraints in the residents’ schedule. Before July 2014, the Web-based tutorial contained 5 modules including information about diagnosis, treatment, and key components of the SDM process in the treatment of ARIs in primary care. After the 2013-2014 residency period, a sixth module on integrating the knowledge acquired in the first 5 modules was added. This addition also integrated parts of the workshop that had been removed 1 year earlier. Figure 1 shows a timeline of the major modifications made over the 3 residency periods.

The SDM tutorial was offered to all second-year residents in the entire network of 12 FPTUs of the Department of Family Medicine and Emergency Medicine at Université Laval, Quebec, Canada. This tutorial is one of the several tutorials available via the intranet on the department’s Web platform. At the beginning of their second year of residency, residents were given the link and invited to complete the tutorial and the pre-posttests over the course of the year. On the Department home page, residents entered their student identification number and password to access the tutorial. Then, they encountered a brief description of the tutorial before starting it (Textbox 1) [33]. The tutorial included videos, exercises, a link to access the decision aid, and a pre-post knowledge test. This tutorial was designed to be completed in 2-3 hours. It was required as part of the family medicine curriculum but not specifically evaluated. However, at the end of their residency, residents were examined on all the subjects they learned via the intranet.

We included all second-year family medicine residents in the Department of Family Medicine and Emergency Medicine at Université Laval from July 2012 to July 2015 who logged in on the department’s Web platform to register for the tutorial, whether they completed it or not. Residency begins on July 15 of each year.

To describe the participation in and usage of the DECISION+2 Web-based tutorial used to teach SDM at Université Laval and its effect on knowledge scores, we collected all available data about (1) its content and scoring system; (2) its history and the incentives offered for undertaking it; (3) residents’ participation in and usage of the tutorial; and (4) changes in their pre-post test scores. In observing the history of the SDM program, we noted any changes made to it and the reasons for change. Data for tutorial registrations between July 2012 and July 2015 were extracted from department’s Web platform. Data included identification number of each registrant, date of the first and last connection, frequency of connection, total time spent on the tutorial, and answers to each of the pretest and posttest questions. With the connection frequencies we were able to determine how many registered residents started the tutorial. Residents who logged in were separated into 3 entry periods (2012-2013, 2013-2014, and 2014-2015) depending on the day of their first connection to the tutorial. For example, a resident who logged in to the tutorial on March 23, 2013, was classified in the 2012-2013 cohort because he or she began between July 15, 2012, and July 14, 2013. We considered the tutorial as completed if the residents answered all the pretest and posttest questions.

The knowledge test used for the pretest and the posttest contained 10 multiple-choice questions and was only available in French (Université Laval is a French-language university). It was based on information in the tutorial and contained key elements considered by the authors as essential knowledge for the practice of SDM regarding the use of antibiotics for treating ARIs. Some questions were multiple response while others were single response, and participants did not know which kind they were answering. Four questions were concerned with diagnosis (all single response), 3 were concerned with treatment (2 single response and 1 multiple response), and 3 were concerned with SDM (1 single response and 2 multiple response; Multimedia Appendix 1). The knowledge scores were displayed as follows: for single response items, 1 point for a correct answer and 0 points for any incorrect answer; for multiple response items, 1 point if all answers were correct and 0 if any answer was missing or incorrect. The maximum score was 10 points. Residents could only see their scores at the end of the posttest questions.

We performed simple descriptive statistics including frequencies, median, and interquartile range (minimum and maximum) to summarize characteristics and modalities of use among all family medicine residents who logged in to the Web-based tutorial, and to understand how residents used the tutorial. Also, we estimated proportions of family medicine residents, per period, who logged in to the tutorial, did the pretest only, did the posttest only, or did both. The Cochran-Armitage test for trend was performed to test the change in proportions of family medicine residents who logged in over time. Because the knowledge score was an ordinal variable and did not respect the normality assumption, we used nonparametric tests. To describe the change in the level of knowledge among those who completed the tutorial, we used Wilcoxon signed-rank test to identify significant pre- or posttest differences between the total scores and scores on each of the 3 knowledge categories (diagnosis, treatment, SDM). This kind of test is used especially for paired samples. We used McNemar test to measure if residents answered the questions correctly after doing the tutorial. Finally, the distribution of knowledge total scores and category scores between periods were compared using Mann-Whitney U test. We considered a difference statistically significant when the P value was <.05. We performed statistical analysis using the SAS version 9.4 (SAS Institute Inc).

As the study was supported by the institution where it was performed, no ethical approval was requested because the Web-based tutorial was part of an academic program and data were provided anonymously.

All 387 second-year family medicine residents were eligible to take the Web-based tutorial. Out of the 387 residents, 247 (63.8%) logged in to the Web-based tutorial. Among the 247 who logged in, 109 (44.1%) completed both the pre- and posttest, 95 (38.5%) completed the pretest only, 2 (0.8%) completed the posttest only, and 41 (16.6%) logged in but did not complete either test. In total, only 28.2% (109/387) of all eligible family medicine residents completed the tutorial (Figure 2).

Table 1 shows simple descriptive statistics of participants for the different entry periods. Proportions of women and the median number of connections were similar between cohorts. However, a relative increase in the median of time spent in the tutorial was observed between the 2013-2014 and 2014-2015 cohorts. Also, the proportion of registered residents who logged in to the tutorial per entry period was 53.7% (65/121) in 2012-2013, 67.7% (90/133) in 2013-2014, and 69.2% (92/133) in 2014-2015 (Cochran-Armitage test; P=.006). Participation rates increased between each succeeding entry period and seemed more pronounced between the 2012-2013 and 2013-2014 cohorts.

Twelve out of 109 residents who completed both tests had a posttest score equal to or lower than their pretest score, that is, the percentage of participants whose total knowledge score did not improve was 11%. Table 2 shows the medians and the interquartile ranges in the total knowledge scores and in each category among those who completed the tutorial. The median total knowledge score and each of the 3 category scores improved significantly between the pre- and posttest (Wilcoxon signed-rank test; all P values <.001). Table 3 shows the frequencies of those who answered each question correctly pre- and posttest. More participants answered the posttest questions correctly (McNemar test; all P values <.01) except for questions 7, 8, and 9 (the SDM category). In addition, low frequencies of those who answered correctly were observed for both pre- and posttest on questions 8 and 9.

Table 4 shows the medians and the interquartile ranges, and the P values of the 2-sided Wilcoxon rank-sum test performed to verify whether the distribution of total scores and the category scores pre- and posttest were the same between entry periods. The 2012-2013 period was not assessed because only 3 family medicine residents completed the tutorial during that period. All P values were greater than 5%, that is, there was no significant difference in the distribution of the total score or in any of the category scores for the pre- and posttests between the 2013-2014 and 2014-2015 cohorts.

Overall, we observed that the data available regarding residents’ participation in, use of, and effects of the Web-based tutorial on knowledge scores were limited. For example, sex was the only demographic data available, and data on time spent on the tutorial by residents per connection period, which pages they visited per connection period, and their participation in the classroom workshop were not available.

This study described the residents’ use of a Web-based training platform over time and attempted to assess whether the residents’ knowledge scores about the diagnosis and treatment of ARIs and SDM regarding the use of antibiotics for ARIs were improved by this Web-based training. It also provided an opportunity to identify what kinds of data are appropriate for evaluating the usage of the training platform and its impact on knowledge. The main results were that residents’ use of the Web-based tutorial increased over time, but not their knowledge scores; residents appeared to perform better on knowledge scores about diagnosing ARIs and treatment options than on SDM; just over a quarter completed the tutorial and one-third did not even start it; and little data appropriate for evaluating the course’s effectiveness were available. Our results led us to make 4 main observations.

First, we reported an increase in the participation rate between the 2012-2013 and 2013-2014 cohorts. This improvement could be explained by the removal of the workshop component of the course because 2012-2013 was the last year in which it was offered in addition to the Web-based tutorial. Perhaps family medicine residents saw the Web-based tutorial as an unnecessary addition to the classroom workshop on SDM in their curriculum and saw it as too much time spent on the topic. Some studies suggest that such Web-based tutorials should be brief and not too complex or intensive for medical students [24,37]. Furthermore, the average increase of about 1 hour in the median of the time spent in the tutorial observed between the 2013-2014 and 2014-2015 cohorts could have been caused by the addition of the sixth module. However, the added module was supposed to integrate all the knowledge acquired in the first 5 modules and compensate for the removal of the classroom workshop. We expected to see an increase in the distribution of the knowledge scores (total and per category) between these 2 periods, but this was not the case. Perhaps the inclusion of an additional review module was not relevant and may even have unnecessarily extended the duration of the training. More data would be needed to confirm this.

Second, our results suggested that the Web-based tutorial had a significantly positive effect on knowledge scores about the diagnosis and treatment of ARIs. However, the questions that the most residents failed were in the SDM category. This could be because the questions were poorly written, or it could be due to the nature of SDM. Unlike diagnosis or treatment, SDM is a subject that is inherently antithetical to unidirectional learning—it is about person-to-person communication and sharing information [17,18]. Perhaps Web-based tutorials performed in solitude are not an appropriate platform for teaching some of the essentials components of SDM to family medicine residents, namely presenting options, communicating risks and benefits, and clarifying values of patients. Moreover, knowledge scores may be an inadequate form of evaluation for SDM. However, in a 2013 randomized controlled trial on physician communication regarding prostate cancer screening, Feng et al assessed a 30-minute Web-based module and found, at 3-month follow-up, that the family physicians who used the Web-based tutorial had more shared decision-making behaviors and were more likely to encourage patients to consider different screening options compared with usual education [38]. Together, these results suggested that Web-based learning about SDM needs to be reassessed in further studies. They might also reflect the significant heterogeneity among SDM training programs [39], not only in their content but in their modes of delivery and evaluation of knowledge acquired. In this era of rapidly growing numbers of SDM training programs [40] and national efforts to offer them on the Web platform, the most effective methods of delivery and evaluation urgently need to be standardized.

Third, a difficulty we encountered was inadequacy of data organization or availability at the university level, which made it hard to evaluate the usage of the SDM training and its effects on knowledge. Indeed, the data available were not adequate for evaluating associations between participation in the tutorial and its impact on the residents’ knowledge scores. Moreover, no data were available that could inform us about whether the loss of the workshop component of the SDM program caused the increase in residents’ use of the tutorial. In addition, although all participants were second-year family medicine residents, we were unable to collect any demographic data on participants except for gender. If we had been able to analyze data extracted from log-in dates, visited pages, and time passed on each page per connection, our interpretation would have been more meaningful. To improve a Web-based training such as this, more information is needed about how the residents use the tutorial, and data collection needs to be adapted to reflect modifications in the program when they take place [41]. With the increase in Web-based interventions, the potential for data extraction is growing exponentially [42]. Moreover, sophisticated data analysis methods already exist that take account of the structure of more complex data such as this [43]. Our findings highlight the need to strengthen partnerships with residency programs so that data are made available in an appropriate form to be useful for evaluation purposes, both by faculties and by researchers.

The limitations of this study included contamination by residents who spent more than 1 year completing their second-year residency, and potential confounding variables. Lack of available demographic data compounded these problems. In terms of the tutorial’s effectiveness, participants were not classified by FPTU. Therefore, we cannot be sure that belonging to the same FPTU did not influence their answers. Also, participants might have logged in to the tutorial and then, rather than doing the tutorial, left the connection open and done something else for several hours (offline) before disconnecting. This might have distorted the time shown as spent on the tutorial. Finally, the psychometric properties of the pre- and post-knowledge tests had not been validated, and therefore scores might not have been valid, consistent, or reliable.

Residents’ use of this Web-based tutorial appeared to increase between entry periods following the changes to the SDM program, and the tutorial seemed less effective in the SDM categories than in the diagnosis and treatment categories. However, to evaluate the use of a Web-based tutorial properly and its impact on knowledge, data collection needs to include the different log-in dates, visited pages, time passed on each page per connection, and more complete sociodemographic characteristics. There is still work to be done to improve data sharing, quality, and availability for evaluation purposes, so that implementation of SDM in the context of antibiotics use for treating ARIs becomes a feature of everyday family practice.