This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.
The transfer of research knowledge into clinical practice can be a continuous challenge for researchers. Information and communication technologies, such as websites and email, have emerged as popular tools for the dissemination of evidence to health professionals.
The objective of this systematic review was to identify research on health professionals’ perceived usability and practice behavior change of information and communication technologies for the dissemination of clinical practice guidelines.
We used a systematic approach to retrieve and extract data about relevant studies. We identified 2248 citations, of which 21 studies met criteria for inclusion; 20 studies were randomized controlled trials, and 1 was a controlled clinical trial. The following information and communication technologies were evaluated: websites (5 studies), computer software (3 studies), Web-based workshops (2 studies), computerized decision support systems (2 studies), electronic educational game (1 study), email (2 studies), and multifaceted interventions that consisted of at least one information and communication technology component (6 studies).
Website studies demonstrated significant improvements in perceived usefulness and perceived ease of use, but not for knowledge, reducing barriers, and intention to use clinical practice guidelines. Computer software studies demonstrated significant improvements in perceived usefulness, but not for knowledge and skills. Web-based workshop and email studies demonstrated significant improvements in knowledge, perceived usefulness, and skills. An electronic educational game intervention demonstrated a significant improvement from baseline in knowledge after 12 and 24 weeks. Computerized decision support system studies demonstrated variable findings for improvement in skills. Multifaceted interventions demonstrated significant improvements in beliefs about capabilities, perceived usefulness, and intention to use clinical practice guidelines, but variable findings for improvements in skills. Most multifaceted studies demonstrated significant improvements in knowledge.
The findings suggest that health professionals’ perceived usability and practice behavior change vary by type of information and communication technology. Heterogeneity and the paucity of properly conducted studies did not allow for a clear comparison between studies and a conclusion on the effectiveness of information and communication technologies as a knowledge translation strategy for the dissemination of clinical practice guidelines.
Success in regularly transferring research knowledge into clinical practice has been limited [
Knowledge translation (KT), the process of implementing knowledge into action, can provide methods for closing the knowledge-to-action gap [
Teaching modalities for medical education, including CPG dissemination, have evolved [
The objective of this systematic review was to summarize the evidence pertaining to the use of ICTs for the dissemination of CPGs to health professionals. Specifically, with this review we sought to provide new knowledge on health professionals’ perceived usability and change in practice behavior when using ICTs to disseminate CPGs.
We conducted this systematic review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [
We excluded studies if they did not meet the selection criteria (
Study selection criteria.
Criterion | Definition |
Population | Health professionals (eg, physicians including medical residents, nurses, and physiotherapists) |
Intervention | Information and communication technologies for disseminating clinical practice guidelines |
Comparator | Information and communication technologies compared with each other or control (eg, no intervention) |
Outcomes | Usability (eg, perceived usefulness and perceived ease of use) |
Practice behavior (eg, barriers, knowledge, skills, social/professional role and identity, optimism, beliefs about capabilities, beliefs about consequences, intentions, memory/attention/decision, environmental context and resources, social influences, and emotion) | |
Study design | Randomized controlled trials |
Nonrandomized comparative controlled trials |
The literature search was performed by an information specialist. Published literature was identified by searching the following bibliographic databases up to the end of December 2015: MEDLINE, Cochrane Central Register of Controlled Trials, EMBASE, CINAHL, ERIC, and PsycINFO. The search was performed using terms to identify peer reviewed research in which ICTs and CPG dissemination were important features (
Titles and abstracts of all citations retrieved from the literature search were independently screened by 2 reviewers using Covidence (Veritas Health Innovation Ltd), a Web-based systematic review software. Full-text articles were then independently reviewed based on the selection criteria. Disagreements were resolved through discussion until consensus was reached.
Both descriptive data and results were extracted by 1 reviewer from each eligible article. The extraction was subsequently verified by a second reviewer. Data extraction forms were designed a priori to document and tabulate relevant study and patient characteristics, study findings, and authors’ conclusions. We did not use data from figures if the data were not explicit. Studies were categorized by the type of ICT intervention used.
One reviewer independently assessed the quality of each study using the Cochrane risk of bias tool [
Given the broad inclusion criteria and heterogeneity of the interventions and methodological characteristics of included studies (guided by PICOS), we deemed a meta-analysis to be inappropriate, and we therefore conducted a narrative synthesis and summary of study findings. The outcomes of interest were the usability of the ICT intervention and practice behavior change.
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of included studies.
The usability outcomes were guided by the technology acceptance model (TAM2) [
The theoretical domains framework (TDF) guided the practice behavior change outcomes [
We identified a total of 2248 citations through the initial search. After removing duplicates, we screened 2122 publication abstracts and titles. We assessed the full texts of 61 articles; of these, we excluded 40 for the following reasons: irrelevant population (8 studies), duplicate report (1 study), irrelevant intervention (19 studies), study protocol (2 studies), irrelevant outcome (6 studies), inappropriate study design (2 studies), and presented as abstract only (2 studies). The excluded studies are listed in
Of the 21 studies that we included in our systematic review, 20 were randomized controlled trials (95%) and 1 was a controlled clinical trial (5%) [
Type of information and communication technology (ICT) used in each included study.
ICT intervention | Number of studies | Studies |
Website | 5 | Balamuth et al [ |
Computer software | 3 | Bullard et al [ |
Web-based workshops | 2 | Epstein et al [ |
Computerized decision support system | 2 | Gill et al [ |
Electronic educational game | 1 | Kerfoot et al [ |
2 | Lobach [ |
|
Multifaceteda | 6 | Bernhardsson et al [ |
aMultifaceted intervention that consisted of at least one ICT component.
In 8 studies, there was no comparison with an intervention [
The use of a website for the dissemination of CPGs to health professionals was assessed in 5 studies [
Summary of findings of included studies by primary information and communication technology (ICT) intervention.
ICT |
Study | Interventions | Outcome(s) | Effect size | Conclusion |
Balamuth, 2010 [ |
Web-based 1-page summary sheet of guidelines (n=128) |
0.82 (0.49-1.4) | No statically significant difference between 2 groups in correctly diagnosing patients according to guidelines. Participants using the Web-based 1-page summary reported that the supplemental materials were more simple to use when compared with the weblink group. | ||
6.1 (2.8-13.6) | |||||
Bell, 2000 [ |
Self-study Web-based guidelines (n=79) |
Web-based: 15.0 (14.0-15.0) |
No statistically significant difference in knowledge at immediate posttest or after 4-6 months. Web-based guideline users were more satisfied with learning. | ||
Web-based: 12.0 (11.0-13.0) |
|||||
Web-based: 17.0 (16.0-18.0) |
|||||
Schroter, 2011 [ |
Website with educational modules (n=48) |
Web-based plus Web material: 47.4% (12.6) to 66.8% (11.5) |
No statistically significant differences in knowledge change or usability between the 2 groups. Participants in Web-based tool plus Web material group found it to be useful. Usefulness was not measured in the other group. | ||
Web-based plus Web material: 77% |
|||||
Sassen, 2014 [ |
Website with educational modules (n=48) |
Website: 6.25 (1.00), 6.06 (1.11) |
No statistically significant differences in intention to use and barriers between interventions groups at 12 months. | ||
Website: 3.11 (1.17), 3.18 (1.12) |
|||||
Wolpin, 2011 [ |
Website enhanced learning (additional case studies) (n=33) |
Overall (pooled both groups): 79.28% (12.17), 82.32% (13.84), |
No statistically significant difference in knowledge or satisfaction at posttest between intervention groups. No statistically significant differences were seen between interventions groups for both outcomes. | ||
Overall (pooled both groups): 4.08 (0.860) |
|||||
Bullard, 2004 [ |
Wirelessly networked mobile computer program (n=10)d |
Wireless: 3.2 (2.6-3.8) |
Statistically significant greater satisfaction for several items (“impact on efficiency,” “increase use of CPGs,” and “saving time”) when using the wireless computer compared with the desktop computer. Other satisfaction items such as “configuration,” “availability,” “reduced communication with staff and patients,” and “accessibility” did not show statistically significant differences (results not shown). Participants appeared to be indifferent regarding the usability of the wireless computer for their efficiency. | ||
Wireless: 4.1 (3.6-4.6) |
|||||
Wireless: 3.30 (2.33-4.27) |
|||||
Butzlaff, 2004 [ |
CPGs via CD-ROM/Internet (n=53) |
CD/Internet: 13 (12-16) |
There was no statistically significant difference between intervention groups at baseline and ~70 postintervention in knowledge scores. | ||
CD/Internet: 15 (12-17) |
|||||
Jousimaa, 2002 [ |
CD-ROM computer-based guidelines (n=72) |
1.07 (0.79-1.44) | There was no statistically significant difference between intervention groups for compliance with CPGs for laboratory, radiological, or physical examinations. | ||
1.09 (0.81-1.46) | |||||
0.74 (0.51-1.06) | |||||
Epstein, 2011 [ |
Web-based didactic education session/workshop (n=27) |
Web: 23.8% |
Statistically significant changes from baseline to 6 months were seen among participants complying with CPG-recommended ADHD care practices, with the exception of 1 recommendation, “Use of parent ratings of ADHD to monitor treatment responses” (results not shown). | ||
Web: 22.6% |
|||||
Web: 47.3% |
|||||
Web: –60.7% |
|||||
Web: 38.7% |
|||||
Fordis, 2005 [ |
Live Web-based CMEg workshop (n=51) |
31.0% (95% CI 27.0%-35.0%), |
A statistically significant improvement in knowledge was seen over time for both Web-based interventions groups. A statistically significant decrease in appropriately screening patients was seen in the live Web-based CME group at 12 weeks posttest compared with baseline. No statistically significant differences were seen for screening patients between interventions groups. There was a statistically significant increase in the proportion of patients appropriately treated by the Web-based CME group compared with the live CME and control groups. Participants in the Web-based interventions were satisfied with the learning experience. | ||
36.4% (95% CI 32.2%-40.6%), |
|||||
5.4% (95% CI 2.6%-8.2%) | |||||
Live Web-based: −3.3 (−5.9 to −0.7) |
|||||
Live Web-based: −1.1 (−4.9 to 2.7) |
|||||
Live Web-based: 100% (49/49) |
|||||
Gill, 2011 [ |
EHRh-based clinical decision support (n=53) |
EHR: 25.4% |
There was a statistically significant difference favoring the EHR intervention compared with no intervention for the proportion of patients receiving guideline-concordant care. | ||
Peremans, 2010 [ |
EHR-based clinical decision support (n=15) |
EHR: –1.79 (–4.97 to 1.65) |
The empowered patient group was the only group that had improved consultation and prescribing skills scores after 5 months postintervention and the only intervention that demonstrated a statistically significant difference compared with no intervention. | ||
Kerfoot, 2009 [ |
Electronic game/survey 2 questions every 2 days (n=735) |
Electronic game 2 questions every 2 days: 48% (18) |
Both electronic game cohorts demonstrated statistically significant improvements in knowledge compared with baseline. | ||
Electronic game 2 questions every 2 days: 100% (3) |
|||||
Lobach, 1996 [ |
Biweekly emails of computer-based audit/feedback program (n=22) |
Email: 35.3% (NRi) |
The email intervention demonstrated statistical significance in greater compliance with guidelines compared with no intervention. | ||
Stewart, 2005 [ |
Email Web-based learning for 2 evidence-based modules (type 2 diabetes, prevention) (n=27) |
Email (diabetes): 66.8 (14.1) |
The intervention group (prevention module) demonstrated statistically significant improvements compared with the control group for knowledge at 2 and 6 months, as well as compliance at 6 months. There was no statistically significant difference with the diabetes modules. | ||
Email (diabetes): 72.7 (14.1) |
|||||
Email (diabetes): 73.2 (7.7) |
|||||
Email (diabetes): 53.8 (12.5) |
|||||
Email (diabetes): 51.7 (12.9) |
|||||
Email (diabetes): 47.1 (9.2) |
|||||
Bernhardsson, 2014 [ |
Multifaceted: implementation seminar/group discussion, website, and email reminders (n=168) |
Intervention: 27.9% |
There was a statistically significant difference favoring the intervention group for change in awareness, knowledge of where to find guidelines, and accessibility of guidelines at 1-year follow-up. There were no significant differences in frequent use of CPGs. | ||
Intervention: 25.2% |
|||||
Intervention: 17.4% |
|||||
Intervention: 9.2% |
|||||
Chan, 2013 [ |
Multifaceted: in-person education session and Web-based support (n=31) |
Intervention: 25.9% (4.2 to 45.5) |
There were statistically significant improvements in self-confidence to use, satisfaction in following, and willingness to follow CPGs among the intervention group at 2 weeks postintervention. There were no significant improvements among the control group. | ||
Intervention: 40.7% (16.1-59.6) |
|||||
Intervention: 0.74 (0.36-1.1) |
|||||
Desimone, 2012 [ |
Multifaceted: in-person education, Web-based support, printed materials (n=11) |
Multifaceted: 69% (1.7) |
There was a statistically significant improvement in knowledge in both groups at 1 month postintervention. There were no observable differences between groups (between-group statistical analyses not performed). | ||
Multifaceted: 83% (2.1), |
|||||
McDonald, 2005 [ |
Multifaceted: email reminder with provider prompts, patient education material, and clinical nurse specialist outreach (n=97) |
Email reminder: –5.7, |
In the email reminder intervention group, there was a decrease in performance, as the probability of nurses completing bowel movement assessments was statistically significantly lower compared with usual care. There was no statistically significant difference compared with the multifaceted group. Other nurse assessment and instruction practices did not reach statistical significance when the email reminder and multifaceted interventions were compared with usual care (results not shown). | ||
Fretheim, 2006 [ |
Multifaceted: educational outreach visit, audit and feedback at outreach visit, computerized reminders, risk assessment tools, patient information material, telephone follow-up (n=257) |
Multifaceted: 11.5% |
There was a statistically significant difference in participants prescribing in concordance to CPGs from baseline to 12 months favoring the multifaceted group compared with passive guidelines dissemination. No statistically significant differences were demonstrated for differences in participants performing risk assessments at 12 months. | ||
1.04 (0.60-1.71) | |||||
Shenoy, 2013 [ |
Multifaceted: Web-based education, audit, feedback (n=24) |
0.04 (1.22-1.31) | There was no statistically significant change in knowledge between intervention groups from baseline to 12 weeks postintervention. There was no statistically significant difference between intervention groups for the proportion of patients receiving CPG-adherent care at 12 weeks postintervention (results not shown). |
aOR: odds ratio.
bNR: not reported.
cCPG: clinical practice guideline.
dCrossover design with same participants in both groups.
eIQR: interquartile range (25th to 75th percentile).
fADHD: attention-deficit/hyperactivity disorder.
gCME: continuing medical education.
hEHR: electronic health record.
iIQR values illustrated in a diagram; however, values are not explicit.
jRR: relative risk.
Perceived usefulness was assessed in 1 study [
Perceived ease of use was assessed in 3 studies [
Knowledge was assessed in 4 studies [
Intention to use CPGs and reduction in barriers were assessed in 1 study [
The use of computer software for the dissemination of CPGs among health professionals was assessed in 3 studies [
Perceived usefulness was assessed in 1 study [
Knowledge was assessed in 1 study [
Skills were assessed in 1 study [
The use of Web-based workshops for the dissemination of CPGs among health professionals was assessed in 2 studies [
Perceived usefulness was assessed in 1 study [
Skills were assessed in both studies [
Knowledge was assessed in 1 study [
The use of CDSSs for the dissemination of CPGs among health professionals was assessed in 2 studies [
Usability was not assessed in any of the included studies that used CDSSs for the dissemination of CPGs.
Skills were assessed in both studies [
The use of an electronic educational game for the dissemination of CPGs among health professionals was assessed in 1 study [
Usability was not assessed in Kerfoot et al [
Both game groups demonstrated statistically significant (
The use of email for the dissemination of CPGs among health professionals was assessed in 2 studies [
Usability was not assessed in any of the included studies that used email for the dissemination of CPGs.
Skills were assessed in both studies [
Knowledge was assessed in 1 study [
The use of a multifaceted intervention including an ICT with more than one CPG dissemination strategy among health professionals was assessed in 6 studies [
Usability was assessed in 1 study [
Perceived usefulness was assessed in 1 study [
Knowledge was assessed in 3 studies [
Skills were assessed in 3 studies [
Beliefs about capabilities and intention to use CPGs were assessed in 1 study [
The aim of this review was to identify research on health professionals’ perceived usability and practice behavior with ICTs for the dissemination of CPGs. In summary, results varied by the type of ICT used. While rapidly changing technologies may pose challenges for the development, implementation, and evaluation of ICT-based interventions, as they may be associated with greater barriers for adoption by health professionals [
Outcome selection was guided by both the TAM2 [
The variable findings in knowledge improvement are supported by a recent systematic review [
The strengths of this systematic review include the broad eligibility criteria that we used, allowing for numerous types of ICTs and various health professional populations (ie, physicians including medical residents, nurses, and physiotherapists) to be included and summarized in this review. Additionally, we used a systematic approach to review the literature and assessed the methodological quality of each included study. This systematic review was conducted following the PRISMA checklist [
Nevertheless, there are limitations of this review that should be considered. We did not include information published in languages other than English; thus, we may have excluded some relevant findings. The small number of included studies per ICT and the heterogeneity between studies in regard to the included health professional populations, definitions of outcomes assessed, selected comparators (some compared interventions against no intervention, while others used active comparators), and duration of studies did not allow for comparisons between studies. As a result, we were not able to calculate pooled effect sizes or perform meta-analyses. The terminology of outcomes in the included studies sometimes differed from the identified concepts in the TAM2 and domains of the TDF that we used to define the usability and practice behavior change outcomes, respectively. Several studies measured numerous outcomes, and it remains uncertain whether these studies were adequately powered to detect meaningful differences. Furthermore, the overall findings were limited by the high loss to follow-up in numerous studies [
The authors of the included studies did not always assess the quality of information being presented or quality of ICT. The quality of information being presented was previously assessed and deemed appropriate by authors in 4 of 5 (80%) studies using websites [
The overall methodological quality of included studies was strong for the website studies, while it was uncertain for the electronic education game, email, and multifaceted studies (
The findings of this systematic review suggest that health professionals’ perceived usability and practice behavior change vary by type of ICT. Website studies demonstrated improvements in perceived usefulness and perceived ease of use, but not for knowledge usability, barriers, and intentions. Computer software studies demonstrated improvements in perceived usefulness, but not in knowledge and skills. Web-based workshop and email studies demonstrated improvements in knowledge, perceived usefulness, and skills. An electronic educational game intervention demonstrated an improvement in knowledge from baseline to 12 or 24 weeks. CDSS studies demonstrated variable findings for improvement in skills. Multifaceted ICT interventions demonstrated improvements in beliefs about capabilities, but not in usability. Most multifaceted ICT studies demonstrated improvements in knowledge, perceived usefulness, perceived ease of use, and beliefs about capabilities. In summary, heterogeneity and the paucity of properly conducted studies did not allow for a clear comparison between studies and a conclusion on the effectiveness of ICTs as a KT strategy for the dissemination of CPGs.
Search strategy.
List of excluded studies.
Included study characteristics.
Methodological assessment of included studies.
attention-deficit/hyperactivity disorder
computerized decision support system
continuing medical education
clinical practice guideline
electronic health record
information and communication technologies
interquartile range
knowledge translation
population, intervention, comparison, outcome, and study design
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
odds ratio
technology acceptance model
theoretical domains framework
GD, LB, GAW, BD, JK, and JM conceived of and designed the systematic review. GD, SC, BD, JK, GAW, and LB analyzed the data and contributed to the methods and review. GD, SC, and LL performed the systematic review. GD wrote the paper. The authors thank Ms April Ripley for her contributions with screening and reviewing the evidence in this systematic review and are thankful for the financial support of the Canadian Institutes of Health Research Doctoral Research Award and University of Ottawa Research Chair.
None declared.