This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.
The progressive use of e-learning in postgraduate medical education calls for useful quality indicators. Many evaluation tools exist. However, these are diversely used and their empirical foundation is often lacking.
We aimed to identify an empirically founded set of quality indicators to set the bar for “good enough” e-learning.
We performed a Delphi procedure with a group of 13 international education experts and 10 experienced users of e-learning. The questionnaire started with 57 items. These items were the result of a previous literature review and focus group study performed with experts and users. Consensus was met when a rate of agreement of more than two-thirds was achieved.
In the first round, the participants accepted 37 items of the 57 as important, reached no consensus on 20, and added 15 new items. In the second round, we added the comments from the first round to the items on which there was no consensus and added the 15 new items. After this round, a total of 72 items were addressed and, of these, 37 items were accepted and 34 were rejected due to lack of consensus.
This study produced a list of 37 items that can form the basis of an evaluation tool to evaluate postgraduate medical e-learning. This is, to our knowledge, the first time that quality indicators for postgraduate medical e-learning have been defined and validated. The next step is to create and validate an e-learning evaluation tool from these items.
E-learning, which also goes by many other names, is taking up a strong position in medical curricula because of its flexibility, richness, and potential for resource sharing and for high value in light of its cost [
However, the debate on what denotes good-quality e-learning is ongoing. More explicitly, the lack of knowledge on what constitutes good-quality e-learning has been identified as one of the main inhibitors of its usefulness [
In response to this debate on what constitutes good-quality medical e-learning, we set out to provide an empirically based set of quality indicators. Thus, we performed a Delphi procedure to evaluate suggested quality indicators from the literature. To our knowledge, this study is the first international consensus by both educational experts and experienced users on quality indicators in postgraduate medical e-learning.
In this study, we performed a Delphi procedure to determine consensus on the possible quality indicators for e-learning in postgraduate medical education.
Escaron et al describe the Delphi method as being well suited to informing health education [
For this Delphi we chose the following, slightly adapted definition from Sangrà et al: “E-learning is an approach to teaching and learning, representing all or part of the educational model applied, that is based on the use of electronic media and devices as tools for improving access to training, communication and interaction and that facilitates the adoption of new ways of understanding and developing learning” [
For this study, we used 2 expert groups: medical educators and end users. Medical educators are experts in the theory and practice of creating e-learning and end users know what it’s like to use the e-learning in their daily practice. A suitable expert is defined in the literature as someone who possesses the relevant knowledge and experience and whose opinions are respected by fellow workers in their field [
We selected experts by means of an inquiry to the National Education Board in the Netherlands and from author contacts. We invited experienced users in the Netherlands and Great Britain because we had local contacts there. An expert panel usually consists of 15 to 30 participants, with 5 to 10 participants per category [
The initial set of indicators was based on 2 previous studies and contained quality characteristics from the literature [
The questionnaire started with introductory text explaining the subjects, providing a definition of e-learning, and asking the experts and users to imagine e-learning that was “just good enough” and targeted at medical postgraduates. After that, the experts and end users evaluated the individual items on a 5-point Likert scale and were able to add comments [
Subjects and themes of postgraduate medical e-learning quality indicators.aThe preparation theme is aimed at e-learning authors only.
After we agreed on the content of the questionnaire, we performed a pilot round with 5 participants (2 educators and 3 end users). After incorporating their feedback on the items, we invited the experts to fill out the questionnaire digitally. We started the first round with 57 items.
After each round, we worked out consensus by calculating the rate of agreement: (agreement – disagreement/agreement + disagreement + indifferent) × 100%. We used a rate of agreement of two-thirds to accept an item. An item was rejected when there was no consensus after 2 rounds, or when an item was rejected by a rate of agreement lower than –66% in the first round (the rate of agreement scale ranges from –100 to 100). There is no consensus in the literature regarding the best rate of agreement to be used; the range used has been between 51% and 80% [
The Ethical Review Board of the Association for Medical Education gave ethical consent (file number 475), after which all participants gave their written informed consent.
We sent the first invitation emails out on March 19, 2017, and received the final response on July 20, 2017. We invited 23 experts, of whom 13 replied and participated, 9 did not reply to the invitation, and 1 did not consider himself an expert on postgraduate medical e-learning. We invited 17 experienced users, of whom 5 did not reply, 2 could not participate due to other obligations, and 10 were able to participate. In total, we had 23 participants, of whom 23 responded in both rounds. Of the participants, 13 (57%) were male. The average age of the experts was 49 years and that of the users was 31 years. The experts came from the Netherlands (n=7), Great Britain (n=3), Canada (n=2), and South Africa (n=1). They had an average of at least 3 years’ experience creating or evaluating medical e-learning and together had published 29 articles. A total of 4 were members of the Dutch Association for Medical Education expert group on e-learning. The users were Dutch (n=7) and British (n=3), and had more than 3 years’ experience as residents, and had attended on average more than 2 e-learnings during their residency.
In the first round, 37 items were accepted as important, with a rate of agreement of above two-thirds. No items were rejected, there was no consensus on 20 items, and 15 new items were added by the participants (
The first explorative question was “Do you think it is possible to define a minimum and general set of criteria that can be generalized for all types of medical e-learning?” A total of 17 participants thought this was possible, 5 were not sure, and 1 thought it was too complicated. Worries about such a list of indicators included the following:
But I would be concerned that to be applicable for all types of medical e-learning it might be too general and therefore not practically useful
Yes, but it’s like evidence-based medicine: you must be able to deviate with motivation.
The experts also raise the concern of a fast-changing definition of e-learning:
the term e-learning is in a fast-changing technological world with different needs and skills for makers (and for users) and is difficult to define—without maker- or user-focused definition and context
e-learning doesn’t mean anything in particular, tech can be used in every aspect of med-ed, and lots of different tech can be used for different purposes….
Participants mentioned that which form of e-learning these indicators are about is very important to explain.
The second explorative question was “Do you think a 10-question survey, like the one mentioned in the introduction, would be of added value to the current evaluation tools?” It was thought by 14 (64%) to be of added value, 7 (32%) were not sure, and 1 (4%) thought it was not of added value. Arguments were
...it would help setting priorities
...general design principles probably will apply to e-learning as well. So why the need of a specific tool? I think there may be added value in evaluating the specific additive value of technology. But I am not sure. That’s why I am participating in this Delphi.
The third explorative question was to explore how many items participants considered to be workable. The general opinion was “the less the better, but as much as needed”. When asked for a number, participants responded with a range of numbers from 10 to 20.
We then evaluated the remaining 35 items (see
Flowchart of the Delphi results.
The final quality indicators. Items 32-37 are expert theme preparation items.
Subject and item | |
1. Create a feeling of importance within the learner | |
2. Create a feeling of responsibility within the learner | |
3. Provide enough time to complete the e-learning | |
4. Define the purpose of the e-learning (knowledge, skills, and behavior or attitude) | |
5. Formulate the learning objectives and preferably visualize them | |
6. Provide an overview of all content | |
7. Prevent concerns about the quality of the content | |
8. Do not force, although obligation might be possible | |
9. Create the feeling that the learner is being taken seriously | |
10. Use a flexible platform, so that the content can be modified by the educator | |
11. Provide easy accessibility from all locations and devices | |
12. Use easy and clear navigation | |
13. Use a simple layout with a sitemap | |
14. Software should be safe and secure | |
15. Access should be fast | |
16. Make clear which device is needed and advise the learner about the skills needed | |
17. Enable the learner to personalize the module | |
18. Allow nonlinear learning | |
19. Show what has already been achieved and what has not yet been done (progress bar) | |
20. Provide technical support | |
21. Add summaries | |
22. Give feedback | |
23. Add exercises and assignments | |
24. Create interaction with the content | |
25. Do not stress or frustrate the learner | |
26. Avoid nonadaptive content | |
27. Do not create too distractive a design or learning activities | |
28. Make the content translatable to the real world | |
29. Update and maintain the e-learning | |
30. Provide sources of information and keep access available after the course is finished | |
31. Evaluate the e-learning after the course and collect feedback | |
32. Know your target audience and adapt learning objectives accordingly | |
33. Identify the authors at the beginning of the e-learning | |
34. Create a timeline with objectives and expectations of the production stage | |
35. Form a development team with at least 1 content expert, 1 educational expert, and 1 information technology expert, and let them all commit a certain amount of time before starting the development | |
36. Plan a feasible budget to prevent incompletion of the e-learning due to lack of funds | |
37. Consider an appropriate learning environment and learning management system |
We performed an international Delphi study with educational experts and experienced users that led to 37 quality indicators for postgraduate medical education. To our knowledge, this is the first list of quality indicators for postgraduate medical e-learning with an evidence-based foundation: first selecting all the indicators mentioned in the literature, then adding to this list by focus group discussions, and finally selecting the items using a Delphi.
Cook et al wrote in 2009 that internet-based learning is associated with a positive effect, but that future research should directly compare different internet-based interventions [
After the first round of the Delphi, the experts expressed the challenges of an evaluation of this type. The term
Potential pitfalls in Delphi studies are the imposition of preconceptions on respondents and poor techniques for summarizing and presenting the group response. We tried to limit these pitfalls by producing a simple and straightforward questionnaire. Participant selection was limited to those who responded and, by choice, from the countries of the authors’ residence. Therefore, our study lacked a certain cultural diversity, making the results possibly less generalizable.
The strength of the final indicators lies in the balance of general aspects of evaluation and the specifics added when needed. We believe that the 6 themes (motivation, barriers, learning enhancers, learning discouragers, real-life translation, and poor preparation) are general enough to be applied to all kinds of e-learning.
Creating e-learning for postgraduates is not enough; evaluation and improvement should not be additional but mandatory to ensure maximum effect. E-learning quality indicators can be sorted into 3 groups (motivate, learn, and apply) with 5 general themes (motivators, barriers, learning enhancers, learning discouragers, and real-life translators) and a list of items that can be used in preparing e-learning resources.
This study provided a list of quality indicators for postgraduate medical e-learning. This list is unique in its evidence-based foundation and in the way that it applies broad themes with specific indicators. The most logical next step is to create and validate an evaluation tool based on these indicators.
Delphi results per question and per round.
We wish to express our greatest gratitude to the participants of the Delphi. Without their feedback, patience, and time, we could not have completed this study.
All authors contributed equally in the preparation of the study protocol. Experts were supplied mainly by RAD and KW, and users were mainly suggested by RAD, FS, and KW. RAD carried out Delphi facilitation, data collection, and analysis. All authors wrote and corrected the final manuscript.
None declared.