This paper is in the following e-collection/theme issue:

Continuing Medical Education (CME) for Doctors (125) e-Learning and Digital Medical Education (705) Training Tools for Medical Students (132)

Published on in Vol 3, No 1 (2017): Jan-Jun

Development and Assessment of an E-learning Course on Pediatric Cardiology Basics

Development and Assessment of an E-learning Course on Pediatric Cardiology Basics

Development and Assessment of an E-learning Course on Pediatric Cardiology Basics

Authors of this article:

Ana Cristina Oliveira1 Author Orcid Image ;   Sandra Mattos2 Author Orcid Image ;   Miguel Coimbra3 Author Orcid Image
Article Authors Cited by (11) Tweetations (6) Metrics

    Original Paper

    1Faculdade de Ciências da Universidade do Porto and Faculdade de Medicina da Universidade do Porto, Porto, Portugal

    2Círculo do Coração, Unidade de Cardiologia e Medicina Fetal, Real Hospital Português de Pernambuco, Recife, Brazil

    3Instituto de Telecomunicações, Faculty of Sciences of the University of Porto, Porto, Portugal

    *all authors contributed equally

    Corresponding Author:

    Ana Cristina Oliveira, MS

    Faculdade de Ciências da Universidade do Porto and Faculdade de Medicina da Universidade do Porto

    Rua Dr Plácido da Costa, s/n

    Porto, 4200 - 450

    Portugal

    Phone: 351 225513622 ext 5743

    Fax:351 225513623

    Email: crisoliveira@med.up.pt


    Background: Early detection of congenital heart disease is a worldwide problem. This is more critical in developing countries, where shortage of professional specialists and structural health care problems are a constant. E-learning has the potential to improve capacity, by overcoming distance barriers and by its ability to adapt to the reduced time of health professionals.

    Objective: The study aimed to develop an e-learning pediatric cardiology basics course and evaluate its pedagogical impact and user satisfaction.

    Methods: The sample consisted of 62 health professionals, including doctors, nurses, and medical students, from 20 hospitals linked via a telemedicine network in Northeast Brazil. The course was developed using Moodle (Modular Object Oriented Dynamic Learning Environment; Moodle Pty Ltd, Perth, Australia) and contents adapted from a book on this topic. Pedagogical impact evaluation used a pre and posttest approach. User satisfaction was evaluated using Wang’s questionnaire.

    Results: Pedagogical impact results revealed differences in knowledge assessment before and after the course (Z=−4.788; P<.001). Questionnaire results indicated high satisfaction values (Mean=87%; SD=12%; minimum=67%; maximum=100%). Course adherence was high (79%); however, the withdrawal exhibited a value of 39%, with the highest rate in the early chapters. Knowledge gain revealed significant differences according to the profession (X22=8.6; P=.01) and specialty (X22=8.4; P=.04). Time dedication to the course was significantly different between specialties (X22=8.2; P=.04).

    Conclusions: The main contributions of this study are the creation of an asynchronous e-learning course on Moodle and the evaluation of its impact, confirming that e-learning is a viable tool to improve training in neonatal congenital heart diseases.

    JMIR Med Educ 2017;3(1):e10

    doi:10.2196/mededu.5434

    Keywords

    distance learning; continuing medical education; pediatrics; cardiology; congenital heart defects 


    Background

    Managing congenital heart diseases is a worldwide problem [1]. Early detection through newborn screening can potentially improve the outcome of these diseases [2]. In newborns, congenital heart diseases can be detected by auscultation, pulse oximetry, radiography, catheterization, although transthoracic echocardiogram, a specialized form of ultrasound, is the elected exam for diagnosis [3,4]. Pediatric cardiologists usually perform this examination, but in developing countries there is shortage of professional specialists, which are often concentrated in larger urban centers, hindering the widespread population screening and causing a need for constant transferal of patients from the isolated regions to reference health centers [5].

    Context

    In response to these challenges, the Health Secretary of Paraíba, in Brazil established a pediatric cardiology network [6]—Rede de Cardiologia Pediátrica (RCP)—in partnership with Círculo do Coração , a civil nonprofit organization from Recife, in order to create a neonatal screening program for the whole state of Paraíba and a hospital facility designed to manage patients. Of the 20 maternity centers covered by this network, only 7 had neonatal ICU beds. The remaining are level-1 district centers. In only one center, in the capital city of Joao Pessoa, there is a pediatric cardiologist available for echo and clinical diagnosis. No center performs cardiac surgery. This network provides perinatal and neonatal care in remote areas supervised by telemedicine; it was created due to the need to train local physicians and involve local professionals on screening, diagnosis, therapeutic treatment and management of congenital heart diseases in fetuses, newborns, and children of the public health system.

    Echocardiography is used to diagnose congenital heart disease when either there is an abnormal clinical examination or an abnormal pulse oximetry. Abnormal pulse oximetry results are automatically noted on a database of the network, allowing the network to contact the clinic and request that they follow up any babies with abnormal test results. These active search protocols track the discharged neonates and ensure that abnormal findings are acted on.

    Echo image acquisition and its interpretation is a combined process developed within this Web-based telemedicine network where trainees are constantly supervised by cardiologists. Not only the whole process is documented in the Web database system to track the apprentice's development, but also whenever needed, live interaction is called to adjust practical aspects. Initially, all echocardiograms needed to be performed with Web-based supervision by the pediatric cardiologist, as part of the neonatologists’ training, but with time cardiologists would only be requested for direct Web-based supervision when pathological findings were suspected. As there are always new neonatologists being trained, this learning process and interaction between teams is a continuous cycle [7].

    E-Learning for Health Care

    With the increasing use of Internet information and communication technologies, e-learning has emerged as a widely accepted modality in medical education [8]. For its convenience and its potential for cost savings, it has become popular among the medical education community [9]. E-learning is known to offer learning opportunities where there is limited access to teaching in a specific field, either because of a lack of qualified or geographically distant teaching institution [10]. Therefore, e-learning can be a powerful tool to increase the capacity of health professionals in constricted contexts for neonatal echocardiography congenital heart disease screening. There is evidence in the literature that e-learning is a useful tool for overcoming barriers to health professionals training [10]. A review carried out by Frehywot et al on e-learning in medical education in resource-constrained LMICs suggests that e-learning may be effective for increasing capacity in rural settings, although evidence is still limited.

    The convenience, the self-paced and learner-centered learning, and the creation of a global learning community, are some significant benefits of e-learning that have been discussed in many articles [8,10-24].

    In medical contexts, e-learning program results such as efficiency and costeffectiveness have typically gone unreported [25]. In nonmedical contexts, there is evidence that e-learning can result in cost savings of up to 50% over traditional learning programs, due to reduced instructor training time, travel and labor costs, institutional infrastructure, and the possibility of expanding programs with new technologies [8,9,26].

    Nowadays, there is an increasing demand for e-learning courses [27]. Many software platforms and learning management systems (LMS) are being used to support Web-based courses for online continuing medical education. Among the various LMS, there is Moodle (Modular Object Oriented Dynamic Learning Environment; Moodle Pty Ltd, Perth, Australia), a well-known platform that is considered one of the best open-source LMS, in what concerns user-friendliness and adaptivity [28]. Moodle allows the integration of a broad range of educational resources, activity modules, such as Forums, Wikis, and Databases, that build a rich collaborative community of learning around a subject matter depending on the learning goal. Moodle can also be used to deliver content to students (such as standard SCORM packages) and for learning assessment; it provides assignments or quizzes. As another advantage, its interface allows surfing through the contents intuitively [28,29]. The reason we decided to use Moodle in this study is because we had access to it in the Faculty of Medicine of the University of Porto that could host the Moodle course, and given its utility, we did not need to address the costly development of a new platform.

    Objectives of the Study

    In this context, as primary outcomes we aim to (1) develop an e-learning Pediatric Cardiology Basics Moodle course for nonspecialists and (2) evaluate its pedagogical impact and user satisfaction.

    As secondary outcomes, we want to understand whether there are significant differences between different types of professionals undergoing the course and to measure the adherence to the course in this specific scenario.


    The E-Learning Course Description

    We chose to build a new course, as the few actual existing options are in English, and also associated with other institutions. The course was in Brazilian Portuguese, implemented on Moodle, hosted on the server at the Faculty of Medicine of the University of Porto, Portugal. The contents of the course were mainly based on the “Cardiologia para o Pediatra” book [30], which was used in this network to teach their professionals, and was written by a specialist Dra Sandra Mattos, stakeholder of the RCP network. This book follows a didactic approach, where the basic concepts for neonatal screening of congenital heart diseases are addressed, such as the current standard clinical protocols and guidelines to apply [31,32]; proposed by the American Association of Cardiology and the Association for European Paediatric and Congenital Cardiology, concepts of cardiac neonatal anatomy and physiology, how to obtain the ultrasound images, and for each anatomic window, which echocardiographic findings to expect and look for. The course was structured into 3 modules, with a total of 8 chapters (Table 1).

    The content was presented using text, images, and videos. Diagnostic images and videos were collected directly from the RCP network at the Real Hospital Português in Recife, Brazil. A total of 22 images and 36 videos were used in this course, which were anonymized and illustrated examples of planes and imaging cuts, without any association with identifiable data from patients. Pediatric cardiology specialists from the cardiology and fetal medicine unit at Real Hospital Português reviewed all contents. Screenshots are presented in Figure 1.

    Table 1. Description of the main contents contained in the intervention e-learning course.
    ModulesChaptersContents
    ForeknowledgeCardiac neonatal anatomy
    Physiology of neonatal circulation
    Physical ultrasound principles    		Internal configuration of the heart
    Fetal circulation
    Neonatal circulatory changes
    The ultrasound properties
    Transducers
    Echocardiography

    Echocardiogram screeningHow to obtain the ultrasound images
    Different echocardiographic modalities    		4-chamber image
    Left ventricle outflow track image
    Right ventricle outflow track image
    Pathologies to exclude in each image
    Bidimensional mode
    M mode
    Doppler mode
    PathologiesIdentified pathologies in the 4-chamber image
    Identified pathologies in the outflow tracts image
    Difficult to diagnose pathologies images in the neonatal period    		Interventricular communication
    Defect of atrioventricular septum
    Tricuspid atresia
    Ductus arteriosus
    Aortic stenosis
    Pulmonary stenosis
    Tetralogy of Fallot
    Transposition of the great vessels
    Truncus arteriosus
    Interatrial communication
    Coarctation of the aorta
    Total anomalous pulmonary venous drainage
    ‎
    Figure 1. Course flow diagram.
    View this figure

    Recruitment and Implementation Strategy

    Participants were recruited via an email disclosure within the RCP network platform with a brief description of the study, the link to the course, a tutorial for Moodle log, and the password for enrolment in the course. To access it, the learners should register on the Moodle website according to their personal information. Thus, website security was guaranteed through an authentication mechanism with username and password. After that, we had access to the participant’s personal email given at the registry, and then we could do the follow-up of each participant by email.

    The course was created and revised between April 2014 and July 2014. We contacted the participants in September 2014, and those who registered to do the course first did the pretest and then had access to the asynchronous course, which was given until November 2014.

    The course was built in a unidirectional way (Figure 2). First, the participants registered on the Moodle website. Then, a pretest of 16 questions was available during 20 min. After the answers submission, the first chapter was available. At the end of each chapter, a summary with the main key-points and a formative test of two multiple-choice questions for self-assessment was presented. These intermediate tests allowed immediate knowledge self-assessment. The next chapter would only be available if the user had given the correct answers. In case of error, the learner would be directed to a summary of the lecture, and then it was possible to go back to the previous lecture or repeat the assessment. At the end of these 8 chapters, a final summative assessment of 16 questions corresponding to the posttest was available during 20 min. After the final approval, a certificate was sent to the learners.

    The learning activities chosen were lessons that corresponded to each chapter. We organized the content, images, and videos in different pages as it corresponds to different topics. The learner could control the lecture flow by pressing control buttons located at the end of that interface, moving forward or backwards, allowing the lecture to flow from the beginning to the end, page by page, managing his own learning process.

    As the course was performed asynchronously, the learner could observe his evolution learning at any time during the course through a progress status bar that updates dynamically in order to manage his self-learning.

    ‎
    Figure 2. Screenshot of one of the echocardiographic views and schemes (left), and of one of the self-assessment tests (right) included in the e-learning course.
    View this figure

    Evaluation Strategy

    An important method of assessing educational training is a framework developed by Kirkpatrick [33,34], which focuses on 4 levels: reaction, learning, behavior, and results.

    Numerous studies [16,21,35-42] have been carried out to assess the benefits of e-learning in medical education. Very often, evaluative education studies [38,43] rely solely on the reaction level and learning over the behavior and results levels [16,44,45].

    Pre- and Postintervention Test

    Concerning the learning assessment, the most common method described in literature is the pre- and posttesting self-controlled method, with multiple-choice test scores.

    In this study, the participants were asked to do a pretest before taking the course and a posttest after completing the course. We used the same questions for pre and posttesting, so that we can guarantee the same level of difficulty and comparable results [46]. The test was structured with 16 different multiple-choice questions. They were all single-select questions and each question had 2-5 response options. The assessment questions were text-based to test knowledge based on the key ideas, learning outcomes, and objectives established for the course. In this test, a total of 2 questions were related to each of the course’s 8 modules (Table 1), which were also the same end-of-chapter practice questions. The participants could only proceed to the next chapter after answering these end-of-chapter practice questions correctly. An expert on pediatric cardiology revised the questions making sure they were appropriate to the course content. To improve the authenticity of the answers by discouraging access to support materials, participants had only 20 min to complete the 16-question test, so a time controller and a progress bar were available. After that time, if they had not submitted, the answers would be automatically saved. At the end, the participants could see the test result, but not the correction, and they could not repeat the test. The technology we used to provide this evaluation tool was the Moodle lesson questions, a free Web-based office suite and data storage service.

    User’s Satisfaction Questionnaire

    The reaction assessment is mostly done by questionnaires. One of the most cited questionnaires for assessing user satisfaction was developed by Wang [47]. Wang created an e-learning satisfaction model that consists of 26 items related to 4 qualities: content quality, learning interface quality, personalization quality, and learning community quality. However, the last 2 questions refer to global measurement in the context of end-user satisfaction, first developed by Doll in 1988 [48], and related to overall satisfaction and overall success. The measurement scale used was a 7-point Likert-type scale, with anchors ranging from “strongly disagree” to “strongly agree.” Globally, this questionnaire was shown to have a reliability (Cronbach alpha) of .95 [47].

    For our study, we used the questions related to content quality, learning interface quality, and personalization quality from Wang’s questionnaire [47]. We did not use questions related to learning interaction quality, because the developed e-learning course was asynchronous. We used a Portuguese translated existing version from Wang’s questionnaire [49]. The technology used to apply this evaluation instrument was the Moodle survey module. The satisfaction questionnaire became available to the participants who had taken the course after they did the postcourse.

    Statistical Analysis

    The purpose of the data analysis was to determine whether there was a significant difference between the test score before and after the course. We considered learning improvement as the ratio of the difference between scores and the preintervention score. The learning efficiency is the learning improvement per hour of the course.

    Data analysis was performed by descriptive and inferential statistics, using the IBM SPSS Statistics software version 22.0.

    According to the fulfillment of the criteria necessary to perform parametric hypothesis testing, it was concluded that the sample did not follow a normal distribution. Thus, we used the following nonparametric tests: Wilcoxon Signed-Rank test, Mann-Whitney U Test and Kruskall-Wallis H test. For all this statistical analysis, we considered a significance level of 5% [50].


    Summary

    The target population in this study was the health care professionals—neonatologists, pediatricians, obstetricians, nurses, and internship medical students—who work in the RCP network. This convenience universe potentially includes a total of around 80 people. We obtained 78% (62/80) registrations. Concerning adherence (Figure 3), 79% (49/62) started the course. At the end, 61% of the participants (30/49) managed to complete the course and 39% (19/49) dropped out (Figure 4). See the percentage of participants who dropped by chapter (Figure 5). From the participants who had taken the course, 67% (20/30) responded to the satisfaction questionnaire.

    ‎
    Figure 3. Screenshot of one of the self-assessment tests included in the e-learning course.
    View this figure
    ‎
    Figure 4. Participant flow diagram showing the enrolled sample and respective dropouts.
    View this figure
    ‎
    Figure 5. Percentage of participants who dropped by chapter.
    View this figure

    Sample Description

    The total sample consisted of 62 registered health care practitioners, including 49 female (79%) and 13 male elements (21%); 49 individuals started the course, including 39 female (80%) and 10 male elements (20%); and 30 participants completed the course, including 22 female (73%) and 8 male elements (27%; Table 2).

    With regard to the registered participants, most are doctors (n=33; 53%) and mostly are of Neonatology (n=24; 39%) and Pediatrics (n=23; 37%) departments. This ratio remains the same for those participants who initiated the course (Table 3).

    Regarding the state where they practice, 45 (73%) works in Paraíba and the other 17 (27%) in Pernambuco.

    Pedagogical Impact

    Regarding the pretest, 67% (n=20) passed (test score≥50%) and 33% (n=10) of participants failed (test score<50%). The test scores ranged between 0 and 100%. However, none of the participants failed to complete all the test items within the time limit. With respect to the posttest, 100% (n=30) of the participants passed.

    The differences between the test scores before and after the course were all positive. There were no negative differences or equal scores before and after the course. A Wilcoxon signed-ranks test indicated that the median [Mdn (P25-P75)=94 (81-100)] posttest ranks were statistically significantly higher than the median [Mdn (P25-P75)=56 (38-69)] pretest ranks (Z=-4.788; P<.001).

    Doing an intent-to-treat analysis by comparing the mean ranks (Mann-Whitney–U Test) of the score from the participants who did not do the course [Mdn (P25-P75)=50 (29-65); n=19] with those who did the course [Mdn (P25-P75)=94 (81-100); n=30], we found that test scores in those who did the course were statistically significantly higher than those who did not (U=21.500; P<.001).

    Globally, the difference between the final and the initial scores can indicate the learning impact of the course [Mdn (P25-P75)=34 (19-50); n=30]. Moreover, the improvement in the results was related to what the participants already knew before the course [Mdn (P25-P75)=61 (36-114); n=30]. The efficiency was how much they improve per hour dedicated to the course [Mdn (P25-P75)=31 (12-80); n=30] (Figure 6). The maximum time dedicated to the course was 09 h 58 min and the minimum 25 min [Mdn (P25-P75)=01:47 (01:08-03:01); n=30].

    We found the initial scores were statistically significantly different (X22=13.8 P=.001) between professions (Table 4). Although nurses hadn’t the highest final score, they were the ones who exhibited the highest difference between the scores, more than doubling it, but with less efficiency, as they also dedicated more time to the course. However, internship medical students had the highest learning efficiency, improving their knowledge per hour, as they dedicated much less time to the course than doctors and nurses. Predictably, doctors have lesser benefits from such an e-learning course. There were significant statistical differences in the improvement between professions (X22=8.6; P=.01).

    Table 2. Frequency and percentage of participants by gender.
    ParticipantsFemale, n (%)Male, n (%)
    Total sample (n=62)49 (79)13 (21)
    Started the course (n=49)39 (80)10 (20)
    Completed the course (n=30)22 (73)8(27)
    Table 3. Frequency and percentage of participants by profession and specialty.
    ParticipantsAssigned
    (n=62), n (%)    		Initiated
    (n=49), n (%)    		Concluded
    (n=30), n (%)
    Profession




    		Doctors33 (53)27 (55)16 (53)

    		Internship medical students10 (16)9 (18)8 (27)

    		Nurses19 (31)13 (27)6 (20)
    Speciality




    		Neonatology24 (39)19 (39)18 (60)

    		Obstetrics6 (10)5 (10)4 (13)

    		Pediatric Cardiology9 (14)8 (16)3 (10)

    		Pediatrics23 (37)17 (35)5 (17)
    Table 4. Median and percentile Tukey’s hinges (P25-P75) of the scores in study by profession.
    IndicatorsDoctors (n=16)Internship medical students (n=8)Nurses (n=6)P valuea
    Initial score63 (53-78)50 (41-59)40 (31-50).001
    Final score99 (88-100)94 (81-94)88 (81-100).24
    Differenceb27 (13-44)34 (28-44)51 (31-63).13
    Improvement (%)c43 (16-89)63 (55-91)128 (63-200).09
    Efficiency (%)/hd15 (8-42)76 (53-153)26 (16-80).01
    Dedication (hh:mm)02:01 (01:32-03:06)00:50 (00:37-01:06)03:07 (02:09-03:54).45

    aKruskal-Wallis H test.

    bFinal score - initial score.

    c([final score - initial score]/ initial score)*100.

    d([final Score - Initial score]/ initial score)*100)/hour.

    Concerning specialty (Table 5), the obstetricians were the ones who benefited the most, with the highest difference between the scores, the best improvement and efficiency (Figure 7), and were the most dedicated to the course. There were statistically significant differences in the improvement (X22=8.4; P=.04) and in the course dedication (X22=8.2; P=.04) between the different specialties.

    Table 5. Median and percentile Tukey’s hinges (P25-P75) of the scores in study by specialty.
    IndicatorsNeonatology (N=18)Obstetrics (N=4)Pediatric Cardiology (N=3)Pediatrics (N=5)P valuea
    Initial score56 (44-63)37 (25-46)87 (68-87)69 (69-81).06
    Final score91 (81-100)88 (81-87)10098 (94-100).15
    Differenceb38 (25-44)60 (45-63)13 (13-32)13 (13-29).06
    Improvement (%)c61 (43-100)170 (101-267)15 (15-58)18 (15-42).039
    Efficiency (%)/hd42 (20-78)50 (18-117)12 (10-50)9 (7-16).28
    Dedication (hh:mm)01:39
    (00:54-03:01)    		03:12
    (02:20-05:30)    		01:20
    (01:14-1:36)    		01:35
    (01:08-2:08)    		.04

    aKruskal-Wallis H test.

    bFinal score-initial score.

    c([final score-initial score]/ initial score)*100.

    d([final score-initial score]/ initial score)*100)/hour.

    ‎
    Figure 6. Improvement (left) and learning efficiency (right) stratified by professional status.
    View this figure
    ‎
    Figure 7. Improvement (left) and learning efficiency (right) stratified by specialty.
    View this figure

    User Satisfaction

    Only 20 (67%), out of the 30 participants who completed the course answered the satisfaction questionnaire. This was disappointing, showing that it was simpler to motivate these professionals to gain knowledge, than to contribute to a research study.

    Globally, the satisfaction with the e-learning course was positive (μ=87%; σ=12%; minimum=67%; maximum=100%), whereas 6 (30%) learners were totally satisfied with the e-learning course. Regarding global measures (Q14 and Q15), 17 (88%) were satisfied and considered that the e-learning course was successful (6 and 7 points in the 7-point Likert-type scale). Considering content quality (Q1-Q4), 17 (88%) learners think that this course fits their needs, is useful, sufficient, and up to date.

    About the learning interface quality (Q5-Q9), 15 (76%) learners found the e-learning course user-friendly, stable, making it easy to find contents needed. Concerning personalization quality (Q10-Q13), 16 (84%) participants think the e-learning course enables them to control the learning progress, to learn the content needed, to choose what to learn and record their learning progress and performance.

    Within these 20 participants, all claim the necessity of continuing medical education (CME) and 18 (90%) said that they would like to do CME by e-learning systems, whereas the other 2 (10%) said that maybe they will be interested in CME by e-learning.


    Principal Findings

    The first main result of this study was the development of an e-learning course for the neonatal screening of congenital heart diseases. Using a free open-access tool such as Moodle and adapting the pedagogical contents of a well-established book for teaching this subject did not require advanced programming skills and led to an effective e-learning course. Given our successful results, it is expectable that future e-learning courses that specifically use best practices for multimedia learning should have an even stronger impact.

    The second principal outcome was the statistically significant results obtained in the used metrics of pedagogical impact, with quite interesting proportions of learning with just a few hours of training. These results meet the general literature reviews that identified e-learning as superior to noneducation intervention [45], or as having similar effects to traditional learning [51], and more effective when combined with traditional learning [29] (b-learning). Although, in some contexts, e-learning may not be an alternative for the traditional face-to-face learning method, it can always offer a contribution and be a complement, and a useful adjunct to traditional education [51,52].

    When comparing professions, we confirmed that the background knowledge about pediatric cardiology varies by professional group. By the end of the course nurses were the ones who learned more, although they did not score the highest final score. The heterogeneity of time spent in learning was also perceived. Internship medical students had the highest learning efficiency, dedicating much less time to the course than doctors and nurses. Predictably, doctors have lesser benefits from such an e-learning course.

    Concerning specialty, the obstetricians were the ones who improved their knowledge the most, with the highest difference between the scores, the best improvement and efficiency, and were the most dedicated to the course. There were statistically significant differences in the improvement and in the course dedication between the different specialties, in this case confirming that more time dedicated to the course does translate into higher knowledge gain.

    The third principal result was the encouraging assessment of the user satisfaction questionnaire. However, a third of the students who completed the course did not complete this questionnaire. This might be an important consideration for future studies, with regard to the interest and motivation of the participants to answer satisfaction questionnaires.

    Regarding the adherence to this study, although the number of participants can appear low, we must understand the context in which this course was applied. From a total of around 80 people who are fully engaged in their daily health care routine and who voluntarily accepted to participate and complete an 8-lesson course in their free time, we consider that 30 is a successful result that reflects the need and enthusiasm generated by the proposed initiative.

    Limitations

    Relevant limitations include the small sample size that affects the generalization of the results, and the limited implementation time, as the learners have their own agenda and priorities.

    For the analysis, we had to use nonparametric tests because some of the variables did not follow a normal distribution and mainly because of the sample size. It would be definitely more interesting to use parametric methods to prove that there were significant differences before and after the course and the interactions between professions and specialty in a more robust manner, but for that possibility we would have to have a greater number of participants.

    In addition, we have the impact of the exposure to the pretest and the end-of-chapter practice questions, which were identical to the ones that were used in the posttest. Although there was an expert validation of the test questions used before and after the course, the test was not assessed for its reliability or validity metrics.

    The validity of the satisfaction assessment should be carefully considered, because we used a nonvalidated translation to a Portuguese version of the user satisfaction questionnaire, as it was not possible to find in the literature a Portuguese validated one for e-learning systems.

    Another limitation of this work is the fact that the socioprofessional questionnaire was applied only at the end of the intervention. This situation limits our information about the description of our sample and other factors that could influence the adherence and learning process.

    Concerning the evaluation strategy, we also faced the risk of a slight bias because we did not control if learners resorted to external sources in order to provide correct answers to the tests.

    Conclusions

    Globally, this study highlights the importance of training neonatologists and other health care professionals in the neonatal care units to screen for congenital heart disease. We consider the high rate of participation an important aspect of our study (78%), which reflects the great interest shown by these professionals to promote their professional skills. They took advantage of this learning opportunity, which confirms that these health care professionals are committed to responding to new challenges and evolving paradigms.

    This study contributes to the Brazilian continuing training programs, as we did not find any similar course related to neonatal screening of congenital heart disease. It would be interesting to conduct additional assessments to demonstrate effective consolidation of knowledge gain. For future work, we also intend to assess the remaining levels of the Kirkpatrick framework, “behavior” and “results,” with respect to change in neonatal screening behavior and improved congenital heart disease detection. To do so, we plan to measure the number of telemedicine consultations conducted by the participants after the course and the number of congenital heart disease detected by them.

    Our global results show that e-learning can provide statistically relevant knowledge gains in health care professionals in a neonatal screening context. We believe that this study underlines the importance of e-learning as a viable technology for training, especially in impoverished contexts. E-learning should be considered for continuing medical education in low- and middle-income countries, not only due to budget constraints, but also due to resource-constrained environments.

    Acknowledgments

    All the authors have made a significant contribution to this manuscript and have approved the final paper, thus meeting the criteria for authorship. All those entitled to authorship are listed as authors.

    The authors thank Real Hospital Português, UCMF, for providing and approving images for the course, and the Faculty of Medicine of the University of Porto for hosting the Moodle course.

    Conflicts of Interest

    None declared.

    1. Bernier P, Stefanescu A, Samoukovic G, Tchervenkov CI. The challenge of congenital heart disease worldwide: epidemiologic and demographic facts. Semin Thorac Cardiovasc Surg Pediatr Card Surg Annu 2010;13(1):26-34. [CrossRef] [Medline]
    2. Knowles R, Griebsch I, Dezateux C, Brown J, Bull C, Wren C. Newborn screening for congenital heart defects: a systematic review and cost-effectiveness analysis. Health Technol Assess 2005 Nov;9(44):1-152, iii [FREE Full text] [Medline]
    3. Huber J, Peres VC, Santos TJ, Beltrão L, Baumont AC, Cañedo AD, et al. Congenital heart diseases in a reference service: clinical evolution and associated illnesses. Arq Bras Cardiol 2010 Mar;94(3):313-8, 333 [FREE Full text] [Medline]
    4. Tubman TR, Shields MD, Craig BG, Mulholland HC, Nevin NC. Congenital heart disease in Down's syndrome: two year prospective early screening study. BMJ 1991 Jun 15;302(6790):1425-1427 [FREE Full text] [Medline]
    5. Fixler DE, Nembhard WN, Xu P, Ethen MK, Canfield MA. Effect of acculturation and distance from cardiac center on congenital heart disease mortality. Pediatrics 2012 Jun;129(6):1118-1124 [FREE Full text] [CrossRef] [Medline]
    6. Mattos S, Hazin SM, Regis CT, de Araújo JS, Albuquerque FC, Moser LR, et al. A telemedicine network for remote paediatric cardiology services in north-east Brazil. Bull World Health Organ 2015 Dec 01;93(12):881-887 [FREE Full text] [CrossRef] [Medline]
    7. Mattos S, Hazin SM, Regis CT, Soares de Araújo J, Albuquerque F, Moser LR, et al. A telemedicine network for remote paediatric cardiology services in north-east Brazil. Bull World Health Organ 2015 Dec 01;93(12):881-887 [FREE Full text] [CrossRef] [Medline]
    8. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med 2006 Mar;81(3):207-212. [Medline]
    9. Sandars J. Cost Effectiveness in Medical Education. United Kingdom: Radcliffe Publishing; 2010.
    10. Frehywot S, Vovides Y, Talib Z, Mikhail N, Ross H, Wohltjen H, et al. E-learning in medical education in resource constrained low- and middle-income countries. Hum Resour Health 2013 Feb 04;11:4 [FREE Full text] [CrossRef] [Medline]
    11. Bouhnik D, Marcus T. Interaction in distance-learning courses. J Am Soc Inf Sci Technol 2006;57:299-305.
    12. Liaw S, Huang H, Chen G. Surveying instructor and learner attitudes toward e-learning. Computers & Education 2007;49:1066-1080. [CrossRef]
    13. Raab R, Ellis W, Abdon B. Multisectoral partnerships in e-learning. The Internet and Higher Education 2001 Jan;4(3-4):217-229. [CrossRef]
    14. Liaw S. Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers & Education 2008 Sep;51(2):864-873. [CrossRef]
    15. McKimm J, Jollie C, Cantillon P. ABC of learning and teaching: web based learning. BMJ 2003 Apr 19;326(7394):870-873 [FREE Full text] [CrossRef] [Medline]
    16. Wutoh R, Boren SA, Balas EA. eLearning: a review of internet-based continuing medical education. J Contin Educ Health Prof 2004;24(1):20-30. [CrossRef] [Medline]
    17. Abdellatief M. A technique for quality evaluation of e-learning from developers perspective. American Journal of Economics and Business Administration 2011 Jan 01;3(1):157-164. [CrossRef]
    18. Zhang D, Zhao J, Zhou L, Nunamaker J. Can e-learning replace classroom learning? Commun ACM 2004 May 01;47(5):75-79. [CrossRef]
    19. Capper J. IST. 2001. E-learning growth and promise for the developing world   URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.600.3600&rep=rep1&type=pdf
    20. Abdullah H. Queen’s University. 2012. The development of an introductory e-learning module on echocardiography   URL: https:/​/qspace.​library.queensu.ca/​bitstream/​handle/​1974/​7161/​Abdullah_Hanin_201202_MSC.​pdf;jsessionid=96A797355F0A45AB86144672136706BB?sequence=1 [accessed 2017-05-03] [WebCite Cache]
    21. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010;10:12 [FREE Full text] [CrossRef] [Medline]
    22. Omar A, Kalulu D, Alijani G. Management of innovative e-learning environments. AELJ 2011;15:37-64.
    23. Harun M. Integrating e-learning into the workplace. Internet High Educ 2001 Jan;4(3-4):301-310. [CrossRef]
    24. Nisar T. E-Learning in public organizations. Public Personnel Management 2004 Mar;33(1):79-88. [CrossRef]
    25. Walsh K. E-learning in medical education: problems with research. J Contemp Med Edu 2016;4(3):132-134. [CrossRef]
    26. Gibbons AS, Fairweather PG. Computer-based Instruction: Design and Development. Eaglewood Cliffs, New Jersey: Educational Technology Publications; 1998.
    27. Loftus J, Stavraky T, Urquhart B. Design it yourself (DIY): in-house instructional design for online pharmacology. Adv Health Sci Educ Theory Pract 2014 Dec;19(5):645-659. [CrossRef] [Medline]
    28. Reyes NR, Candeas PV, Galan SG, Viciana R, Canadas F, Reche PJ. Comparing open-source e-learning platforms from adaptivity point of view. : IEEE; 2009 Presented at: EAEEIE Annual Conference; June 22-24, 2009; Valencia, Spain.
    29. Núñez JC, Cerezo R, Bernardo A, Rosário P, Valle A, Fernández E, et al. Implementation of training programs in self-regulated learning strategies in Moodle format: results of a experience in higher education. Psicothema 2011 Apr;23(2):274-281. [Medline]
    30. Mattos SS. Cardiologia para o Pediatra - Manual Pratico para o Reconhecimento e Manuseio das Cardiopatias na Infancia. Brasil: Caduceus; 2004.
    31. Lai WW, Geva T, Shirali GS, Frommelt PC, Humes RA, Brook MM, Task Force of the Pediatric Council of the American Society of Echocardiography, Pediatric Council of the American Society of Echocardiography. Guidelines and standards for performance of a pediatric echocardiogram: a report from the Task Force of the Pediatric Council of the American Society of Echocardiography. J Am Soc Echocardiogr 2006 Dec;19(12):1413-1430. [CrossRef] [Medline]
    32. Mertens L, Helbing W, Sieverding L, Daniels O, Working Group on Cardiac Imaging of the Association for European Paediatric Cardiology. Guidelines from the Association for European Paediatric Cardiology: standards for training in paediatric echocardiography. Cardiol Young 2005 Aug;15(4):441-442. [CrossRef] [Medline]
    33. Galloway D. Evaluating distance delivery and e-learning is kirkpatrick's model relevant? Nonprofit Management Leadership 2005 Apr;44(4):21-27. [CrossRef]
    34. Alliger GM, Janak EA. Kirkpatrick's levels of training criteria: thirty years later. Pers Psychol 1989 Jun;42(2):331-342. [CrossRef]
    35. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA 2008 Sep 10;300(10):1181-1196. [CrossRef] [Medline]
    36. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? a review of the evaluation literature. Acad Med 2002 Oct;77(10 Suppl):S86-S93. [Medline]
    37. Cook DA, Garside S, Levinson AJ, Dupras DM, Montori VM. What do we mean by web-based learning? a systematic review of the variability of interventions. Med Educ 2010 Aug;44(8):765-774. [CrossRef] [Medline]
    38. Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Med Educ 2005 Jun;39(6):561-567. [CrossRef] [Medline]
    39. Cobb SC. Internet continuing education for health care professionals: an integrative review. J Contin Educ Health Prof 2004;24(3):171-180. [CrossRef] [Medline]
    40. Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud 2014 Jan;51(1):136-149. [CrossRef] [Medline]
    41. Schittek M, Mattheos N, Lyon HC, Attström R. Computer assisted learning. A review. Eur J Dent Educ 2001 Aug;5(3):93-100. [Medline]
    42. Feng J, Chang Y, Chang H, Erdley WS, Lin C, Chang Y. Systematic review of effectiveness of situated e-learning on medical and nursing education. Worldviews Evid Based Nurs 2013 Aug;10(3):174-183. [CrossRef] [Medline]
    43. Curran VR, Fleet LJ, Kirby F. A comparative evaluation of the effect of internet-based CME delivery format on satisfaction, knowledge and confidence. BMC Med Educ 2010 Jan 29;10:10 [FREE Full text] [CrossRef] [Medline]
    44. Maloney S, Chamberlain M, Morrison S, Kotsanas G, Keating JL, Ilic D. Health professional learner attitudes and use of digital learning resources. J Med Internet Res 2013;15(1):e7 [FREE Full text] [CrossRef] [Medline]
    45. Stark CM, Graham-Kiefer ML, Devine CM, Dollahite JS, Olson CM. Online course increases nutrition professionals' knowledge, skills, and self-efficacy in using an ecological approach to prevent childhood obesity. J Nutr Educ Behav 2011;43(5):316-322. [CrossRef] [Medline]
    46. Maki WS. Evaluating Outcomes. In: Assessment and Education Research. Texas Tech University, Lubbock, TX: AAAS; 2004:27-32.
    47. Wang Y. Assessment of learner satisfaction with asynchronous electronic learning systems. Information & Management 2003 Oct;41(1):75-86. [CrossRef]
    48. Xiao L, Dasgupta S. Measurement of user satisfaction with web-based information systems: An empirical study. 2002 Presented at: Eighth Americas Conference on Information Systems; December 2002; AIS Electronic Library (AISeL) p. 1149-1155.
    49. Moreira IC, Ventura SR, Ramos I, Rodrigues PP. Development and assessment of an e-learning course on breast imaging for radiographers: a stratified randomized controlled trial. J Med Internet Res 2015 Jan 05;17(1):e3 [FREE Full text] [CrossRef] [Medline]
    50. Marôco J. Análise Estatística com o SPSS Statistics. Lisboa: ISPA; 2014.
    51. Nkenke E, Vairaktaris E, Bauersachs A, Eitner S, Budach A, Knipfer C, et al. Acceptance of technology-enhanced learning for a theoretical radiological science course: a randomized controlled trial. BMC Med Educ 2012 Mar 30;12:18 [FREE Full text] [CrossRef] [Medline]
    52. Mahnken AH, Baumann M, Meister M, Schmitt V, Fischer MR. Blended learning in radiology: is self-determined learning really more effective? Eur J Radiol 2011 Jun;78(3):384-387. [CrossRef] [Medline]

    ‎
    CME: continuing medical education
    LMS: learning management system
    Moodle: Modular Object Oriented Dynamic Learning Environment

    Edited by CL Parra-Calderón; submitted 11.12.15; peer-reviewed by A Levinson, P Woei Bing; comments to author 30.01.16; revised version received 14.11.16; accepted 02.02.17; published 10.05.17

    Copyright

    ©Ana Cristina Oliveira, Sandra Mattos, Miguel Coimbra. Originally published in JMIR Medical Education (http://mededu.jmir.org), 10.05.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.