Managing congenital heart diseases is a worldwide problem [1]. Early detection through newborn screening can potentially improve the outcome of these diseases [2]. In newborns, congenital heart diseases can be detected by auscultation, pulse oximetry, radiography, catheterization, although transthoracic echocardiogram, a specialized form of ultrasound, is the elected exam for diagnosis [3,4]. Pediatric cardiologists usually perform this examination, but in developing countries there is shortage of professional specialists, which are often concentrated in larger urban centers, hindering the widespread population screening and causing a need for constant transferal of patients from the isolated regions to reference health centers [5].

In response to these challenges, the Health Secretary of Paraíba, in Brazil established a pediatric cardiology network [6]—Rede de Cardiologia Pediátrica (RCP)—in partnership with Círculo do Coração , a civil nonprofit organization from Recife, in order to create a neonatal screening program for the whole state of Paraíba and a hospital facility designed to manage patients. Of the 20 maternity centers covered by this network, only 7 had neonatal ICU beds. The remaining are level-1 district centers. In only one center, in the capital city of Joao Pessoa, there is a pediatric cardiologist available for echo and clinical diagnosis. No center performs cardiac surgery. This network provides perinatal and neonatal care in remote areas supervised by telemedicine; it was created due to the need to train local physicians and involve local professionals on screening, diagnosis, therapeutic treatment and management of congenital heart diseases in fetuses, newborns, and children of the public health system.

Echocardiography is used to diagnose congenital heart disease when either there is an abnormal clinical examination or an abnormal pulse oximetry. Abnormal pulse oximetry results are automatically noted on a database of the network, allowing the network to contact the clinic and request that they follow up any babies with abnormal test results. These active search protocols track the discharged neonates and ensure that abnormal findings are acted on.

Echo image acquisition and its interpretation is a combined process developed within this Web-based telemedicine network where trainees are constantly supervised by cardiologists. Not only the whole process is documented in the Web database system to track the apprentice's development, but also whenever needed, live interaction is called to adjust practical aspects. Initially, all echocardiograms needed to be performed with Web-based supervision by the pediatric cardiologist, as part of the neonatologists’ training, but with time cardiologists would only be requested for direct Web-based supervision when pathological findings were suspected. As there are always new neonatologists being trained, this learning process and interaction between teams is a continuous cycle [7].

With the increasing use of Internet information and communication technologies, e-learning has emerged as a widely accepted modality in medical education [8]. For its convenience and its potential for cost savings, it has become popular among the medical education community [9]. E-learning is known to offer learning opportunities where there is limited access to teaching in a specific field, either because of a lack of qualified or geographically distant teaching institution [10]. Therefore, e-learning can be a powerful tool to increase the capacity of health professionals in constricted contexts for neonatal echocardiography congenital heart disease screening. There is evidence in the literature that e-learning is a useful tool for overcoming barriers to health professionals training [10]. A review carried out by Frehywot et al on e-learning in medical education in resource-constrained LMICs suggests that e-learning may be effective for increasing capacity in rural settings, although evidence is still limited.

The convenience, the self-paced and learner-centered learning, and the creation of a global learning community, are some significant benefits of e-learning that have been discussed in many articles [8,10-24].

In medical contexts, e-learning program results such as efficiency and costeffectiveness have typically gone unreported [25]. In nonmedical contexts, there is evidence that e-learning can result in cost savings of up to 50% over traditional learning programs, due to reduced instructor training time, travel and labor costs, institutional infrastructure, and the possibility of expanding programs with new technologies [8,9,26].

Nowadays, there is an increasing demand for e-learning courses [27]. Many software platforms and learning management systems (LMS) are being used to support Web-based courses for online continuing medical education. Among the various LMS, there is Moodle (Modular Object Oriented Dynamic Learning Environment; Moodle Pty Ltd, Perth, Australia), a well-known platform that is considered one of the best open-source LMS, in what concerns user-friendliness and adaptivity [28]. Moodle allows the integration of a broad range of educational resources, activity modules, such as Forums, Wikis, and Databases, that build a rich collaborative community of learning around a subject matter depending on the learning goal. Moodle can also be used to deliver content to students (such as standard SCORM packages) and for learning assessment; it provides assignments or quizzes. As another advantage, its interface allows surfing through the contents intuitively [28,29]. The reason we decided to use Moodle in this study is because we had access to it in the Faculty of Medicine of the University of Porto that could host the Moodle course, and given its utility, we did not need to address the costly development of a new platform.

In this context, as primary outcomes we aim to (1) develop an e-learning Pediatric Cardiology Basics Moodle course for nonspecialists and (2) evaluate its pedagogical impact and user satisfaction.

As secondary outcomes, we want to understand whether there are significant differences between different types of professionals undergoing the course and to measure the adherence to the course in this specific scenario.

We chose to build a new course, as the few actual existing options are in English, and also associated with other institutions. The course was in Brazilian Portuguese, implemented on Moodle, hosted on the server at the Faculty of Medicine of the University of Porto, Portugal. The contents of the course were mainly based on the “Cardiologia para o Pediatra” book [30], which was used in this network to teach their professionals, and was written by a specialist Dra Sandra Mattos, stakeholder of the RCP network. This book follows a didactic approach, where the basic concepts for neonatal screening of congenital heart diseases are addressed, such as the current standard clinical protocols and guidelines to apply [31,32]; proposed by the American Association of Cardiology and the Association for European Paediatric and Congenital Cardiology, concepts of cardiac neonatal anatomy and physiology, how to obtain the ultrasound images, and for each anatomic window, which echocardiographic findings to expect and look for. The course was structured into 3 modules, with a total of 8 chapters (Table 1).

The content was presented using text, images, and videos. Diagnostic images and videos were collected directly from the RCP network at the Real Hospital Português in Recife, Brazil. A total of 22 images and 36 videos were used in this course, which were anonymized and illustrated examples of planes and imaging cuts, without any association with identifiable data from patients. Pediatric cardiology specialists from the cardiology and fetal medicine unit at Real Hospital Português reviewed all contents. Screenshots are presented in Figure 1.

Participants were recruited via an email disclosure within the RCP network platform with a brief description of the study, the link to the course, a tutorial for Moodle log, and the password for enrolment in the course. To access it, the learners should register on the Moodle website according to their personal information. Thus, website security was guaranteed through an authentication mechanism with username and password. After that, we had access to the participant’s personal email given at the registry, and then we could do the follow-up of each participant by email.

The course was created and revised between April 2014 and July 2014. We contacted the participants in September 2014, and those who registered to do the course first did the pretest and then had access to the asynchronous course, which was given until November 2014.

The course was built in a unidirectional way (Figure 2). First, the participants registered on the Moodle website. Then, a pretest of 16 questions was available during 20 min. After the answers submission, the first chapter was available. At the end of each chapter, a summary with the main key-points and a formative test of two multiple-choice questions for self-assessment was presented. These intermediate tests allowed immediate knowledge self-assessment. The next chapter would only be available if the user had given the correct answers. In case of error, the learner would be directed to a summary of the lecture, and then it was possible to go back to the previous lecture or repeat the assessment. At the end of these 8 chapters, a final summative assessment of 16 questions corresponding to the posttest was available during 20 min. After the final approval, a certificate was sent to the learners.

The learning activities chosen were lessons that corresponded to each chapter. We organized the content, images, and videos in different pages as it corresponds to different topics. The learner could control the lecture flow by pressing control buttons located at the end of that interface, moving forward or backwards, allowing the lecture to flow from the beginning to the end, page by page, managing his own learning process.

As the course was performed asynchronously, the learner could observe his evolution learning at any time during the course through a progress status bar that updates dynamically in order to manage his self-learning.

An important method of assessing educational training is a framework developed by Kirkpatrick [33,34], which focuses on 4 levels: reaction, learning, behavior, and results.

Numerous studies [16,21,35-42] have been carried out to assess the benefits of e-learning in medical education. Very often, evaluative education studies [38,43] rely solely on the reaction level and learning over the behavior and results levels [16,44,45].

The purpose of the data analysis was to determine whether there was a significant difference between the test score before and after the course. We considered learning improvement as the ratio of the difference between scores and the preintervention score. The learning efficiency is the learning improvement per hour of the course.

Data analysis was performed by descriptive and inferential statistics, using the IBM SPSS Statistics software version 22.0.

According to the fulfillment of the criteria necessary to perform parametric hypothesis testing, it was concluded that the sample did not follow a normal distribution. Thus, we used the following nonparametric tests: Wilcoxon Signed-Rank test, Mann-Whitney U Test and Kruskall-Wallis H test. For all this statistical analysis, we considered a significance level of 5% [50].

The target population in this study was the health care professionals—neonatologists, pediatricians, obstetricians, nurses, and internship medical students—who work in the RCP network. This convenience universe potentially includes a total of around 80 people. We obtained 78% (62/80) registrations. Concerning adherence (Figure 3), 79% (49/62) started the course. At the end, 61% of the participants (30/49) managed to complete the course and 39% (19/49) dropped out (Figure 4). See the percentage of participants who dropped by chapter (Figure 5). From the participants who had taken the course, 67% (20/30) responded to the satisfaction questionnaire.

The total sample consisted of 62 registered health care practitioners, including 49 female (79%) and 13 male elements (21%); 49 individuals started the course, including 39 female (80%) and 10 male elements (20%); and 30 participants completed the course, including 22 female (73%) and 8 male elements (27%; Table 2).

With regard to the registered participants, most are doctors (n=33; 53%) and mostly are of Neonatology (n=24; 39%) and Pediatrics (n=23; 37%) departments. This ratio remains the same for those participants who initiated the course (Table 3).

Regarding the state where they practice, 45 (73%) works in Paraíba and the other 17 (27%) in Pernambuco.

Regarding the pretest, 67% (n=20) passed (test score≥50%) and 33% (n=10) of participants failed (test score<50%). The test scores ranged between 0 and 100%. However, none of the participants failed to complete all the test items within the time limit. With respect to the posttest, 100% (n=30) of the participants passed.

The differences between the test scores before and after the course were all positive. There were no negative differences or equal scores before and after the course. A Wilcoxon signed-ranks test indicated that the median [Mdn (P25-P75)=94 (81-100)] posttest ranks were statistically significantly higher than the median [Mdn (P25-P75)=56 (38-69)] pretest ranks (Z=-4.788; P<.001).

Doing an intent-to-treat analysis by comparing the mean ranks (Mann-Whitney–U Test) of the score from the participants who did not do the course [Mdn (P25-P75)=50 (29-65); n=19] with those who did the course [Mdn (P25-P75)=94 (81-100); n=30], we found that test scores in those who did the course were statistically significantly higher than those who did not (U=21.500; P<.001).

Globally, the difference between the final and the initial scores can indicate the learning impact of the course [Mdn (P25-P75)=34 (19-50); n=30]. Moreover, the improvement in the results was related to what the participants already knew before the course [Mdn (P25-P75)=61 (36-114); n=30]. The efficiency was how much they improve per hour dedicated to the course [Mdn (P25-P75)=31 (12-80); n=30] (Figure 6). The maximum time dedicated to the course was 09 h 58 min and the minimum 25 min [Mdn (P25-P75)=01:47 (01:08-03:01); n=30].

We found the initial scores were statistically significantly different (X²₂=13.8 P=.001) between professions (Table 4). Although nurses hadn’t the highest final score, they were the ones who exhibited the highest difference between the scores, more than doubling it, but with less efficiency, as they also dedicated more time to the course. However, internship medical students had the highest learning efficiency, improving their knowledge per hour, as they dedicated much less time to the course than doctors and nurses. Predictably, doctors have lesser benefits from such an e-learning course. There were significant statistical differences in the improvement between professions (X²₂=8.6; P=.01).

Concerning specialty (Table 5), the obstetricians were the ones who benefited the most, with the highest difference between the scores, the best improvement and efficiency (Figure 7), and were the most dedicated to the course. There were statistically significant differences in the improvement (X²₂=8.4; P=.04) and in the course dedication (X²₂=8.2; P=.04) between the different specialties.

Only 20 (67%), out of the 30 participants who completed the course answered the satisfaction questionnaire. This was disappointing, showing that it was simpler to motivate these professionals to gain knowledge, than to contribute to a research study.

Globally, the satisfaction with the e-learning course was positive (μ=87%; σ=12%; minimum=67%; maximum=100%), whereas 6 (30%) learners were totally satisfied with the e-learning course. Regarding global measures (Q14 and Q15), 17 (88%) were satisfied and considered that the e-learning course was successful (6 and 7 points in the 7-point Likert-type scale). Considering content quality (Q1-Q4), 17 (88%) learners think that this course fits their needs, is useful, sufficient, and up to date.

About the learning interface quality (Q5-Q9), 15 (76%) learners found the e-learning course user-friendly, stable, making it easy to find contents needed. Concerning personalization quality (Q10-Q13), 16 (84%) participants think the e-learning course enables them to control the learning progress, to learn the content needed, to choose what to learn and record their learning progress and performance.

Within these 20 participants, all claim the necessity of continuing medical education (CME) and 18 (90%) said that they would like to do CME by e-learning systems, whereas the other 2 (10%) said that maybe they will be interested in CME by e-learning.

The first main result of this study was the development of an e-learning course for the neonatal screening of congenital heart diseases. Using a free open-access tool such as Moodle and adapting the pedagogical contents of a well-established book for teaching this subject did not require advanced programming skills and led to an effective e-learning course. Given our successful results, it is expectable that future e-learning courses that specifically use best practices for multimedia learning should have an even stronger impact.

The second principal outcome was the statistically significant results obtained in the used metrics of pedagogical impact, with quite interesting proportions of learning with just a few hours of training. These results meet the general literature reviews that identified e-learning as superior to noneducation intervention [45], or as having similar effects to traditional learning [51], and more effective when combined with traditional learning [29] (b-learning). Although, in some contexts, e-learning may not be an alternative for the traditional face-to-face learning method, it can always offer a contribution and be a complement, and a useful adjunct to traditional education [51,52].

When comparing professions, we confirmed that the background knowledge about pediatric cardiology varies by professional group. By the end of the course nurses were the ones who learned more, although they did not score the highest final score. The heterogeneity of time spent in learning was also perceived. Internship medical students had the highest learning efficiency, dedicating much less time to the course than doctors and nurses. Predictably, doctors have lesser benefits from such an e-learning course.

Concerning specialty, the obstetricians were the ones who improved their knowledge the most, with the highest difference between the scores, the best improvement and efficiency, and were the most dedicated to the course. There were statistically significant differences in the improvement and in the course dedication between the different specialties, in this case confirming that more time dedicated to the course does translate into higher knowledge gain.

The third principal result was the encouraging assessment of the user satisfaction questionnaire. However, a third of the students who completed the course did not complete this questionnaire. This might be an important consideration for future studies, with regard to the interest and motivation of the participants to answer satisfaction questionnaires.

Regarding the adherence to this study, although the number of participants can appear low, we must understand the context in which this course was applied. From a total of around 80 people who are fully engaged in their daily health care routine and who voluntarily accepted to participate and complete an 8-lesson course in their free time, we consider that 30 is a successful result that reflects the need and enthusiasm generated by the proposed initiative.

Relevant limitations include the small sample size that affects the generalization of the results, and the limited implementation time, as the learners have their own agenda and priorities.

For the analysis, we had to use nonparametric tests because some of the variables did not follow a normal distribution and mainly because of the sample size. It would be definitely more interesting to use parametric methods to prove that there were significant differences before and after the course and the interactions between professions and specialty in a more robust manner, but for that possibility we would have to have a greater number of participants.

In addition, we have the impact of the exposure to the pretest and the end-of-chapter practice questions, which were identical to the ones that were used in the posttest. Although there was an expert validation of the test questions used before and after the course, the test was not assessed for its reliability or validity metrics.

The validity of the satisfaction assessment should be carefully considered, because we used a nonvalidated translation to a Portuguese version of the user satisfaction questionnaire, as it was not possible to find in the literature a Portuguese validated one for e-learning systems.

Another limitation of this work is the fact that the socioprofessional questionnaire was applied only at the end of the intervention. This situation limits our information about the description of our sample and other factors that could influence the adherence and learning process.

Concerning the evaluation strategy, we also faced the risk of a slight bias because we did not control if learners resorted to external sources in order to provide correct answers to the tests.

Globally, this study highlights the importance of training neonatologists and other health care professionals in the neonatal care units to screen for congenital heart disease. We consider the high rate of participation an important aspect of our study (78%), which reflects the great interest shown by these professionals to promote their professional skills. They took advantage of this learning opportunity, which confirms that these health care professionals are committed to responding to new challenges and evolving paradigms.

This study contributes to the Brazilian continuing training programs, as we did not find any similar course related to neonatal screening of congenital heart disease. It would be interesting to conduct additional assessments to demonstrate effective consolidation of knowledge gain. For future work, we also intend to assess the remaining levels of the Kirkpatrick framework, “behavior” and “results,” with respect to change in neonatal screening behavior and improved congenital heart disease detection. To do so, we plan to measure the number of telemedicine consultations conducted by the participants after the course and the number of congenital heart disease detected by them.

Our global results show that e-learning can provide statistically relevant knowledge gains in health care professionals in a neonatal screening context. We believe that this study underlines the importance of e-learning as a viable technology for training, especially in impoverished contexts. E-learning should be considered for continuing medical education in low- and middle-income countries, not only due to budget constraints, but also due to resource-constrained environments.