The Use of Open-Source Online Course Content for Training in Public Health Emergencies: Mixed Methods Case Study of a COVID-19 Course Series for Health Professionals

Background: The onset of the COVID-19 pandemic generated an urgent need for credible and actionable information to guide public health responses. The massive open-source online course (MOOC) format may be a valuable path for disseminating timely and widely accessible training for health professionals during public health crises; however, the reach and effectiveness of health worker–directed online courses during the pandemic remain largely unexplored. Objective: This study investigated the use of an open-source online course series designed to provide critical COVID-19 knowledge to frontline health workers and public health professionals globally. The study investigated how open-source online educational content can be optimized to support knowledge sharing among health professionals in public health emergencies, particularly in resource-limited contexts. Methods: The study examined global course enrollment patterns (N=2185) and performed in-depth interviews with a purposive subsample of health professionals enrolled in the course series (N=12) to investigate the sharing of online content in pandemic responses. Interviewed learners were from Ethiopia, India, Kenya, Liberia, Malawi, Rwanda, Thailand, Uganda, the United Arab Emirates, and the United States. Inductive analysis and constant comparative methods were used to systematically code data and identify key themes emerging from interview data. Results: The analysis


Background
As the COVID-19 pandemic began to spread rapidly in 2020, the global health community needed to act quickly to curb the spread of the highly infectious virus. To understand how to prevent and treat COVID-19 infections, communities worldwide turned to local health professionals for answers, creating an urgent need for trustworthy information to guide public health responses and to inform health care workers and the general public [1]. The rapidity of the response required for the public health emergency meant that health professionals had to rely on the limited training available. This study examined global enrollment in an open-source, self-paced, online library of COVID-19 courses developed in response to the pandemic. It aimed to better understand how the massive open-source online course (MOOC) format can be used to rapidly educate and support the sharing of knowledge among health professionals during health crises. In particular, the study aimed to understand how a single MOOC was taken up by learners across vastly different contexts and countries with differing languages, cultural backgrounds, pandemic experiences, technology access, and public health systems.
Given the novelty of the SARS-CoV-2 virus, as well as the new roles required to respond to and manage the pandemic, frontline health professionals and community health workers were in need of specific training and support [2][3][4][5]. Research has shown that frontline health care staff training in skills and knowledge related to disease outbreaks can lead to improvements in awareness of the disease, screening and reporting, and more proactive involvement in disease prevention and control efforts, as well as increased confidence to cope with managing disease outbreaks [4,6]. However, community health workers and other frontline health professionals, especially in resource-limited settings, are not often well supported and equipped with the training and resources needed to contain the spread of pandemics, such as COVID-19, despite the pivotal role that they play in the response [2][3][4]7]. While there is a wide variation in how community health workers and frontline health workers are defined and trained, and what tasks they are expected to carry out, in the midst of the disease outbreak, their roles expanded from the provision of ongoing public health services to also include pandemic mitigation measures, such as case detection, contact tracing, and triaging patients for care, amidst overwhelmed health systems [2,4].
Digital health content and tools have been used successfully to support the training and education of frontline and community-based health care workers [5,6,[8][9][10][11]. With the requirement for social distancing, as well as countrywide and regional lockdowns caused by the COVID-19 pandemic, reliance on a variety of digital education tools has grown [10]. The demand for information on the novel coronavirus, particularly at the start of the pandemic, was strong, as evidenced by substantial enrollment in COVID-19-focused online courses globally [12][13][14]. Yet, there is still a gap in the literature regarding the specific ways in which health professionals used online course content in their pandemic responses globally and the ways in which health care professionals can be better supported to use and share knowledge from online content in their health emergency responses [15].

Intervention
In April 2020, a consortium of 7 international organizations with expertise in health education, community health program implementation, global innovation, and digital technologies convened to develop a novel curricula and platform for community-based health workers and public health professionals responding to COVID-19 outbreaks in their communities. The consortium developed the COVID-19 Digital Classroom as a library of free, open-source, and mobile-friendly online courses. The series of resources consisted of 8 self-paced online courses on different topics related to COVID-19, including general information about the virus, prevention and protection, contact tracing, home-based care and isolation, community-based surveillance, risk communication and community engagement, mental health and wellness, and continuity of primary health care during COVID-19. Each course was estimated to take between 45 and 60 minutes to complete, with the exception of 1 course on contact tracing that was estimated to take over 90 minutes to complete.
The consortium designed the COVID-19 Digital Classroom specifically to support community-based health workers in lowand middle-income countries (LMICs). The courses were first developed in English (and later translated into additional languages) and included a variety of interactive activities, video animations, and infographics to overcome literacy limitations and language barriers. The video content animations were designed as standalone pieces of content that could be shared on social media or downloaded and shared via other channels, such as WhatsApp, to address limitations in bandwidth or technology access. The first course was launched in English in June 2020, with subsequent English-language courses added to the series through December 2020. The series was promoted to health workers and the general public by members of the consortium through webinars, emails, and social media channels.

Objectives
This study examined the use of this MOOC series in order to better understand how this type of digital education content can be optimized to reach and support knowledge sharing among community-based health professionals in public health emergencies, particularly in resource-limited contexts. We examined global enrollment patterns to explore demand, ability to access online content, and sharing of course content among learners. Through in-depth interviews, we investigated the ways in which course content was used, adapted, and shared among health professionals as part of the COVID-19 pandemic response.

Data Sources
The study leverages the following 2 sources of data: (1) course enrollment data between June 2020 and July 2021 (N=2185) and (2) in-depth interviews with a purposive sample of health professionals, with a focus on enrollees who shared course content as part of community-based pandemic responses (N=12).

Enrollment Data
Descriptive enrollment data provided a framework for understanding demand and ability to access the online courses. These data also served as the sampling frame for constructing a purposive qualitative sample. We examined data collected through a registration survey administered to all enrollees of the course series, including information on learners' country of residence, gender, institutional affiliation, profession, and type of involvement in the COVID-19 response, and how learners heard about the course series. We focused on the first year of enrollment in the English-language version of the course series that was first launched between June 2020 and July 2021 to better understand how content was used in the early pandemic response.

Interview Data
For in-depth interviews, the research team selected a purposive sample of learners (N=12) across health professions from different regions globally. Because the course series was designed for community-based health workers, sampling focused on identifying health professionals with experience using and sharing course content as part of community-based pandemic responses.
The study considered the following learners for recruitment: (1) learners who indicated that they had shared course content with others in their network in a voluntary follow-up course satisfaction survey (administered by the consortium in December 2020; N=112); (2) learners in a community-based health worker role or in a supervisory role in a position to share information with community-based health workers (ie, doctors, nurses, health worker trainers/supervisors, and technical assistance providers) and at an organization with more than one enrollee in the course series; and (3) learners holding an educator role at a higher education institution with more than one enrollee in the course series. Learners who indicated that they did not consent to be contacted further in the follow-up course satisfaction survey were excluded from recruitment.
A total of 119 learners met the purposive sampling criteria and were recruited to participate via email in the study. Learners were sent an introductory recruitment email, and those who did not respond to the initial email were sent several follow-up email requests. Fourteen learners responded with willingness to participate in an in-depth interview (11.8% response rate). The research team was able to schedule in-depth interviews with a sample of 12 of these learners and made efforts to ensure representation across geographic regions and from LMICs. No additional recruitment was deemed necessary as the research team determined saturation was achieved.
The 12 in-depth interviews were conducted one-on-one in English via videoconference by 2 investigators (NAS and NJ) using a semistructured interview guide. The interviewers asked learners about their experiences with the curriculum and their roles in using, adapting, and disseminating the curriculum. The interviews were audio recorded and transcribed. Interviews lasted between 20 and 57 minutes, with a mean duration of 38 minutes.

Data Analysis
Enrollment data were summarized using descriptive statistics (response rate, mean, and SD). Statistical analyses were performed using Stata SE version 15 (StataCorp). The interview transcripts were analyzed through thematic coding using Dedoose (SocioCultural Research Consultants). Inductive analysis and constant comparative methods were used to systematically code data and identify key themes emerging from interview data. Each interview transcript was independently coded by a member of the team (NAS, NJ, and JSJ) and then independently reviewed by a second coder. None of the transcripts were coded by investigators who conducted the interview. The research team met multiple times to confer and calibrate the coding interpretation and to further refine and recalibrate coding schemes. The analysis was concluded with a final pass of the transcripts by 2 coders.

Ethics Approval
Informed consent was obtained from all interview participants. Approval for all aspects of this study, including for consent, outreach, data collection, interviewing, and data analysis, was obtained from the Stanford University School of Medicine Institutional Review Board (protocol number: 61266).

Enrollment Patterns
To investigate the types of learners seeking out open-source online education in the first year of the COVID-19 pandemic, we examined the characteristics of the 2185 learners who enrolled in at least one of the courses in the course series between June 2020 and July 2021. As shown in Table 2, while enrollees were distributed globally across 104 countries, a majority of learners were from North America (1551/2185, 71.0%) and primarily the United States (70.3%), followed by Sub-Saharan Africa (315/2185, 14.4%). Among all learners, 12.2% (266/2185) were from lower-middle-income countries and 5.4% (118/2185) were from low-income countries, as classified by the World Bank [16]. The preferred language across learners was predominantly English (1664/2185, 76.2%), which was not unexpected given the focus on enrollment in the English-language version of the course series, which was initially the only version available and the only one examined in this data set. Nevertheless, 3.8% (83/2185) of all enrollees and 7.8% (30/384) of enrollees in LMICs indicated preference for a language other than English.
As shown in Table 3, enrollment patterns indicated that learners heard about the course series through personal and professional networks. Overall, 30.1% (658/2185) of learners heard about the MOOC from their employers, while another 8.5% (186/2185) heard about the MOOC from friends or colleagues. The proportion hearing about the series from friends or colleagues was higher in LMICs at 21.6% (83/384). In LMICs, the proportion hearing about the MOOC from direct promotion by the consortium that developed the course series was also higher at 44.3% (170/384).
At registration, learners were given a chance to identify their specific organization affiliation, and 1504 learners identified as being affiliated with 779 unique organizations, of which 110 organizations had more than two learners. In LMICs, 303 learners identified as being affiliated with 221 unique organizations, of which 35 organizations had more than two learners. The average number of learners per organization with multiple learners was lower in LMICs (4.9 learners) than in the overall sample (7.6 learners). This analysis included only learners identified through enrollment registration analytics.

In-Depth Interviews
Three major themes emerged in the thematic analysis. First, the COVID-19 Digital Classroom helped fill a critical gap in trustworthy COVID-19 training information globally, motivating learners to share the course series within their personal and professional networks. Second, although the comprehensive nature of the MOOC provided valuable information, the density of the course series made it difficult to navigate and use with all audiences, especially frontline and community-based health workers, as well as with the general public. Third, while participants shared the knowledge they gained from the courses, the vast majority of participants did not attempt to adapt the courses (eg, make content changes, such as add translations into local languages or add local contextual information, or make technical changes, such as disseminate the courses via different modalities including SMS text messages) to share within their communities, despite their expressed need, due to technological and logistical barriers.

Theme 1
Interviews revealed that in the early stages of the pandemic, there was a major need for trustworthy information about COVID-19 alongside an expectation that health professionals fill those gaps within communities. One community health worker from Kenya stated: Respondents shared that they sought out information from a variety of online sources to rapidly access new knowledge to respond to the COVID-19 pandemic. In addition to the COVID-19 Digital Classroom course series, respondents sought information from the World Health Organization, regional health organizations, the Centers for Disease Control in various countries and regions, their countries' ministries and departments of health, media sources, and higher education institutions, as illustrated in the following statement by a health professor in Ethiopia:

I believe that because when you give advice, you need to be updated with what is new and what is going on. I used to be informed about any updates [from] the Digital Classroom and the different sources, definitely the WHO website. I followed the news as well [for] continuous updates. [#119]
Several of the interview participants reflected on their responsibility as health care professionals to be informed in order to support their community. One community health worker from the United States stated: Everybody just didn't know what to do, and I just wanted to be able to position myself and be able to sign up for the courses so I can be well versed and well-rounded and know how to assist those that were in need during this pandemic. [#615] The personal and professional networks of the health care workers enabled access to the MOOC. Many of the respondents learned about the course series through recommendations from employers or colleagues. Sharing of the COVID-19 Digital Classroom itself (ie, recommending that others within the network enroll in the courses) occurred primarily through several informal channels, including online messages and conversations, or via social media. Interviewed learners stated that they recommended the courses by name, shared the link to the courses through their networks, and often enrolled in the courses based on recommendations. In discussing this process, a learner from Ethiopia shared: Whenever someone found a piece or documents we would share and see how we could bring a resource to share in our setup… There are two of my colleagues that did it [completed the course] that I know of... because…. we keep on asking each other for the latest information, and so when I got the link I also forwarded it to several colleagues. [#782] Interviewees indicated that the course series filled a knowledge gap in basic information and built confidence in their understanding of the pandemic. The interviewed learners indicated that they incorporated the knowledge gained from the courses into policies and protocols for working with patients, and the courses served as ongoing resources for their pandemic response. A doctor from Kenya shared: Because you're a doctor you're expected to understand something. You can't tell your patients 'I don't know what it is. It's a new disease.' That's not right. So at least I had to be able to get some information on how people can protect themselves. In fact, out of this information I was able to learn a lot. It helped us develop a process for our organization, a process for people with NCDs who are, for example, at a higher risk, with precautions they need to take. [#891] The interviewed community health workers and frontline health workers also reported that they added the presented information into their community workshops, and the doctors, professors, and health worker trainers incorporated the information into their trainings and classes for their health care students and community health workers. One community health worker described how having the knowledge and resources built her confidence in being able to do her work in her community. In referring to a workshop she led in her community, she stated "I hate when I stand before people and do not have information… the video is very good" [#158]. This learner went on to share that the course content was also used as a tool for communication with her patients: I even remember at one particular time I played one of the videos with the class members to shed light on our hand hygiene and skills that I actually picked from the lessons, so I was even more empowered than before, because I was able to describe [the process]. [#158]

Theme 2
The second theme that emerged from the interviews was that while the COVID-19 Digital Classroom course series and the knowledge sharing that occurred through the respondents' networks enabled them to access critical COVID-19 information, the density of the course series made the consumption and sharing of content more challenging, especially with community health worker audiences and community members. In discussing the challenge of using the COVID-19 Digital Classroom course series with community health workers, a health professional from Malawi said: With community health workers, we have two groups. We have clinicians and nurses, who will be able to understand the course. Then we have the assistants, who are connected to the community in remote areas and provide basic health care, like vaccines, bring [people] to the clinics, [and] family planning. Most clinicians and nurses, they can be able to understand this, the courses, but the assistants may need a little bit more assistance for them to understand the courses. [#782] Many respondents indicated that there remained a large unmet need for resources on COVID-19 that could be more easily shared and used within their local communities. The health care professionals interviewed indicated that they needed resources to communicate with the general public, as well as resources to support the learning of health care workers they needed to train. Respondents expressed a need for additional scaffolding support to help navigate the dense information offered in the COVID-19 Digital Classroom course series through thoughtful platform design. They indicated a need for content structuring and platform functionality that could better facilitate the sharing of recommendations and encourage instructor, learner, and peer interactions, which would also help navigate the learning process. A doctor in Rwanda stated: I think it became really clear really quickly that if we can't get the protection, the knowledge, the tools to communities we're not going to fight this virus, and we also saw that pre-existing trust or mistrust in the health system, made a huge difference in whether or not people were able to quickly adapt and adopt the recommendations that were being sent out around the world. So, for us, we saw locally, our role changed. [#560] As they embraced the new responsibility to share COVID-19 information rapidly in their communities, respondents shared how the ability to adapt existing online content, including the COVID-19 Digital Classroom, and easily share key recommendations would allow for the incorporation of local contextualization that could make a difference in how information is perceived in communities.

Theme 3
The final theme to emerge from the qualitative analysis was that despite participants' interest in having the course series adapted to meet their local needs, due to technological and logistical barriers, the vast majority of participants did not attempt large-scale content adaptations of the course series (eg, adding translations into local languages, adding local contextual information on COVID-19 responses, or adding local contextual cultural information). Only 1 large international organization, who partnered directly with the consortium, led full technological adaptation of the curriculum into an SMS text message format to expand access to course content. The international organization partnered with the consortium and a higher education institution to provide the course materials to those organizations who partnered with their LMIC offices and who only had access to basic phones.
Interviewed learners indicated they wanted adaptations that would involve adding, removing, and editing components to make the content relevant to their local context. They also expressed the need for translations of course materials in local languages. For example, a doctor in Rwanda desired an adapted version with key points, and this doctor stated "…the key points, and if you want to dive in deeper, here's the way to do it, but otherwise here [are] the key takeaways you need to know to make educated decisions for your health and your patients." [#560]. The doctor went on to suggest that the MOOC could be further adapted for different audiences, including 1 version for nurses, 1 for community health workers, and 1 for community members. Respondents indicated that having the flexibility to be able to add or remove content to reflect different contextual needs would support a meaningful adaptation that would localize recommendations to better educate their communities. Some respondents suggested that certain content was not relevant for local contexts and could be shortened or was relevant but needed more explanation. For example, a health educator shared: There is a desire and a need for mental health programming and training, but it was too high level for the community health workers to understand… it wasn't being effective.
[#458] Different audiences were also unable to use or access the digital platform due to technological barriers. However, offline and low-tech needs and solutions vary by region and audience. Learners indicated that they were very interested in having the ability to adapt the courses to offline or low-tech options in order to share materials with their communities. In discussing the challenges associated with the technology needs of different audiences, especially community health workers, a doctor in Rwanda stated: Learners indicated that having the flexibility and guidance on how to fully adapt courses or parts of courses to different offline and low-tech delivery modalities would help support the spread of the content to a broader range of learners, especially in remote and low-resource settings.

Discussion
To investigate the potential of MOOCs as a strategy to rapidly educate health professionals across vastly different contexts during public emergencies, we examined how health professionals globally used a newly developed open-source online course series, the COVID-19 Digital Classroom, in their local responses during the first year of the COVID-19 pandemic. Enrollment data showed that health professionals across 104 countries who were engaged in a range of pandemic response activities (including contact tracing, surveillance, testing, treatment, and risk communication and community engagement) sought out and used the online content in their work. This was particularly the case in LMICs where nearly 90% of enrollees were engaged in the COVID-19 response.
In-depth interviews revealed that the COVID-19 Digital Classroom included content areas and features that were useful for learners and institutions across a variety of contexts globally. Interviewees leveraged their networks with other health professionals to share content knowledge from the MOOC to fill gaps in knowledge needed to respond to the unfolding emergency. The interviewees also reported that they had a need for trustworthy health information to help them implement health education training and information initiatives to reach those in their networks more broadly.
Analysis of course enrollment patterns supports the qualitative finding that health professionals shared information within their personal and professional networks, recommending others to enroll in the MOOC. With nearly a third of enrollees reporting that they learned of the course series through an employer recommendation, along with clusters of learners observed at specific organizations, the enrollment data suggest that organizations, including NGOs, government agencies, and health care providers, were using the MOOC to train employees.
The use of the COVID-19 Digital Classroom as an education tool at a range of academic institutions and universities suggests that higher education institutions globally also sought out online course materials about COVID-19 for their health students. These institutions were predominantly located in North America, but included a range of university types, including nursing schools, public health schools, and community colleges.
Despite the intention of the course developers to create an online content platform specifically to support pandemic responses in LMICs, use of the MOOC, as reflected by enrollment analytics, was low relative to that in high-and middle-income countries. A high proportion of enrollees in LMICs learned about the MOOC from direct consortium promotion, suggesting a need