This study is a part of the Advancing Co-creation of RLOs to Digitise Healthcare Curriculum (ACoRD) project, which was funded by the European Union Erasmus+ project [17]. It is a capacity building project involving Universiti Malaya, Universiti Putra Malaysia, Taylor’s University, University of Nottingham, Karolinska Institutet, and University of Stavanger. The ACoRD project aimed to introduce innovative digital pedagogy methods by developing, evaluating, and disseminating high-quality, peer-reviewed RLOs that benefit health care and biomedical science learners in Malaysia. RLOs were developed, implemented, and evaluated over 3 phases (Figure 1). The implementation strategies of RLOs were developed and performed in both the development (preimplementation phase) and implementation phases.

RLOs were developed following the ASPIRE (Aim, Storyboarding, Populating, Implementing, Release, and Evaluation) framework [18]. The development phase began by establishing a coalition between international institutions within the project. We evaluated the process and challenges of transnational partnerships and knowledge transfer by using a qualitative study, providing insights on guiding effective partnership in e-learning development in the future [17]. This was followed by stakeholder engagement at 3 Malaysian institutions, involving higher education institution management, medical education units, respective departments, and curriculum coordinators. To identify the gaps and topics for RLO development, the ACoRD team reviewed the existing curricula and determined how RLOs can be integrated into the existing topics, learning activities, and teaching delivery methods in the curricula. We also conducted a needs assessment by using a Delphi survey to compare the differences between educators and learners in prioritizing topics for RLO development [19]. After determining the topics and scope of 23 RLOs, we engaged the learners in RLO cocreation, especially in the storyboard development step. We evaluated learners’ knowledge, confidence, and satisfaction of the storyboarding session by using a pre-post survey and explored their perception and experience of the cocreation process qualitatively [20]. We provided RLO specification writing training to the educators who subsequently wrote the content for RLOs. Once RLO specifications were finalized, a content review was conducted before moving on to technical development. A small pilot test was conducted with student champions on RLO usability before full implementation.

The 23 RLOs (Table 1) were made available through the open-access ACoRD repository website [21], Health e-Learning and Media-Open under the University of Nottingham [22], Multimedia Educational Resource for Learning and Online Teaching [23], and the institutional online learning portal and curriculum guidebook.

The focus was on the actual integration of RLO resources into the health sciences curricula. At Universiti Malaya, RLOs were incorporated into various block postings for medicine and an undergraduate semester course for nursing. In Universiti Putra Malaysia, RLOs were used in a professionalism module block, and at Taylor’s University, they were embedded as part of semester courses in pharmacy and biomedical science. Various strategies were implemented to improve the usage of RLOs, including identifying early adopters (department and educators) to use RLOs in teaching, integrating RLOs into the existing curricula, sending reminders to educators, receiving feedback from students, establishing an effective dissemination team, monitoring RLO performance and feedback to educators, and periodic re-examining of the implementation strategies.

This study was performed across 3 Malaysian institutions (Universiti Malaya, Universiti Putra Malaysia, and Taylor’s University). Participants in this evaluation were users who used RLOs during the implementation period from May 1, 2020, to February 24, 2022. We used a universal sampling method for data collection.

We evaluated the implementation of RLOs by using the RE-AIM framework (Figure 2). The reach domain is defined as the access of RLOs by the users. We assessed reach by measuring the absolute number of RLO users, location of access, types of devices used by the users, and how users found out about our RLOs.

For the effectiveness domain, we applied the 4-level Kirkpatrick model: Reaction, Learning, Behavior, and Results [24]. It is a model that is commonly used in medical education to evaluate the effectiveness of a training program [25]. Reaction was assessed by gauging learners’ perceived helpfulness of RLOs, and learning was measured using pre-RLO and post-RLO knowledge and confidence scores. Behavior measures whether learners are applying the newly acquired knowledge from RLOs in clinical practice, while results measure the ultimate impact of training using RLOs, which includes impact at an organizational level. We did not measure the behavior and results domains in this study due to limitations in the study design.

Adoption refers to the number of institutions that agreed to use RLOs in teaching and learning activities. We captured information on the institutions, departments, and existing modules in which RLOs were implemented. The implementation domain refers to users’ fidelity in using RLOs for their learning. We measured the number of users who completed RLOs and the number of nonbounced users. Nonbounced users are those who are engaged with the site either by clicking on links, visiting multiple pages, or triggering events that indicate interaction, while bounced users are those who visited the RLO web page but only viewed a single page without interacting further before exiting.

The maintenance domain refers to the extent to which the sustained use of RLOs becomes institutionalized or part of the routine teaching and learning practice. We measured the number of users who visited the RLO web page 4 months after the project ended.

We used 2 instruments to capture the RLO implementation outcomes: questionnaires and Google Analytics (Table 2). The questionnaires were designed to gather user feedback and usage patterns. We used 2 types of questionnaires: (1) a web-based questionnaire was administered at the completion of each RLO to assess users’ perception of its usefulness (using a 4-point Likert scale: 1-very unhelpful, 2-unhelpful, 3-helpful, and 4-very helpful) and how they discovered RLOs, and (2) a confidence Likert scale and RLO-specific knowledge questionnaire were administered before and immediately after RLO usage. For the knowledge score, RLOs 16, 17, and 18 (using nicotine gum, nicotine patches, and varenicline for smoking cessation) were assessed using 1 set of knowledge questions, while RLOs 20 and 21 (DNA repair and replication) were assessed using another set of knowledge questions related to specific RLO topics. We used GAMM1-Google Analytics in Universiti Malaya and Universiti Putra Malaysia and Moodle monitoring in Taylor’s University to capture the patterns of access (location, number of users, number of completing users).

For the questionnaire items, categorical data were reported descriptively using proportion and percentage. The mean difference in the pre-RLO and post-RLO scores on knowledge and confidence were analyzed using Mann-Whitney U test, as the data were skewed. Statistical analysis was considered significant when P<.05. Google Analytics data were reported descriptively using proportion and percentage. All the data were analyzed using SPSS (version 27; IBM Corp) and Microsoft Excel.

Ethics approval was granted by the Universiti Malaya Research Ethics Committee (reference UM.TNC2/UMREC-997). All participants provided their written consent. All participant information has been anonymized and deidentified. There was no financial compensation provided to the participants.

From the launch of the first RLO on May 1, 2020, until February 24, 2022, Google Analytics recorded a cumulative 7622 users from 48 countries (Figure 3). We have reported the results according to the RE-AIM framework.

Among the 7622 global users, the majority (n=6817, 89.4%) were from Malaysia, while 10.6% (n=805) were from other countries. Desktop computers were the predominant devices used to access RLOs (6045/7622, 79.3%); 20.7% (1577/7622) used portable devices such as mobile phones or tablets. The number of users for each RLO varied from 92 to 1014 (Table 3). The RLOs attracting the highest number of users were RLO 7 (How to Conduct Literature Research, n=1014), RLO 1 (Prescription Writing: Back to Basics, n=530), and RLO 17 (Using Nicotine Patches for Smoking Cessation, n=511). RLOs with fewer than 100 users were RLO 10 (Doctor-Patient Relationship) and RLO 23 (Nervous Regulation of the Heart).

Out of 2118 respondents to the questionnaire, the majority used RLOs because they were part of the course learning resource (n=1901, 89.8%), while others received a recommendation from peers, colleagues, or lecturers (n=87, 8.8%). A small proportion of the respondents (30/2118, 1.5%) found RLOs through the Health e-Learning and Media open database, open educational catalogs, or general internet searches.

Most users rated RLOs as very helpful (1452/2071, 70.1%) and helpful (601/2071, 29%). We evaluated the preknowledge and postknowledge scores across all RLOs. The mean preknowledge score ranged from 0.33 to 9.6, while the postknowledge score ranged from 5.20 to 9.85. A significant improvement in the knowledge scores was reported for all RLOs after their use (P<.05) (Figure 4), except for RLO 4 and RLO 6, which had high mean baseline preknowledge scores of 6.61 and 8.80, respectively (Table S1 in Multimedia Appendix 1). Additionally, we assessed the confidence scores for 17 RLOs (excluding 5 RLOs, ie, RLOs 16-21 due to missing data) and found a significant improvement in the mean confidence scores after utilization for all 17 RLOs (P<.05) (Table S2 in Multimedia Appendix 1).

All RLOs have been adopted by the 3 Malaysian universities (Universiti Malaya, Universiti Putra Malaysia, and Taylor’s University). At Universiti Malaya, the departments of primary care medicine, medicine (geriatrics), pediatrics, and nursing have incorporated RLOs into their teaching and learning curricula. Universiti Putra Malaysia has used RLOs within its professional development module, while Taylor’s University has implemented them in the pharmacy and biomedical undergraduate courses.

The completion rates for RLOs varied widely, with percentages ranging from 5.6% (10/179) to 85% (78/92) as shown in Table 4. Only RLO 8 (Confidentiality), RLO 15 (Social Media Professionalism), and RLO 23 (Nervous Regulation of the Heart) had a completion rate exceeding 50%. The proportion of nonbounced users, defined as users who viewed more than one page, ranged from 16.3% (165/1014) to 88.5% (370/418). Notably, data on nonbounced users for RLOs 8-15 from Universiti Putra Malaysia were unavailable due to the absence of a tracking function on each RLO page in Google Analytics (because of technical errors).

After the ACoRD project ended on February 24, 2022, a total of 2107 users continued to access RLOs in the subsequent 4 months, ranging from 15 to 187 accesses per RLO (Figure 5).

This study reports the successful implementation of RLOs in health care education across 3 Malaysian higher educational institutions by using the RE-AIM framework. Our findings offer a broad perspective on how the impact and outcomes of e-learning objects can be measured and evaluated systematically to inform e-learning implementation strategies.

Our study shows that RLOs were accessed by a more diverse group of learners than initially anticipated, extending beyond the Malaysian institutions, where RLOs were intended to be used. Leveraging on the unique characteristics of reusability in RLOs [26], we adopted an open educational resource approach in their deployment. This strategy significantly broadened the reach of RLOs, making them available to learners across different disciplines. Consequently, RLOs addressing more general subjects such as literature review, prescription writing, and smoking cessation attracted a larger user base compared to those focusing on discipline-specific topics.

Throughout the ACoRD project, pragmatic implementation and dissemination strategies were employed to engage a broader audience. These strategies encompassed identifying early adopters and educator champions, sending of reminders to educators about incorporating RLOs in their teaching, monitoring of RLO performance at regular intervals, and providing feedback to educators. A study on mobile health e-learning courses highlighted the advantages of iterative feedback among developers, early adopters, and end users, which aid in the refinement of existing e-learning implementation strategies and the development of new ones [27]. The ACoRD project featured a dedicated dissemination team that organized and executed activities to promote RLOs through conferences and engagement workshops, extending beyond our institutions. Establishing collaborative partnerships with other higher educational institutions is an important strategy to boost visibility and ensure the long-term use of e-learning resources [28].

In our study, users perceived RLOs as beneficial for their learning, with most RLOs demonstrating an improvement in users’ knowledge and confidence levels. These findings are aligned with previous studies that have reported improvements in knowledge and understanding following RLO usage [16,29,30]. However, evaluating the effectiveness of e-learning resources remains a challenge. Most studies measured learners’ reactions and learnings such as knowledge, usefulness, confidence, satisfaction, and motivation [31]. Some studies examining the effectiveness of RLOs demonstrated behavior change in relation to prescribing behavior and hearing aid use [32,33]. Measuring higher levels of learning such as behavior change and impact of learning is complex due to the time and cost involved [34]. We propose that the evaluation of RLO effectiveness should go beyond individual learning outcomes to assess their impact within the wider teaching and learning ecosystem. This could include evaluating RLOs’ effectiveness from educators’ perspectives and examining how RLOs contribute to student empowerment in self-directed learning [6].

In the adoption domain of RE-AIM, RLOs developed by the ACoRD project were adopted by all 3 Malaysian institutions. Our study underscores the significance of planning and initiating implementation strategies during the development phase (preimplementation) with stakeholders and institutional engagement. Engaging institutional faculty, students, and stakeholders is critical for the successful adoption of e-learning [35]. The cocreation process and involvement of stakeholders throughout the ASPIRE process facilitates the feeling of ownership of the materials produced, thereby leading to use and reuse. To ensure the widespread adoption of e-learning innovation, it is crucial to engage with and consider the perspectives of diverse stakeholders, as the implementation of e-learning necessitates changes in teaching, learning, management, and infrastructure within the institution [36].

In our study, although RLOs reached a significant number of users, the proportion of users who completed RLOs (5.6%‐84.4%) and nonbounced users (16.3%‐81.5%) was relatively low for most RLOs. Chen et al [37] reported a similar finding, with a low nonbounced rate of 40% for an undergraduate online course. The wide discrepancy in the completion and nonbounced rates among RLOs may be attributed to the different levels of learning across various learner groups. RLOs integrated into the curriculum as teaching and learning materials achieved a completion rate greater than 70% (RLO 8 and RLO 23), suggesting that learners are more invested in completing these RLOs. Conversely, RLO 7 attracted the highest number of users (n=1014); yet, its completion rate was only 7.4% (75/1014). Our findings show that Google Analytics is beneficial for examining the fidelity of RLO usage, as it provides insights into learners’ behaviors. Assessing the fidelity of RLO usage is crucial for educators and institutions to gauge the effectiveness of RLOs in teaching and learning [38].

Users continued to access RLOs after the conclusion of the ACoRD project, which indicates the sustainability of RLOs. The main strategies that we identified were the integration of RLOs into the existing health care curriculum and developing RLOs by using a robust cocreation methodology. Identifying the learning needs of teachers and learners prior to the development of RLOs facilitated their integration into the curriculum [19]. We also implemented the educator-as-champion strategy, which involves educators in content development and subsequently utilizing RLOs in their teaching. e-Learning champions among academicians in higher educational institutions are the key players in fostering the integration of technology into teaching and learning [39].

Our study has several limitations. Since each RLO was developed and launched at different times, discrepancies in the implementation periods made data interpretation challenging. We used Google Analytics to capture the reach of our RLOs. However, we could not confirm whether access came from unique users or unique IP addresses. There is a possibility that some users could have accessed RLOs multiple times by using different IP addresses. As this study employs a pragmatic approach to capture evaluation outcomes in real-time, missing data were inevitable; however, it did not substantially impact the overall findings. Strategies were implemented pragmatically throughout the project, with practical consideration and changes made to address implementation issues; as a result, we were unable to identify and measure the effectiveness of each implementation strategy.

We employed the RE-AIM framework to systematically evaluate the implementation success of e-learning resources, identifying gaps and strategies for improvement. This study highlights the development, implementation process, and implementation outcome indicators of open-access RLOs in health care education. To enhance future e-learning implementation efforts, we recommend incorporating the RE-AIM framework into outcome evaluations to provide a more comprehensive evaluation of e-learning implementation.