An RE design [53] was used to identify CMO patterns under routine curricular settings. Accordingly, no randomization or blinding was applied. The evaluation was conducted at a German medical school (6-year curriculum). Within the faculty-wide digitalization initiative WueMedica, a general mLearning module was developed and implemented across multiple disciplines. In this study, we focused on its use in the fifth-semester microbiology course, comprising bacteriology, parasitology, mycology, and virology. The subject-specific component of the module was delivered via the eSquirrel app (eSquirrel GmbH). In addition to a voluntary lecture series and mandatory weekly laboratory sessions, mLearning was offered as a supplementary learning resource. Teaching staff delivered the standard curriculum independently, and no direct contact occurred between the study team and students.

The intervention consisted of a voluntary app-based course designed to support examination preparation alongside the curriculum. Content was developed by trained student assistants in collaboration with faculty experts and was based on official course materials, the Nationaler Kompetenzbasierter Lernzielkatalog Medizin (the German national competency-based learning objectives catalog) [58], and standard textbooks. The app-based course was organized into chapters with interactive tasks (such as multiple-choice, matching, and fill-in-the-blank formats) and supplemented by images and links to relevant textbook chapters. Gamification features such as point scoring, a leaderboard, push notifications, and an adaptive spaced-repetition algorithm were included. The course comprised 88 chapters in 2023 and was expanded to 138 in 2024.

The study was approved by the local independent review board (ref 20230217 01 and ref 202580-ka) and conducted in accordance with the European Union General Data Protection Regulation. All study data were stored on secure institutional servers of the University Hospital. The app complied with General Data Protection Regulation requirements and supported cross-platform and offline use.

Students were informed about the study during an in-person session at the beginning of the semester and provided written informed consent. Consent included permission to link the pseudonymized student identifiers with examination results and study data temporarily. Access to the mLearning app was provided via the Moodle-based institutional learning management system (Moodle Pty Ltd, West Perth, Australia) and required an official university email address. Student IDs were used solely for data linkage purposes and removed from the merged dataset prior to analysis. Data from students who used the app but did not consent to participation were excluded. It was determined in 2023 that multiple registrations were technically possible, but these were effectively prevented in 2024 through email verification and ID matching. Participation was voluntary and had no impact on students’ courses, assessments, or academic progression.

The evaluation spanned 2 consecutive summer semesters (2023 and 2024). Eligible participants were students enrolled in the microbiology course and taking the oral examination at the end of the course for the first time. Repeat examinees or students without formal registration were excluded.

Data were collected from four sources:

In 2023, 117 students qualified for participation. On the basis of an expected 80% (94/117) participation rate, the study was powered to detect a moderate effect size (Cohen d=0.58) at α=.05 with 80% power.

Standardized usage and demographic variables (z-scores, calculated as z=(X–M)/SD, X as individual values, means, and SDs) were entered into a 2-step cluster analysis (log-likelihood distance and the Bayesian information criterion) to identify user profiles. Students were categorized according to app usage status (nonusers vs app users). App users were further grouped into usage clusters based on engagement patterns. All groups were compared with respect to survey responses and performance outcomes.

Survey data were analyzed using principal component analysis with pairwise deletion for missing values. Components with eigenvalues >1 were extracted and subjected to oblique (Oblimin) rotation. Reliability was assessed with Cronbach α, and component scores were linearly rescaled to a 1-to-5 scale to ease interpretation. Variance contributions of the final scales were approximated using regression-based scores from the unrotated solution.

For the 2024 data, an analysis of covariance was conducted to test for any potential intervention effect. The written M1 examination score as a percentage was used as the academic baseline, and the analysis examined whether oral microbiology examination results deviated significantly from this baseline. Data from 2023 were not included due to methodological limitations. These included incomplete coverage of exam-relevant chapters in the app and the inability to identify app users clearly.

All analyses were conducted using SPSS Statistics (version 29.0.0.0; IBM Corp) at a 95% significance level, applying Welch t tests (2-tailed) for unequal variances [63,64] and standard t tests when variances were equal, with 2-sided P values reported unless otherwise specified.

Reporting followed the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth V.1.6.1; Checklist 1) checklist for mobile health interventions [65] and the RAMESES-II (Realist and Meta-narrative Evidence Syntheses: Evolving Standards-2; Checklist 2) standards for REs [66].

There were 245 eligible students, of whom 220 (89.8%) participated. Participant recruitment and data availability are summarized in Figure 1. Participants were categorized according to user status, with 110 app users and 110 nonusers. The survey produced 74 valid responses out of 220 participants, corresponding to an overall response rate of 33.6%. In total, 53 participants (24.1%) both used the app and completed the survey. Demographic characteristics were representative of the student population (study participants: 70% female; mean age 23.1 years, SD 3.1 years and overall study population: 69% female; mean age 23.9 years, SD 3.3 years). No significant differences were observed between semesters. Baseline characteristics of participants are provided in Multimedia Appendix 1.

Cluster analysis (average silhouette coefficient=0.40) of the 110 app users identified 3 usage clusters that differed primarily in the timing and intensity of engagement throughout the semester. Figure 2 illustrates the corresponding longitudinal z-standardized engagement trajectories, with user performance defined as successfully completed in-app tasks within chapters.

Cluster analysis revealed three distinct usage patterns:

No significant differences in age or gender were observed across clusters.

The bar charts illustrate changes in standardized performance (z scores) over the course of the semester. Positive values indicate above-average gains relative to all users; negative values indicate below-average gains. For example, between days 39 and 37 before the examination, cluster 3 users achieved gains around 0.75 SD above the mean.

For the 2024 cohort, end-of-course oral examination results were available for 95 participants; 15 students (13.6%) were excused for medical reasons. Grades ranged from 1 (excellent) to 5 (failed; Table 1). App users (n=64) achieved better examination results than nonusers (n=46; mean grade 2.3, SD 1.1 vs 2.8, SD 1.3; 1-sided Welch t_63,0=1.90; P=.03). Among app users, cluster 2 (n=20) presented the best performance (mean 2.1, SD 0.9; P=.008 vs nonusers), whereas cluster 1 (n=34; mean 2.4, SD 1.1; P=.09) and cluster 3 (n=10; mean 2.5, SD 1.3; P=.26) did not differ significantly from nonusers. A higher proportion of nonusers than users were excused from the examination (24% vs 6%). No statistically significant differences were observed between the 3 app-user clusters.

Sampling adequacy was acceptable (Kaiser-Meyer-Olkin test=0.750; Bartlett test P<.001). Principal component analysis initially identified 6 scales explaining 74.3% of the variance. Scales 3 to 5 were excluded due to low internal consistency (Cronbach α=0.46-0.09) and content overlap. Scale 1 was divided into learning outcome and user experience in line with the theoretical framework [67,68], with 2 items reassigned to enhance conceptual fit. Scale 2 was retained without modification. Three final scales were identified: learning outcome (perceived gains in learning efficiency, comprehension, and long-term retention), user experience (usability, enjoyment, motivational impact, feedback on progress, curricular relevance, and willingness to recommend), and study motivation (general motivation, avoidance of superficial learning, openness to engaging topics, and preference for self-testing) with item loadings depicted in Table 2. Together, the 3 scales required no further rotation and explained 50.2% of the total variance (Table 3).

Detailed results according to usage status and usage clusters are reported in Multimedia Appendix 2.

We compared app users and nonusers at both the scale and item level. We observed no statistically significant differences at the scale level. At the item level, however, app users consistently rated more favorably, most notably with regard to enjoyment.

Cluster 1 was not found to have any advantage over nonusers on any scale. At the item level, enjoyment was the only aspect rated higher than by nonusers.

Cluster 2 exhibited the most favorable self-reported learning experience. Compared with nonusers, this cluster scored significantly higher on learning outcome and user experience scales. Furthermore, learning outcome scores were also higher than those of cluster 1. No differences were observed for study motivation. At the item level, cluster 2 exhibited consistently higher ratings across multiple items related to perceived learning gains, enjoyment, feedback on learning progress, satisfaction with studying, and willingness to recommend the app.

Cluster 3 did not differ from nonusers or other clusters for outcomes at the scale level. At the item level, enjoyment was again rated more favorably than by nonusers. Descriptive questionnaire results are provided in Multimedia Appendix 2.

Open-ended responses are provided as tabulated summaries in Multimedia Appendix 3. Among nonusers and cluster 1, the most frequently reported themes were time constraints and a perceived misalignment between the app content, predominantly consisting of selected-response formats, and the oral examination format. Participants from clusters 2 and 3 more frequently highlighted motivational aspects and emphasized the app’s contribution to learning the subject of microbiology. In these clusters, positive experiences related to enjoyment, perceived progress, and engagement were more prominent. For example, there were frequent references to gamification features such as affirmation through comparative rankings.

Analyses focused on the summer semester of 2024, including 35 nonusers and 60 app users distributed across 3 usage clusters. The written M1 examination percentage score served as the primary standardized measure of prior academic ability (Table 4).

At this baseline, users had slightly higher written M1 examination scores than nonusers (73.4% vs 70.2%; P=.04). Comparing usage clusters, only cluster 3 scored significantly higher than nonusers (76.6% vs 70.2%; 1-sided P=.02), while cluster 1 scores did not differ significantly from those of nonusers (72.3% vs 70.2%; 1-sided P=.12). Cluster 2 exhibited a nonsignificant trend toward higher performance (73.8% vs 70.2%; 1-sided P=.06).

Within-group analyses of oral microbiology grades relative to the written M1 baseline revealed significant improvement for nonusers (P=.04), cluster 1 (P=.001), and cluster 2 (P=.01). In contrast, cluster 3, despite the highest written M1 examination score (76.6%), did not exhibit any significant improvement (P=.34). Among users, cluster 3 had the lowest oral microbiology grade (mean 2.5), whereas nonusers scored slightly lower (mean 2.8). A significant interaction effect between cluster 2 and nonusers was observed for oral microbiology grades (1-sided P=.04).

Written M1 examination scores predicted oral microbiology grades significantly (P=.006; η_p²=0.08). After analysis of covariance adjustment, the group×examination interaction for users versus nonusers remained statistically significant (P=.03; η_p²=0.05), whereas cluster interaction effects were not significant (P=.23; η_p²=.05). Effect sizes were small.

Established models of mLearning emphasize that learning outcomes emerge from the interaction between learning context, technological affordances, and individual learner characteristics, rather than from the technology alone [17,19,20]. However, evidence derived from authentic curricular settings remains limited. The primary aim of this study was not to demonstrate the effectiveness of a specific mLearning intervention. Instead, we examined how learner characteristics and contextual conditions shape engagement and learning outcomes when mLearning is offered as a voluntary supplement within a routine medical curriculum [16,17]. By integrating effectiveness-related outcomes with technology acceptance perspectives within a RE framework, we moved beyond average effects. This enabled us to identify CMO configurations that explain heterogeneity in engagement patterns and learning outcomes (Multimedia Appendix 4) [55].

A central finding of the study was that prior academic performance remained the strongest predictor of examination scores, while the independent effect of app use was small after statistical adjustment. Rather than constituting a limitation, this finding represents a key empirical result. Students with higher baseline academic performance were more likely to engage with the app, reported more favorable learning experiences, and achieved better outcomes. This pattern is consistent with previous research illustrating that academically stronger students tend to display greater self-regulatory capacity, stronger goal orientation, and a higher propensity to engage with optional learning resources [45,69-71]. Accordingly, we found no evidence that the mLearning intervention compensates for lower baseline academic performance. This raises important equity concerns when mLearning is offered as a voluntary supplement, as it may preferentially benefit academically stronger students and reinforce existing differences [72]. This observation aligns with broader discussions on digital inequities in health professions education, in which students with weaker baseline skills or fewer resources are at risk of being left behind [73-75].

Self-selection provides valuable insight in this setting and should not be regarded solely as a source of bias. However, it limits causal inference regarding any independent intervention effect. At the same time, it directly addresses the core question of this study: Who engages with mLearning under routine curricular settings? It also helps explain why mLearning tools may fail to benefit those learners who arguably need them most. This gap is reinforced by the fact that most existing studies report average effects without stratifying outcomes according to baseline performance, leaving equity-related questions largely unaddressed [42,76].

The cluster analysis further illustrated substantial heterogeneity in engagement trajectories, reinforcing evidence that mLearning use is highly individualized [77]. Neither sporadic and low-intensity use nor early but temporary engagement was associated with favorable outcomes. In contrast, students who integrated the app strategically into existing study routines and used it in an examination-oriented manner exhibited the most favorable profiles. These findings indicate that mLearning does not compensate for absent or inconsistent learning strategies, but rather amplifies existing ones [78]. Instructional features such as spaced-repetition, feedback, or gamification should not be interpreted as mechanisms in themselves, but as affordances that may activate motivational, metacognitive, and self-regulatory processes in specific learner groups [69,77]. Consistent with the RE perspective, mechanisms emerge from the interaction between learner dispositions and contextual conditions, which may render identical features effective for some learners while remaining largely ineffective for others [55,56,60].

In contrast to much of the prior technology acceptance research [59,79-81], which has predominantly emphasized perceived usefulness as the key driver of adoption, our findings indicate that enjoyment emerged as a more robust correlate of sustained engagement and positive learning experiences [76,80,82,83]. In addition, intensive and strategic users were consistently more satisfied with their learning experience. This suggests that affective components play a central role in learners’ intention to engage with mLearning tools, particularly when use is voluntary. In this context, the prominent role of enjoyment highlights the importance of addressing affective and cognitive dimensions of learning [77,80,84].

Our findings indicate that mLearning effectiveness is context-dependent rather than technology-driven. Favorable outcomes were observed linked to strategic integration into existing study routines. Accordingly, commonly used instructional elements such as spaced repetition [85-87], gamified feedback [88-90], just-in-time learning [91,92], and microlearning [93-95] should be understood as supportive conditions rather than universally effective components. For curriculum designers, mLearning needs to be embedded within structured learning pathways. Evidence suggests that didactic scaffolding (clear pedagogy, feedback, and structured learning activities) [96] and systemic monitoring of engagement patterns [97] can substantially strengthen mLearning. In addition, learning analytics or AI-supported personalization may help address learner heterogeneity by enabling more adaptive and context-sensitive learning pathways [98-101].

To further operationalize these principles, implementation strategies should target both initial uptake and sustained engagement. First, low-threshold access and structured onboarding processes (eg, seamless integration into learning management systems, single-sign-on, and visible entry points such as QR codes) are essential to reduce barriers to participation. Second, curricular alignment and active implementation strategies, including faculty endorsement and in-app prompts such as push notifications, can reinforce learning routines and support sustained, self-regulated learning engagement [39,102]. Third, interventions should explicitly address both perceived usefulness and affective engagement, as both are critical determinants of adoption and continued use in voluntary learning contexts [103]. Fourth, content should follow microlearning principles, using small, manageable units; intuitive navigation; and adaptive difficulty to support learners with different prior knowledge and promote equitable participation [92,95]. Finally, gamification elements (eg, leaderboards, badges, and progress indicators) can support engagement by making learning progress visible, as implemented in the app used in this study.

The quasi-experimental design and voluntary app use limit causal inference, as observed differences in academic outcomes cannot be attributed to the intervention alone. In addition, the absence of randomization and the reliance on observational data restrict conclusions regarding independent intervention effects. Self-selection by higher-performing students precludes attribution of observed differences in academic outcomes to the intervention alone. Within an explanatory framework, however, this self-selection reflects authentic adoption behavior and therefore constitutes a meaningful finding rather than a methodological flaw.

Further limitations include potential expectancy effects, the timing of the postexamination survey, unmeasured learner characteristics (eg, self-regulated learning strategies or time management skills), and limited statistical power in subgroup analyses. Exactly half of the 220 participants used the app, but only 33.6% (n=37) completed the questionnaire. This potential nonresponse bias limits the generalizability of conclusions linking self-reported evaluations to actual usage behavior. The exclusive reliance on quantitative data further constrained interpretive depth and limited insight into learners’ subjective experiences and decision-making processes.

Future research should integrate summative academic outcomes with longitudinal analyses of engagement behavior to examine how learner characteristics and curricular embedding jointly shape mLearning use and effectiveness.

Prior academic performance proved to be the strongest predictor of engagement in our study. However, other potentially relevant factors were not assessed, such as digital literacy, personality traits, or interest in the subject, which may partly explain cluster formation and differential usage patterns [104-107]. These unmeasured characteristics may influence both students’ interaction with mLearning interventions and the resulting learning outcomes, thereby contributing to the heterogeneous engagement patterns observed. Consequently, future research should systematically investigate these variables, ideally using mixed methods designs that combine quantitative usage tracking with qualitative insights into learners’ intentions and experiences.

Learning analytics and AI-enabled personalization can help address differences between learners. One example of this is providing feedback on engagement and learning behavior to support more individualized learning [108,109] However, their potential to support learners with lower baseline academic performance requires careful empirical evaluation.

Overall, this study highlights the importance of examining mLearning as a situated educational practice, emphasizing the need for designs and evaluations that reflect the complexity of students’ attitudes toward and actual use of mLearning within their learning contexts.