A cross-sectional, web-based survey was conducted from November 3, 2022, to January 3, 2023, to explore the perspectives of educators and resident physicians on assessment platforms in emergency residency training. As a preliminary investigation, this study aimed to assess the feasibility of mobile-based assessments, refine survey methodology, and identify key themes for future large-scale research.

We calculated the sample size using a t test for 2 independent proportions. Given that mobile-based programs constitute approximately one-fourth of RRC programs, we based our estimation on a 1:4 ratio. Assuming a 1-point difference in evaluations regarding training individualization, we determined that a minimum of 11 mobile-based participants and 44 paper- and web-based participants would be needed to achieve 80% power at α=.05 [20].

The survey targeted program hosts, trainers, and trainees across emergency residency training sites in Taiwan, inviting them to participate anonymously (Multimedia Appendix 1). A stratified random sampling method was applied, proportionally selecting participants from accredited emergency residency training hospitals as recognized by the National Emergency Medicine Association. From a total of 1068 qualified educators and 313 resident physicians, a 5% sampling rate was used, resulting in invitations sent to 55 educators and 20 resident physicians. In addition, to gather insights on the perspectives of RRC hosts, 10 were invited to participate in the survey [20].

This study evaluates assessment designs used across different emergency residency training sites. Participants from diverse programs were categorized into paper- and web-based or mobile-based evaluation methods for detailed analysis. Paper- and web-based programs were grouped together due to shared limitations, such as delayed feedback, lack of real-time performance tracking, and reliance on periodic evaluations rather than continuous competency monitoring. In contrast, mobile-based programs provided immediate feedback and seamless trainee progress tracking, distinguishing them as a separate category.

The questionnaire was developed using the Delphi method, ensuring expert consensus on its content and structure. It included a background survey capturing details, such as the name and level of the training site, the number of trainers and trainees, and the participant’s role in the training program. The core survey items consisted of Likert-scale questions (5-point scale, ranging from “Strongly Disagree” to “Strongly Agree”) designed to assess perceptions of CBME core elements. These responses were analyzed and compared to evaluate differences in assessment approaches. To ensure the integrity and reliability of the collected data, only fully completed surveys were included in the final analysis. Incomplete responses were excluded to maintain data accuracy. In addition, responses underwent internal consistency checks to detect and exclude any contradictory or inconsistent answers.

The questionnaire was distributed on the web, with participants invited via email to complete it voluntarily and anonymously. No monetary incentives were provided; however, participants were informed that their input would contribute to improving workplace-based assessments in emergency residency training. While hosts offered additional insights, their responses were categorized under trainers for analysis, as their primary role aligned with faculty responsibilities.

Analyses were conducted using IBM SPSS Statistics (version 23). Dependent variables included assessment satisfaction, assessment and feedback duration, frequency of missed assessments, ability to identify performance trends, ability to review and respond within 24 hours, ability to individualize training programs based on performance results, and ability to track the completion rate of required clinical scenarios and assessments.

Independent variables included the training site level (district, regional, or center hospitals), participants' age, participant role (trainee or trainer), and assessment platform (mobile-based or paper- and web-based).

The Mann-Whitney U test was used to compare differences in dependent variables across training sites and participant roles. A linear regression model with repeated measurements at the hospital level was applied to evaluate the ability to individualize training programs based on performance results, adjusting for potential confounders, including the number of trainers and trainees. The chi-squared test was used to analyze differences in assessment platform performance, with statistical significance set at P<.05. The chi-square test was used to analyze differences in assessment platform performance, with statistical significance set at P<.05.

This pilot study received ethical approval from the Taipei City Hospital institutional review board (TCHIRB-11110007-E). The study was conducted after being approved by the Taipei City Hospital institutional review board on November 24, 2022. The survey was anonymous. The informed consent was waived by the Taipei City Hospital institutional review board.

In this study, 62.22% of emergency residency training sites nationwide participated, amounting to 28 out of 45 sites. A total of 74 valid responses were collected from 28 emergency residency training sites across Taiwan, including district, regional, and center hospitals. Among the respondents, 49 trainers participated, reflecting an 89.1% response rate, while 16 resident trainees responded, yielding an 80% response rate. To further explore the perspectives of RRC hosts, 10 were invited to participate, with 9 providing responses. Regarding assessment platform usage, 14 participants reported using mobile-based assessments, whereas the remaining 60 relied on either paper- or web-based platforms. Within this group, 3 participants used paper-based assessments, and 57 used web-based systems.

The questionnaire demonstrated strong reliability for trainee responses, with a Cronbach α of 0.83, indicating high internal consistency. For trainers, the reliability was moderate, with a Cronbach α of 0.61, which is acceptable for exploratory research. In addition, the content validity index of 0.96 confirmed strong expert agreement on item relevance, supporting the overall validity of the questionnaire.

The analysis revealed no significant differences between trainers and trainees concerning the distribution of assessment platforms, satisfaction with the evaluation process, the time taken for evaluation and feedback, the likelihood of forgetting mutual assessments, or responses to core CBME-related questions (Table 1). However, when comparing mobile-based assessments with paper- and web-based platforms, mobile interfaces demonstrated several advantages. Respondents using mobile-based platforms reported higher satisfaction with the assessment process, a lower likelihood of missing assessments, and an improved ability to identify performance trends. In addition, they were more likely to review feedback within 24 hours, found it easier to tailor training programs based on performance results, and experienced greater ease in identifying required clinical scenarios and necessary assessments (Table 2).

Further statistical analysis indicated that the use of MB platforms was significantly associated with a 2.08-point increase (95% CI 1.73‐2.43, P=.002) in the ability to individualize training programs based on performance results (Table 3).

This pilot study found that mobile-based programs more frequently used visualized performance trends (85.71%) than paper- and web-based programs (73.33%), enhancing trainees’ understanding (mean 4.71, SD 0.61 vs mean 2.72, SD 1.06; P<.001). Mobile-based platforms also enabled faster review of ad hoc responses within 24 hours (mean 4.5, SD 0.85 vs mean 2.22, SD 0.88; P<.001) and better supported individualized training in alignment with CBME principles (mean 4.36, SD 0.84 vs mean 2.27, SD 0.78; P<.001). Statistical analysis indicated a significant association between MB assessments and improved training individualization (2.08-point increase, 95% CI 1.73‐2.43; P=.002), suggesting mobile-based platforms facilitate more timely and adaptive program adjustments than paper- and web-based (Table 3).

Since 2009, CBME has become a global standard in medical training [1]. Oudkerk Pool et al [14] highlight that competency judgments are formed through an iterative process of acquiring and synthesizing evidence, underscoring the need for structured, multisource assessments. However, effective CBME implementation requires more than data collection; it demands meaningful integration and interpretation among instructors, residents, and program coordinators, including RRC hosts and CCC members [14,16].

Despite advancements in assessment methods, paper- and web-based evaluations remain prevalent in Taiwan’s residency programs (Table 2). These static, retrospective tools limit real-time performance tracking and individualized feedback. Chan et al [16] demonstrated that programmatic workplace-based assessments, such as the McMaster Modular Assessment Program (McMAP), improve competency evaluations by replacing single-assessor recall with multisource, continuous feedback. Likewise, our findings suggest that mobile-based platforms offer a promising alternative by enabling timely communication, visualizing performance trends, and tracking clinical exposures in real time, aligning with CBME’s emphasis on progressive competency tracking.

However, while convenience is improved, adopting a mobile-based platform alone does not guarantee the effective integration of summative performance trends. As Oudkerk Pool et al [14] emphasize, competency judgments require active evidence synthesis rather than passive data aggregation. Therefore, mobile-based platforms must be supported by faculty training and institutional commitment to continuous assessment. Beyond technology, mobile-based platform development reflects institutional investment in assessment culture. Successful implementation requires standardized tools, integration, and a culture that embraces real-time feedback. Chan et al [16] highlight that workplace-based assessment systems like McMAP not only streamline logistics but also normalize structured, frequent evaluations, shaping residency training [12]. This may explain why, in our study, the mobile-based platform yielded higher satisfaction and a more tailored training experience compared with the paper- and web-based group.

Compared with mobile-based assessment evaluations, paper- and web-based assessments, though practical, are often less accessible, time-consuming, and associated with lower compliance [19]. Mobile-based platforms provide easy access and real-time feedback, allowing trainers and trainees to review performance trends anytime and anywhere. In addition, their integration with smartphone features—such as cameras, note apps, calendars, GPS, and barcode scanners—enhances efficiency and usability [18,19,21-23]. However, while mobile-based platforms improve accessibility, Walsh [5] noted that mobile learning in medical education may also introduce distractions, potentially affecting engagement and focus during training sessions.

Previous studies have demonstrated the feasibility and effectiveness of mobile-based assessment tools in medical training. Nethala et al [11] found that mobile apps enable real-time assessment and individualized skill training in urology residency programs, facilitating structured competency tracking. Similarly, Sung and Park [22] reported that a mobile-based training program improved nurses’ competence through enhanced accessibility and interactive learning. Green highlighted the cost-effectiveness and usability of smartphone platforms for surgical resident evaluations, reinforcing the practical benefits of mobile-based assessments in medical education [23].

In our study, while assessment and feedback durations did not differ significantly between mobile-based and paper- and web-based platforms, mobile-based assessments yielded higher satisfaction and a lower frequency of missed evaluations (P<.001 and P=.02, respectively; Table 1). These findings suggest that the convenience and accessibility of mobile platforms may enhance trainer engagement and encourage more consistent assessment documentation.

Beyond technological advantages, integrating multisource evaluations into visualized trend tracking on mobile-based platforms may enhance participant engagement, foster active participation, and support continuous improvement. According to self-determination theory, providing trainees with tools for real-time performance monitoring fosters autonomy, competence, and relatedness, key psychological needs that enhance motivation and self-directed learning [13]. When trainees and trainers can instantly track progression and performance trends, they gain a clearer perception of their self-determination (Figure 2), which may drive greater accountability and active participation in their own professional growth. This aligns with Association for Medical Education in Europe Guide No. 59, which highlights the role of real-time feedback in enhancing intrinsic motivation and engagement in competency development [12,13].

Our survey found that although some web-based programs included visualized trends and completion rates, mobile-based programs had a significantly greater impact on both trainers and trainees (mean 2.3, SD 0.8 vs mean 4.4, SD 0.8; P<.001; Table 2). This advantage likely stems from the mobile-based platform’s accessibility, enabling real-time review compared with paper- and web-based assessments, which are typically evaluated only during biannual or quarterly CCC meetings [24]. Delayed review in paper- and web-based evaluations may hinder timely performance adjustments, whereas immediate access to evaluation results in mobile-based programs enables trainees to modify their performance on the next shift, reinforcing self-directed learning [13].

The observed advantages of mobile-based platforms should not be attributed solely to their technology but rather to the broader institutional commitment to structured assessment and a culture of continuous feedback. While mobile-based platforms facilitate real-time performance tracking and self-directed learning, their effectiveness depends on institutional investment in faculty training, standardized assessment frameworks, and integration within CBME structures [4,16]. The shift from paper- and web-based biannual review model [24] to continuous monitoring reflects not just technological convenience but a deeper transformation in how competency is assessed and supported. As previous studies highlight, frequent, structured feedback is essential for meaningful competency development [12,15], and the adoption of MB platforms likely signifies an institutional prioritization of learner-centered training and assessment rigor rather than merely a technological upgrade. This shift, reinforced by evidence-based coaching and real-time decision-making, aligns with a growing recognition that assessment culture, not just digital tools, drives improved training experience [13,16].

The higher satisfaction with mobile-based platforms compared with paper- and web-based (P<.001, Table 2) may be attributed to their ability to provide real-time accessibility, structured assessments, and visualized performance trends, benefiting trainees, trainers, and RRC hosts. For trainees, mobile-based platforms enabled immediate feedback and continuous performance tracking, allowing timely adjustments rather than relying on periodic CCC reviews. This aligns with CBME principles by supporting self-directed learning and competency progression. In addition, mobile-based platforms facilitated easier identification of required clinical scenarios and assessments (P<.001), which may contribute to a more structured training experience.

For trainers, mobile-based platforms allowed more flexible and timely assessments, reducing recall bias and documentation burden. The ability to complete evaluations outside clinical hours (P=.02; Table 2) suggests increased convenience and consistency in providing feedback. The integration of smartphone tools, such as cameras and note apps, may have contributed to streamlined documentation and more structured evaluations [21]. For RRC hosts, mobile-based platforms provided a longitudinal perspective on trainee progress, reducing reliance on episodic assessments and allowing competency decisions to be based on continuous performance data rather than retrospective impressions [14,16]. The findings suggest that mobile-based platforms improve accessibility and assessment efficiency, reflecting a shift toward more continuous, data-driven evaluation in residency training.

The preference for mobile-based platforms in tailored training (P<.001; Table 2) stems from their ability to provide real-time feedback, trend visualization, and individualized learning, aligning with CBME’s emphasis on continuous assessment [2,4]. Unlike paper- and web-based periodic evaluations, mobile-based platforms empower trainees with self-directed learning and immediate performance adjustments while enabling trainers to deliver data-driven coaching with minimized bias [12,16]. For RRC hosts, mobile-based platforms enhance longitudinal competency tracking, overcoming the limitations of CCC’s fixed evaluation intervals and allowing timely interventions [1,3,24]. However, their success depends on structured implementation, faculty engagement, and integration within CBME frameworks to ensure meaningful assessment without cognitive overload [5,6,17].

Overall, while mobile-based platforms show promise in improving feedback timeliness and training individualization in our pilot study, further research is needed to evaluate their long-term impact on competency development, clinical outcomes, and residency training culture.

First, mobile-based platforms provide real-time feedback and continuous performance tracking, complementing scheduled CCC reviews and addressing paper- and web-based limitations in integrating multisource evaluation results. Second, structured mobile-based evaluations improve engagement and training individualization, offering more timely and adaptive learning opportunities that better align with CBME principles compared with paper- and web-based platforms. Finally, successful integration depends on faculty training and strategic implementation, enhancing assessment validity, competency tracking, and institutional commitment to resident development, ultimately leading to greater satisfaction.

Structured mobile-based evaluations improve engagement and training individualization, offering more timely and adaptive learning opportunities that better align with CBME principles compared with paper- and web-based platforms. Successful mobile-based integration depends on faculty training and strategic implementation, enhancing assessment validity, competency tracking, and institutional commitment to resident development, ultimately leading to greater satisfaction.

First, the study results were based on a convenience sample from an internet survey. However, enrollment only covered 62.22% of the emergency residency training sites in the country. Second, the survey results indicated that only 14 out of 74 responses applied mobile-based assessment. Further investigation to evaluate mobile-based assessment will be needed as more training programs start being delivered through the mobile interface. Finally, in the survey, participants provided responses based on their experience with the interface’s ability to facilitate tailored training adjustments, rather than on direct evidence of its actual implementation. Further research is needed to assess the long-term impact of mobile-based platform implementation on training outcomes.

The survey results indicated that only 14 out of 74 responses applied mobile-based assessment. Further investigation to evaluate mobile-based assessment will be needed as more training programs start being delivered through the mobile interface.

In the survey, participants provided responses based on their experience with the interface’s ability to facilitate tailored training adjustments, rather than on direct evidence of its actual implementation. Further research is needed to assess the long-term impact of mobile-based platform implementation on training outcomes.

In conclusion, the mobile-based interface emerges as a dynamic and effective platform for emergency residency training programs. It facilitates rapid updates and individualized program modifications, thereby increasing the engagement and satisfaction of trainers, trainees, and other stakeholders. Contrarily, the conventional paper- and web-based methods face limitations due to prolonged review periods in committee meetings, which may lead to delays in crucial program modifications and impede the progression of the program. The adoption of mobile-based technology in this context demonstrates its capacity to greatly enhance the efficiency and efficacy of CBM, particularly in making tailored adjustments. This technology also promotes better-informed and more collaborative interactions among all involved parties.