Distance Learning During the COVID-19 Lockdown and Self-Assessed Competency Development Among Radiology Residents in China: Cross-Sectional Survey

Introduction

The COVID-19 pandemic threatens the health of people globally and has brought unprecedented pressure to health systems [1,2]. The national public health system plays a vital role in fighting against pandemics by taking measurements such as surveillance and epidemiological investigations [3], case finding and management [4], and collective quarantine of close contacts [5]. However, potential challenges including insufficient alerts, low efficiency of reporting to higher authorities, and workforce shortages still exist [1]. Of note, the training of health care providers and the improvement of their professional skills have been underscored for their great significance in medical service delivery and health systems resilience [6-8].

Residency training systems serve the purpose of cultivating a qualified health workforce [9]. Standardized residency training (SRT) was initiated in 2013 in China, aiming to train doctors to meet the needs of population health [10]. With the increasing trend of competency-based medical training in global medical education, the assessment of competencies has gained ground in practice [11]. The US Accreditation Council of Graduate Medical Education (ACGME) identified 6 core competencies for physicians (ie, patient care [PC], medical knowledge [MK], system-based practice [SBP], practice-based learning and improvement [PBLI], professionalism [PROF], and interpersonal communication skill [ICS]) [11] and implemented milestones by the Next Accreditation System initiative in July 2013 [12]. Residency education and competency-based practice assessed by the milestones are common requirements of ACGME and have been used in residency training in China [9,13].

COVID-19 has changed medical education dramatically, especially during the lockdown period. The impacts of COVID-19 on medical education in radiology, surgery, and emergency medicine have gained attention [14-16]. Radiology is related to other medical specialties and all levels of health care delivery [17]. Radiology residents were typically required to rotate between different specialties to obtain knowledge and clinical skills [18]. The mandatory social distancing challenged the traditional training on radiology trainee approaches such as teaching at workstations [19]. In China, SRT in radiology spans 3 years and involves workstation-based training throughout rotations in various specialties and departments [9]. In the first year, residents undergo rotations in the departments of radiology, ultrasound medicine, nuclear medicine, pathology, and relevant clinical departments. In the second and third years, they receive advanced rotational training within radiology subspecialties such as computed tomography, magnetic resonance imaging, x-ray, and interventional radiology [9]. Due to COVID-19, the mode of residency training has been switched from traditional in-person classes to distance learning [20], posing challenges to the effectiveness of rotational training and the developing competency of residents.

Numerous benefits have been found for distance learning. For instance, residents can schedule more flexibly and access the courses more easily [21]. They can learn at their own pace with the help of recorded lectures and communicate with professionals and peers on the web at their own convenience [22]. The positive acceptance and a higher level of satisfaction with distance learning have been reported by residents in Canada and the United States [21,23]. However, the practice of digital readout in distance learning is similar to the experience of in-person reading, in addition to the difficulties of gauging body language during practical operations or in the use of medical instruments [24], which may lead to unsatisfactory outcomes in radiology education. Meanwhile, the COVID-19 pandemic has impacted the mental health status of health care professionals dramatically [25]. Students who experienced distance learning during the pandemic had been found to have psychological distress [26,27]. According to the Job Demands-Resources model, mental health is a personal resource that helps residents to deal with job challenges by moderating their performance [28-31]. Accordingly, it is of great importance to take care of the mental health of residents who have experienced distance learning [32]. To date, during the COVID-19 lockdown, when workstation-based training was difficult to deliver, the role of distance learning remains unclear. It is also uncertain whether psychological status affects the effectiveness of distance learning.

In sum, the COVID-19 lockdown had brought substantial challenges to radiology training programs, which had to transition from face-to-face instruction to remote learning. In the meantime, mental distress caused by factors such as social distancing may make residency training rather difficult. Given that distance learning remains a primary alternative when traditional teaching is not feasible (such as during pandemic outbreaks and lockdowns), yet few studies have explored its effects, we aimed to investigate the impact of distance learning on the development of self-assessed competencies as well as the moderating effect of mental health status. We hypothesize that radiology residents who received distance learning during the COVID-19 lockdown were more likely to report higher self-assessed competencies compared to those who did not receive organized, formal training during the same period (ie, nondistance learners) and that this association was moderated by poor mental health. To test this, we used a nationwide survey dataset of radiology residents in China, which collected information on distance learning and mental health status during the lockdown (January-May 2020), and self-assessed competencies 6 months later. Previous studies have shown a strong positive correlation between the assessments by Clinical Competency Committees and residents’ self-evaluations using the milestones. This suggests that residents are generally able to accurately assess their own competencies, which in turn supports the validity of using milestone assessments as an effective measure of self-assessed competency in this study [33].

Methods

Ethical Considerations

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the institution review board of Tsinghua University, China (20210140). Informed consent was obtained before the research started, and data were deidentified to ensure participant privacy. Participation in this survey was voluntary, participants did not receive any incentives to take part in the study.

Study Setting and Population

A nationwide retrospective cross-sectional survey was conducted on the web by the Chinese Association of Radiologists (CAR) during December 1, 2020, to April 30, 2021, targeting all the radiology residents in 407 radiology residency programs across 31 provinces of China.

To complete the distribution of the questionnaire and to ensure the participation of the radiology residents, we contacted the directors of the targeted hospitals’ radiology departments by email or telephone to inform them of the purpose and details of the survey initiated by the CAR. The directors were then instructed to share the link of the questionnaire posted on the popular web-based survey platform “Wenjuanxing” with radiology residents. Anonymous responses to the questionnaire were submitted. All participants were informed that the questionnaire could only be filled out once, that participation was completely voluntary, and that they could withdraw at any time without penalty. During the 5-month survey period, the research team cooperated well with the CAR for the monitoring of the participation. If there was a low rate of submission, the CAR would require the hospital to improve the quality and quantity of their survey response. This proactive approach could help increase the number of responses and the representativeness of the population. Residents who did not undergo residency training during the COVID-19 epidemic between January and May 2020 would be excluded.

Measures and Outcomes

Distance Learning

Distance learning was asked “Was your training institution changed the teaching arrangements to the form of distance learning to reduce the negative impact during the initial COVID-19 outbreak?” The response options were either “yes or no.” In contrast, nondistance learning participants (ie, nondistance learners) were those who did not receive organized, formal training during the same period. They primarily stayed at home on leave, reported daily health status, and engaged in delayed teaching plans or self-directed learning.

Mental Health Status

Mental health status was measured by the question “Did you suffer psychological distress during the initial COVID-19 outbreak? (from January to May 2020 in China).” A 5-point Likert scale was used to measure the degree of mental health struggles: 1=no impact, 2=mild impact, 3=moderate impact, 4=severe impact, and 5=extremely severe impact. Radiology residents can choose any rating between 1 and 5 (single choice). The variable is correlated with long-term mental health (depression and burnout) measured by the Depression and Anxiety Stress Scale—Depression and Maslach Burnout Inventory scales (Multimedia Appendix 1). The Cronbach α reliability coefficient for depression and burnout was 0.930 and 0.957, respectively.

Self-Assessed Competency

Milestone-based assessment of competencies for residence is one of the common requirements of the ACGME [11]. Self-assessment plays a key role in this process by fostering reflection on professional actions, identifying learning needs, and enabling residents to develop and refine personalized improvement plans [34]. Moreover, residents’ self-assessments showed a strong alignment with the Clinical Competency Committee evaluations across postgraduate year levels [33,35,36]. To estimate the long-term outcomes of radiology residents’ training after the initial COVID-19 outbreak, our study measured 6 core competencies developed by the ACGME that were assessed by radiology residents themselves. As is suggested by the experts, we selected 9 subcompetencies from diagnostic radiology milestones to represent the 6 ACGME core competencies: 2 PC subcompetencies, 2 MK subcompetencies, 2 PBLI subcompetencies, 1 SBP subcompetency, 1 PROF subcompetency, and 1 ICS subcompetency.

A dedicated section of the questionnaire is designed to assess 9 subcompetencies with 9 single-choice questions. Radiologists are able to select a score ranging from 0 to 9 for each competency. Examples of milestone sets for each subcompetency are shown in Figure 1. The primary outcome was self-evaluation milestone (SEM) scores (range 0‐9 scores) for 9 subcompetencies and the average SEM scores.

‎

Sociodemographic Characteristics

The sociodemographic information included age (≤27 or >27 years), sex (male or female), educational level (bachelor’s degree or master’s degree or above), training year (the second year or the third year), training sites level (grade-a tertiary general hospital, grade-a tertiary specialized hospital, grade-b tertiary general hospital, or others), undergraduate major (clinical medicine, medical imaging, or others), working hours per week (≤40, 40‐48, or >48), annual after-tax income in 2020 (analyzed as continuous variable), and types of residents (professional master or nonprofessional master).

Statistical Analysis

The SEM scores of radiology residents for 9 subcompetencies were reported by means and SDs). The differences in SEM scores of residents between distance learning and nondistance learning were compared using independent samples 2-tailed t test. To explore the association between distance learning and competencies, multiple linear regression (MLR) models were constructed. The dependent variables were SEM scores of 9 subcompetencies, and the key explanatory variable was distance learning. The moderating effect of mental health on distance learning was explored by the MLR model. The significance of the moderating effect was tested by simple slope analysis. All models were controlled for participants’ characteristics. A variance inflation factor was used to detect the multicollinearity of independent variables for all models (variance inflation factor scores <3). A P value of <.05 was considered statistically significant for 2-tailed tests (t test or chi-square test). A conservative level of P value of <.10 was used to assess potential moderators in the regression, which was reported by a coefficient (β) and 90% CIs [37,38]. All statistical analyses were performed by STATA (version 17.0; StataCorp LLC).

Results

Participants’ Characteristics

Of the 8008 targeted radiology residents, 2381 (overall effective response rate: 29.7%) participated in this survey (Figure 2). As is shown in Table 1, the mean age of the participants was 27.8 (SD 2.4) years. In total, 58.5% (n=1392) of them were female, and 50.5% (n=1202) were in the third-year training. The majority of the participants received training in a grade-a tertiary hospital (n=2310, 97%), had a bachelor’s degree (n=2187, 91.9%), and their undergraduate major was medical imaging (n=2016, 84.7%). The median annual after-tax income was 40,000 RMB (IQR 10,000‐60,000; a currency exchange rate of 1 RMB=US $0.145 is applicable), with 25.8% (n=614) of them earning more than 60,000 RMB (about US $ 8698.9). The average working hours per week was 44.3 (SD 12.5) hours, and 23.1% (n=551) of the participants worked more than 48 hours per week. During the initial COVID-19 outbreak from January 2020 to May 2020, 71.4% (n=1699) of the radiology residents participated in distance learning, and 73.2% (n=1742) of them reported slight or severe mental health struggles. In total, 35.8% (n=853) of the participants contributed to the prevention and control of COVID-19.

‎

SEM Scores of Radiology Residents Between Distance Learning and Nondistance Learning

The mean score of competencies and a comparison of subcompetencies scores were presented in Table 2. The overall average score of radiology residents’ competency was 3.37 (SD 1.47). The average score of participants who received distance learning was 3.46 (SD 1.49), higher than those who did not (mean 3.13, SD 1.39; P<.001). Residents who received distance learning outperformed in all subcompetencies than those without distance learning during the initial COVID-19 outbreak (PC-1: P=.007; MK-1: P=.004; and others: P<.001), except for PC-2 (P=.09; Table 2). Due to the low response rate, Mann-Whitney tests were performed. The results were similar (Multimedia Appendix 2). For radiology residents who had not participated in COVID-19–related activities (1528/2381; Multimedia Appendix 3), the differences between residents’ competencies or subcompetencies showed the same trends, except for the PC-1 (P=.10).

Association Between Distance Learning and Competencies Among Radiology Residents

As is shown in Figure 3 (see also Multimedia Appendix 4), MLR analyses showed that radiology residents who were trained through distance learning were more likely to report high competencies (β=0.35, 90% CI 0.24‐0.45) after adjusted by participants’ characteristics, including age, sex, educational level, training years, working hours per week, annual after-tax income in 2020, and types of residents. This was particularly evident in the competencies of “interpersonal and communication skills” (β=0.55, 90% CI 0.39‐0.70) and “contrast agent safety” (β=0.52, 90% CI 0.38‐0.67). Whereas, the competency of “patient care and technical skills” benefited the least from distance learning (β=0.14, 90% CI 0.01‐0.26). The effect of each explanatory variable on the overall average SEM score is shown in Multimedia Appendix 5. Factors associated with higher competencies included older age (>27 years: β=0.12, 90% CI 0.01‐0.22), being male (β=0.28, 90% CI 0.18‐0.37), and having a longer SRT training year (third year: β=0.51, 90% CI 0.41‐0.61).

‎

The Moderating Effect of Mental Health Status Between Distance Learning and Competencies

The association between mental health status and competencies is shown in Multimedia Appendix 6. We controlled the covariates mentioned earlier to investigate the moderating effect of mental health on the association between distance learning and competencies. As is shown in Table 3, there was a significant association between distance learning and radiology residents’ competencies (β=0.63, 90% CI 0.34‐0.91; P<.001) and was moderated by mental health status (β=–0.15, 90% CI −0.27 to −0.02; P=.06). The relationship between distance learning and competencies at low and high (mean−SD and mean + SD, respectively) mental health scores is shown in Figure 4. Poor mental health caused by the pandemic may offset the positive effect of distance learning on residents’ competencies. The highest level of competencies was found in individuals who reported less mental distress and adopted distance learning. Furthermore, the moderating effect of poor mental health on 4 subcompetencies (ie, MK-1, MK-2, SBP-2, and ICS) was similar to it on total competencies (P<.10; Multimedia Appendix 7).

‎

Discussion

Principal Findings

Based on a national survey of radiology residents in China, our study found that radiology residents who received distance learning were more likely to report high proficiency in key competencies after 1 year. This was particularly true for learning knowledge and communication skills but was less evident in obtaining technical skills. In addition, we found a significant moderating effect of mental health status on the association between distance learning and competencies during COVID-19.

Web-based training programs are proposed to mitigate the loss of learning from clinical rotations during the pandemic [39,40]. Consistent with the results of previous findings that distance learning has the potential to improve learners’ academic performance, skill development, and engagement [39,41], our study observed a positive impact of distance learning on radiology residents’ self-assessed competencies. However, it should be noted that the impact of distance learning on professional competency varies among radiology residents. Our study enriches the understanding of distance learning by using milestones of 9 subcompetencies for 6 core competencies. We also assessed the long-term outcomes of distance learning among radiology residents in the initial stage of the COVID-19 pandemic (between January and May 2020). These findings can be used to help inform evidence-based policies to improve residency training in the future.

Knowledge gain is an important indicator of what trainees have learned during SRT training [42]. The radiology residents in our study reported a substantial gain of knowledge and communication skills during SRT, particularly in ICS competency and SBP competency that include clinical knowledge and medical humanities, with a regression coefficient >0.5. Whereas, the effect of distance learning on professional attitudes (PROF) and professional growth in clinical practice (PBLI) was moderate, with a regression coefficient between 0.3 and 0.4, which is in line with previous studies [43-45]. Distance learning was found to be supportive in the continuity of teaching and learning during COVID-19 [39], which might explain why residents who received distance learning reported a higher score in the competence of medical knowledge than nondistance learners. In addition, radiology residents are able to participate in web-based conferences and conduct digital medical consultations to enhance their competency of communication [43], though it could be hard to evaluate their body language in distance learning [46]. Our findings validated the positive role of distance learning in fulfilling the objectives of training by creating an enabling environment, especially in the domain of knowledge and communication. In other words, distance learning can help keep the consistency of residency programs during pandemics, facilitating the resilience and recovery of health systems.

Our findings showed that competence in procedures (PC-2) benefited the least from distance learning, followed by competencies in technical skills (including PC competencies that reflect radiology residents’ professionalism and MK competencies that reflect mastery of professional knowledge and imaging technology). The result is consistent with previous findings that most medical students feel unable to acquire practical clinical skills through web-based teaching during COVID-19 [47]. This could be explained by the reduction in daily cases due to COVID-19, which was one of the deficiencies in distance learning [20]. To address this challenge, new technologies such as touchscreen Anatomage Table and Touch Surgery application have been used to strengthen trainings of plastic surgery [48]. These new approaches can be integrated into web-based residency training to compensate for the shortage of training in technical skills with distance learning. Other training programs, such as competence in procedures, could also be carried out among radiology residents in the post–COVID-19 era.

Health professional education involves maintaining a sense of purpose and mental well-being (eg, balance work life, stress, depression, burnout, and more) among residents [49-51]. Our study extended the prior results by identifying that poor mental health moderates the relationship between distance learning and competency among radiology residents. During the COVID-19 pandemic, physicians experienced more mental health disorders [52] with negative impacts (eg, higher turnover intention and lower work performance) [53,54]. However, good mental health may help trainees to receive distance learning consistently [55]. When trainees are in a suboptimal mental status, they tend to have negative attitudes toward learning and are discouraged from receiving distance learning continuously [55]. Health workforce is the backbone of health systems in response of pandemics [56]. In addition to the structural changes required for the improvement of health systems’ resilience, it could also be necessary to provide long-term psychological support for the health workforce to help them overcome psychological distress [57]. In summary, several recommendations can be drawn from this study: (1) distance learning can be used for transferring knowledge to residents particularly when challenges exist in the traditional offline approaches; (2) clinical skills have a crucial role in offline training, which should be noted and well used by training institutions; (3) instructors could use a scientific and caring approach with a special focus on learners’ psychological well-being when preparing material and organizing courses; and (4) physicians who have engaged in distance learning could enhance their learning experience through an enhancement of their offline learning environment. This promotes a better psychological state for learning to become more effective.

Health system resilience is critical in training the required competencies of the health workforce [49]. In addition to competency-based education, the COVID-19 pandemic highlighted the application of digital interprofessional education [58]. In the future, distance learning may help physicians gain knowledge and skills in public health, interprofessional communication, and teamwork. In this regard, remote residency training should be developed as a holistic educational concept rather than a mere substitution for traditional in-person learning. Distance learning enables physicians to have a flexible schedule, a feasible access to classes, and an opportunity to keep a good balance between work and life [14]. What is more, the facilitation of distance learning contributes to the sharing of high-quality educational resources in the post–COVID-19 era. In particular, for health professionals in resource-constrained places (eg, rural areas in western China), the provision of high-quality distance learning could help them overcome geographical constraints and thus reduce inequalities in access to education [49,59]. Nevertheless, it is not possible to replace in-person teaching with distance learning completely. An integrated model of the 2 is encouraged to maximize the advantages of different teaching modalities.

Limitations

This study has several limitations. First, we used SEM for the self-evaluation of radiology residents’ competencies; results may be influenced by the Dunning-Kruger effect, where participants with lower abilities tend to overestimate their competencies and the potential self-reporting bias [60]. Second, the lower response rate may be subject to selection bias, as our survey is voluntary in nature. However, based on sample size calculations [61], we obtained 1573 valid samples (precision=5%; baseline proportion=0.50), which accurately represent the characteristics of the residents. Importantly, this represents the largest nationally representative sample of radiology residents in China to date, which may help to minimize the potential bias. Third, although mental health status was asked by a single question based on the self-report psychological distress on a Likert scale, this variable is correlated with long-term mental health status (depression and burnout). Other potential moderating factors that may influence learning status, such as the design of the web-based course, courseware, and teaching styles, could be explored in future research. Fourth, the generalization of the results is another limitation of our study [49].

Future Directions

As distance learning is anticipated to be applied across various fields, additional research is warranted to substantiate our findings. Longitudinal studies are recommended for future research to fully assess the long-term effects of distance learning on competence and mental health.

Conclusions

Distance learning helps mitigate the negative impact of the COVID-19 lockdown on the education of health professionals. Meanwhile, attention should be paid to the disadvantages of distance learning and the mental health status of learners, as they may negatively influence the effectiveness and sustainability of distance learning. Our study provides insights into the role of distance learning in residency training during the pandemic.