Contrast-enhanced ultrasound (CEUS) has become an indispensable diagnostic tool for liver diseases, offering real-time, radiation-free characterization of focal liver lesions and dynamic depiction of tissue microcirculation [1]. Its clinical value has been endorsed by more than 20 international guidelines from organizations such as the World Federation for Ultrasound in Medicine and Biology (WFUMB) and the European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB) [2,3]. Nevertheless, the diagnostic accuracy of CEUS is highly operator-dependent [4]. Effective application of CEUS demands high technical proficiency from operators, as the procedure is time-sensitive and susceptible to visual interference [5]. Therefore, ensuring that trainees attain standardized and objectively verifiable competencies before transitioning to independent practice is essential not only for safeguarding training quality but also, more critically, for protecting patient safety and ensuring diagnostic accuracy.

In response to this need, competency-based medical education (CBME) has emerged as a paradigm in medical training, shifting the focus from time-based curricula to demonstrable outcomes that ensure trainees acquire the knowledge, skills, and attitudes necessary for independent practice [6,7]. This approach mandates the use of standardized, validated assessment tools to objectively evaluate clinical competencies and safeguard training quality. According to the Messick unified validity framework, validity is not a property of the test itself but of the interpretations and actions based on test scores. Therefore, assessment tools must gather evidence across 5 sources—content, response process, internal structure, relations to other variables, and consequences—to support high-stakes decisions such as certification or independent practice [8,9]. However, in the field of CEUS—particularly for liver applications, which represent a core clinical scenario—there is a lack of validated assessment tools to objectively evaluate the theoretical knowledge and operational competence of trainees.

The importance of standardized competency assessment in medical education has been increasingly recognized. Recent systematic studies have shown that competency-based curricula require robust evaluation practices to ensure training quality, yet significant variation exists in how assessment tools are developed, implemented, and reported [10,11]. Alharbi [10] systematically reviewed evaluation practices in CBME curricula and found that, while questionnaires remain the primary assessment tool, numerous studies fail to clarify evaluation objectives, approaches, and standards, highlighting the need for more rigorous and transparent assessment methodologies. Similarly, a systematic review of undergraduate CBME assessment identified a range of constructivist techniques—including concept maps, portfolios, rubrics, and workplace-based assessments—as promising approaches, but noted that their implementation remains inconsistent and often lacks validity evidence [11].

Within the ultrasound education domain, recent years have witnessed significant efforts to establish competency frameworks. The ULTRA-Metrics study, a Delphi-derived international consensus initiative, developed a modular framework for assessing ultrasound competency across clinical applications and disciplines, identifying 3 core competency domains (experience, skills, and autonomy) and 4 skill domains (indication, acquisition, interpretation, and clinical integration) [12]. This framework highlighted that existing approaches often lack generalizability, overemphasize exam counts, and fail to assess key skills such as interpretation, thereby limiting ultrasound’s safe and effective integration into clinical practice. Similarly, Delphi consensus studies have been used to define ultrasound competencies in various specialties, including emergency medicine, anesthesiology, and radiology residency programs, reflecting a growing recognition of the need for competency-based ultrasound training [13,14]. However, these frameworks remain largely generic and have not been translated into organ-specific, validated assessment instruments.

Specifically, in the field of CEUS, several efforts have been made to develop competency assessment tools, yet substantial psychometric gaps persist. Jacobsen et al [15,16] developed a theoretical multiple-choice test for core CEUS competencies and gathered validity evidence according to the Messick framework, but the final 47-item test showed a considerable risk of false positives and false negatives when establishing a pass/fail score. The same research group also developed an immersive virtual reality simulation-based test with acceptable reliability (Cronbach’s α=0.85), yet it carried a 23.6% risk of false negatives and focused exclusively on precontrast preparations rather than on comprehensive liver CEUS competency [17]. Furthermore, a scoping search of the literature on CEUS curricula and assessment revealed very limited published work in this area, confirming the scarcity of validated assessment tools.

Beyond CEUS, the broader ultrasound education literature provides instructive parallels. A systematic review of endoscopic ultrasound (EUS) competency assessment tools identified several instruments with strong validity evidence—including TEESAT (The EUS and Endoscopic Retrograde Cholangiopancreatography Skills Assessment Tool), GAPS-EUS (Global Assessment of Performance and Skills in EUS), and EUSAT (EUS Assessment Tool)—but noted that these tools remain specialty-specific and are not directly transferable to other ultrasound applications [18]. Similarly, organ-specific ultrasound assessment tools have been developed for various applications, including lung ultrasound (LUS)–Objective Structured Assessment of Ultrasound Skills (OSAUS) and cranial ultrasound, demonstrating the feasibility of organ-specific competency assessment [19,20]. Paradoxically, although the liver is a primary indication for CEUS, accounting for the majority of global CEUS examinations, no equivalent organ-specific tool has been developed for liver CEUS competency assessment. A 2024 study on CEUS Liver Imaging Reporting and Data System (LI-RADS) classification training for radiology residents demonstrated that dedicated training improved diagnostic performance, yet no standardized assessment tool accompanied that training, underscoring a critical gap between educational intervention and rigorous outcome evaluation [21].

The convergence of the above evidence reveals a clear and urgent knowledge gap: there is currently no validated, organ-specific, and psychometrically sound competency assessment tool for liver CEUS. While generic ultrasound competency frameworks have been proposed, they lack the specificity needed for liver CEUS, which requires unique knowledge of hepatic vascular anatomy, LI-RADS classification, contrast kinetics, and image-guided interventional procedures. Furthermore, previously published CEUS tools have been limited by insufficient validity evidence (eg, undefined cutoffs) or overly narrow scope (eg, covering only precontrast preparations). This deficiency has direct clinical consequences; without robust evaluation criteria, high-stakes decisions regarding whether trainees can independently perform liver CEUS examinations become subjective, ultimately compromising patient safety and health care quality. Given the high operator dependency of CEUS and the growing clinical demand for liver examinations, developing a rigorous assessment tool is an urgent quality assurance priority.

To address these gaps, the present study aimed to develop and validate the first standardized competency assessment tool specifically for liver CEUS (the CEUS Competency Assessment for the Liver [CCAL]). We hypothesized that the CCAL would demonstrate the following characteristics: (1) strong content validity, as established through a Delphi expert consensus process (content evidence); (2) high internal consistency reliability and satisfactory item discriminability (internal structure evidence); and (3) robust external validity across diverse training sites (relations to other variables evidence). Accordingly, through a Delphi process involving 21 experts and a multicenter validation study comprising over 600 trainees, this study sought to provide a reliable, psychometrically sound instrument for objective competency evaluation in liver imaging, thereby filling a critical void in CBME for CEUS training.

This study involved human participants and adhered to the Declaration of Helsinki. Ethical approval was obtained from the Institutional Review Board (IRB) of The Third Affiliated Hospital of Sun Yat-sen University (IRB number: [2020] 02-137-01) and was registered with the Chinese Clinical Trial Registry (ChiCTR2000035750). Written informed consent was secured from all participants prior to enrollment, and they were informed of their right to withdraw at any time without academic or other repercussions. All collected data were anonymized and deidentified before analysis, with no personal identifiers retained in the research database. Participation was voluntary and part of routine assessment procedures; therefore, no compensation was provided. The manuscript and supplementary materials contain no images or content that could identify individual participants.

In phase 1, 6 experts generated items independently without formal enrollment or attrition tracking. In phase 2, all 21 invited experts and 10 recruited trainees completed the Delphi process and cognitive interviews, respectively, with no attrition. For phase 3 internal validation, 111 trainees met the eligibility criteria; 11 declined participation, and the remaining 100 were enrolled and completed the assessment in full, with no missing data or further withdrawals. In phase 4 external validation across 25 hospitals, 646 trainees were assessed for eligibility; 67 declined participation, and 579 were enrolled and completed the assessment with no missing data or withdrawals (Figure 1).

Recruitment was conducted in 4 phases. Phase 1 (item generation) took place in July 2020. Phase 2 (expert consultation and cognitive testing) ran from September to December 2020. Phase 3 (internal validation) was conducted in January 2021 at 4 preselected teaching hospitals. Phase 4 (external validation) occurred in April 2021. The study used a cross-sectional design with no repeated measures or follow-up.

The OQS demonstrated excellent discrimination, with a discrimination index of 0.429, exceeding the established threshold for good discrimination (D≥0.4). Moreover, the OQS items were appropriately distributed across difficulty levels, covering a broad range of trainee ability: 66.0% (35/53, 95% CI 0.526‐0.773) of items were categorized as simple questions (covering basic knowledge points), 32.1% (17/53, 95% CI 0.211‐0.455) as medium-level questions (pertaining to clinical applications), and 1.9% (1/53, 95% CI 0.003‐0.099) as challenging questions (involving advanced skills).

Quantitative analysis of module-specific pass rates (Table 4) identified critical competency gaps: the pass rate for “indications and contraindications” was 81.3% (471/579, 95% CI 0.780‐0.843), indicating solid command of fundamental knowledge, whereas the pass rate for “imaging features for procedure guidance” was only 32.5% (188/579, 95% CI 0.288‐0.364), highlighting insufficient training in CEUS application for clinical procedures.

This study developed and validated the first organ-specific competency assessment tool for liver CEUS. Following a 3-round Delphi process and cognitive interviews, the CCAL was tested in internal and external multicenter cohorts. It demonstrated strong content validity, high internal consistency reliability, and robust construct and discriminant validity. Subdomain analysis revealed a pronounced competency gap in image-guided interventions, confirming the tool’s ability to identify specific educational needs beyond global pass or fail outcomes.

The CCAL advances the field of CEUS competency assessment through several innovations that directly address the limitations of existing tools. Jacobsen et al [15,16] laid important groundwork by developing a core CEUS syllabus and operationalizing it into a 47-item multiple-choice test; however, their instrument was explicitly “not recommended for certification purposes” due to the risk of false positives and negatives, and its utility was limited to entry-level screening. The same research group later developed an immersive virtual reality simulation-based test focusing on precontrast preparations, yet it carried a 23.6% risk of false negatives and did not address comprehensive liver CEUS competency [17]. More broadly, other sonographic assessments, such as the OSAUS or generic radiology in-training examinations, assess general scanning skills or factual knowledge but lack the organ-specific, multidimensional construct needed to capture the unique demands of liver CEUS, including contrast-agent safety, dynamic enhancement patterns, and image-guided interventional decision-making [12,35,36]. These tools, while pioneering, were not organ-specific and lacked a self-assessment component, leaving an unmet need for a multidimensional, liver-dedicated assessment.

The CCAL addresses this gap through 5 defining strengths. With respect to content validity and theoretical foundation, item generation followed the 2020 WFUMB liver CEUS guidelines and was refined through a 3-round Delphi process involving 21 experts, which was followed by cognitive interviews, establishing the CCAL as the first liver-specific, expert-consensus-driven CEUS competency tool [2,37]. This expert-consensus approach stands in marked contrast to assessments developed in isolation by a single institution or those lacking systematic, evidence-informed item vetting. The high internal consistency and a 3-factor structure consistent with the theoretical blueprint confirm that the CCAL measures a coherent yet multidimensional construct—encompassing theoretical knowledge, image interpretation, and clinical judgment—rather than a single global ability. In comparison, prior CEUS tests either reported no reliability data or were confined to narrow procedural tasks [15-17]. The demonstration of known-group discriminant validity, with scores distinguishing trainees by experience and professional title, supports the tool’s use for benchmarking progress along a developmental continuum, a critical requirement for competency-based education [38,39]. Methodologically, the CCAL used a 4-phase mixed methods design and is the first tool to integrate objective knowledge testing with structured self-evaluation within a single instrument, enabling multidimensional assessment of theoretical knowledge, self-awareness, and clinical decision-making rather than a unidimensional pass or fail judgment [40,41]. Finally, the online platform with remote proctoring and automated scoring ensures that the CCAL can be deployed at low cost across diverse clinical settings, lowering the barriers to standardized assessment in resource-limited environments [42].

Beyond global pass-or-fail decisions, the CCAL offers granular diagnostic capability through subdomain analysis. Pass rates were high in fundamental areas such as indications and contraindications but dropped substantially for advanced competencies, like image-guided navigation for biopsy and tumor ablation. This finding aligns with published training challenges in interventional ultrasound, where the cognitive load of integrating real-time imaging with needle manipulation is high and structured curricula are often lacking [43]. The ability to pinpoint such a specific competency gap enables program directors to target educational interventions, such as simulation-based training modules or dedicated rotations, rather than applying blanket remediation [44,45]. This function aligns with the principles of assessment for learning, whereby diagnostic information directly informs curricular improvement [46].

The incorporation of structured self-evaluation enhances the comprehensiveness and validity of the assessment. A statistically significant positive correlation between objective assessment and self-evaluation scores indicates that trainees demonstrate reasonable metacognitive awareness of their clinical competence. However, the presence of notable outliers—individuals who consistently overestimate or underestimate their performance by a clinically meaningful margin—raises important patient safety concerns and warrants targeted remediation [46,47]. Overestimation in image-guided procedures could lead to premature independent practice and an increased risk of complications [48]. Conversely, underestimation may unnecessarily erode confidence and limit procedural engagement [49]. By flagging such discrepancies, the CCAL enables supervisors to provide targeted feedback that fosters accurate self-assessment—a skill increasingly recognized as essential for lifelong learning and safe unsupervised practice [50,51]. This dual-modality reporting thus moves beyond traditional knowledge tests that offer no window into trainees’ metacognitive awareness [52].

Collectively, these characteristics position the CCAL as a practical tool for implementing CBME in CEUS training. Standardized, multisource data on cognitive competence can inform entrustment decisions, document training progression for accreditation, and facilitate cross-institutional quality assurance, thereby addressing the persistent reliance on subjective, uncalibrated assessments that currently dominate the field. While the CCAL explicitly does not assess hands-on procedural skills and should not be used in isolation for high-stakes credentialing, within its intended scope, it provides a transparent, evidence-based foundation for evaluating and advancing the cognitive component of liver CEUS competence.

This study had the following limitations. First, the validity evidence chain remains incomplete because of the absence of predictive validity verification—specifically, the correlation between test scores and real-time clinical decisions during procedures (eg, video-based assessments or supervisor ratings [53,54]). Second, easy items predominate (66% of the objective section), which may limit discrimination among advanced practitioners, although this was intentional given the target population (novices to competent operators) [55]. Third, standardization frameworks require further development as follows: norm-referenced benchmarks (eg, 60% as the passing threshold for first-year residents) remain unestablished, and institutional variations necessitate multilevel modeling to delineate confounding factors (eg, trainer expertise and equipment availability) [56]. Fourth, the CCAL was validated only in Chinese trainees across 25 Chinese hospitals; generalizability to non-Chinese populations (different curricula, certification, patient populations, and cultural self-assessment tendencies) remains unproven, requiring international validation [57,58]. Fifth, the relationship between self-perceived competency in the SES and objectively measured clinical performance needs further study, as we did not include direct observation (eg, live scanning or video-recorded exams)—the gold standard for criterion validity [41,59]. Future research should examine concurrent validity against supervisor ratings or objective structured assessment of technical skills (OSATS) during actual patient encounters [60]. Collectively, these advantages—content validity grounded in expert consensus, high reliability, robust construct validity, multicenter validation, a multidimensional construct, and fine-grained diagnostic capacity—constitute clear innovations over prior tools, while the acknowledged limitations delineate a focused agenda for future research.

The CCAL is the first organ-specific competency assessment tool for liver CEUS, innovatively combining objective knowledge testing with structured self-evaluation. Unlike generic or simulation-based CEUS tests, it is liver-specific and validated across a large multicenter cohort. It brings a standardized, refined framework that identifies specific competency gaps. Practically, the tool supports CBME by enabling data-driven assessment, enhancing rigor in CEUS examinations, guiding targeted curricular improvements, permitting cross-institutional benchmarking, and supplying evidence for specialty societies to update training standards, though further validation is required before high-stakes use. It must be emphasized that the CCAL evaluates theoretical competence exclusively and should not replace procedural assessment for certification decisions. Within its intended scope, the CCAL advances liver CEUS training toward greater standardization and evidence-based feedback. International validation and the establishment of criterion-related and predictive validity are essential next steps.