This qualitative description design study [35] was conducted using semistructured interviews to explore how to effectively communicate information about AI-CDSS in medical decision-making. Interviews were conducted among two groups: (1) clinicians, to understand their preferences and informational needs, and (2) AI experts, to gain insights into what experts consider necessary for clinicians to safely and appropriately use AI tools. For this study, AI expertise was established based on practical professional experience rather than a specific formal education in AI. The expert group consisted of individuals with direct experience in developing or implementing AI models for or in the health care sector. The inclusion of both groups was based on the assumption that informational needs go beyond the clinicians’ expressed preferences, but also expert-identified information is necessary for informed and safe use of AI in clinical settings. This article was written in accordance with the COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist (Checklist 1) [36].

Study participants were recruited by convenience and snowball sampling between July and November 2024. A target sample size of 8 participants per group was predetermined. This fixed quota was established to accommodate the limited availability of clinicians while ensuring balanced representation between the 2 groups. Consequently, recruitment was concluded upon reaching this target rather than based on theoretical saturation. Potential participants were identified through professional networks associated with the research team and targeted searches on LinkedIn (Microsoft). The potential participants were contacted by email, accompanied by study information detailing the study’s objectives, procedures, and confidentiality measures for the recording of the interviews. The interviewees interested in participating filled in a consent form and sent back an email to schedule a meeting, either online or face-to-face. Inclusion criteria required participants to be either (1) practicing clinicians in the Netherlands or (2) AI experts, entailing those with experience in developing and/or implementing diagnostic or treatment recommendation AI models within the health care setting. Participants who had less than 2 years of work experience were excluded.

The semistructured interviews were conducted by a researcher (SM), who is proficient in both Dutch and English. The interviews followed an interview guide (Multimedia Appendices 1 and 2), which contained different questions for clinicians and AI experts, each consisting of 4 parts (Table 1). The guides were translated to English for those with no sufficient command of the Dutch language by researchers SM and EA.

In part one, the participants were asked demographic questions concerning their age, medical or technical specialization, and work experience history. In the second part, clinicians were asked about their experiences with AI in the health care setting and about instructions they received before using AI. AI experts received questions about their experience in developing AI for clinical use and which information they provided to the end users. Part three contained questions about which information they would ideally either receive as clinical end users or provide as AI developers.

In the last part, all participants received an example of a Model Card [32], Model Facts [33], and the TRIPOD-AI checklist [22,24] (Multimedia Appendix 3). The formats were discussed sequentially: participants reviewed one example and provided immediate feedback on its information before moving to the next. This stepwise, fixed sequence ensured that each format was evaluated individually. These three formats were selected because they originate from distinct contexts and serve different purposes. Model Cards are intended for communication among developers and are therefore more technical in nature. In contrast, Model Facts are specifically designed for clinicians. The TRIPOD-AI checklist is a reporting standard on AI models in academic publications. To create these examples, a Bayesian network was selected from a public repository, as it provided a balanced level of complexity, being more complex than a linear regression but still more transparent than a deep learning model. One researcher (EA) completed each format based on the available model information. To preview a variation in communication styles, the same type of information was presented in bullet points in one format, while in another format, it was written in narrative form, and yet another displayed this information in a table. After examining the templates, participants were asked to provide feedback on the amount, structure, and comprehensiveness of the information listed in these examples.

SMcandidate in Clinical Data Science (CDS) at Maastricht University, working in a department that is generally tech-positive, which may have shaped her perspectives on AI in health care. Over the course of this research project, SM engaged with different paradigms, initially approaching research from a postpositivist stance but increasingly shifting toward a constructivist orientation. This perspective informs the study’s recognition that a single “ground truth” of information provision is not attainable and that knowledge is context-dependent and co-constructed.

HP is a PhD student researching trust in AI within CDS. MT is a PhD student at Maastro Clinic, researching risk communication and shared decision-making in breast cancer radiation therapy, with a focus on patient decision aids. EA is a PhD student within CDS investigating AI model descriptions based on the model representation standards and requirements. AvH is a PhD student in shared decision-making within radiation oncology (CDS and Maastro), with advanced training in qualitative methodology, and advised SM on qualitative methods. RF is an assistant professor in AI and clinical decision-making within CDS and Maastro and supervises the PhD candidates.

After data collection, recordings were transcribed verbatim and analyzed according to codebook thematic analysis [37] using the Atlas.ti software (version 25.0.1; ATLAS.ti Scientific Software Development GmbH). This method helped identify common themes and topics that were mentioned repeatedly in interviews. The analysis process started with open, inductive coding by a researcher (SM). As no predefined coding framework was used, codes were created to descriptively summarize the underlying meaning of the selected text segments. This initial phase resulted in the construction of a preliminary codebook. Two other researchers (MT and HP) independently applied codes from the codebook constructed by the first researcher (SM) to a subset of the interviews (n=4). Disagreements in codes were resolved by revisiting the transcripts to verify context and debating code definitions until consensus was reached. This iterative process changed the codebook by renaming codes, refining code definitions, ensuring consistent application of the codes, and merging overlapping codes. These codes were later grouped into general themes by identifying clusters within assigned codes for clinicians (Multimedia Appendix 4) and for AI experts (Multimedia Appendix 5). A member check of the analysis was not conducted to avoid influencing participants’ perspectives in potential follow-up research.

Before the start of the study, the Research Ethics form of the Faculty of Health, Medicine, and Life Sciences (FHML-REC, FHML/HDT/2024.022.) was evaluated, and ethical approval was granted. Prior to the interviews, all participants received an information letter detailing the study’s objectives and data handling procedures. Written informed consent was obtained from all participants for both participation and audio recording. To ensure confidentiality, all transcripts were pseudonymized, and any identifying information was removed. Participants were provided access to both their audio recording and the corresponding transcript, which they could review at any time. Participants received no compensation for their participation.

In total, 26 clinicians and 11 AI experts were invited to participate. Among invited clinicians, 8 declined due to lack of time, while 10 did not provide a reason. Three invited AI experts also did not provide a reason for nonparticipation. Between August and November 2024, 16 interviews were conducted with clinicians (n=8) and AI experts (n=8). Half of the clinician interviews were held through Microsoft Teams, and the other half were face-to-face, whereas most AI expert interviews were face-to-face (n=6), 2 AI expert interviews were held in English, while all remaining interviews, including all clinician interviews, were conducted in Dutch. Interviews lasted between 30 minutes and an hour. All participating clinicians practiced within tertiary care settings, employed either at academic medical centers or a specialized institute. Concerning prior experience with AI among clinicians, all reported having encountered AI tools in their professional practice. The majority (n=5) had experience with prediction models; some (n=2) had worked exclusively with AI-driven imaging applications or generative AI in the form of chatbots (n=1). Participant characteristics are presented in Table 2.

Through conducting thematic analysis, 4 key themes were identified that clinicians and AI experts expressed as essential for effective communication and use of AI-CDSS. These themes include (1) understanding the target population, which refers to the need to understand the data on which the model was trained; (2) clinically meaningful outcomes, focusing on how model reliability and clinical relevance are communicated; (3) warnings, limits, and safe use, highlighting the importance of clearly defining boundaries, limitations, and misuse risks; and (4) accessible design, which addresses the structure and presentation of content. These themes and subthemes can be found in Table 3 and are discussed in detail below.

A central theme expressed by most clinicians and experts was the ability to compare an individual patient during consultation with the training population for the AI model. This was fundamental for clinicians to evaluate whether the recommendations from the model would be reliable in a specific clinical context or not.

Initially, participants described this need as knowledge about the origin of the data, more specifically why, where, and how the data were collected. Clinicians stressed that this way, they could extrapolate whether this reflected local populations, which was especially important when the model was not locally validated.

Although the concern about the representativeness of the data and its relevance to local populations was present from the start, it became more concrete when participants reviewed the TRIPOD-AI example. Clinicians remarked that the section “class imbalance” was of great importance to make the comparison. However, the way it was written in the example felt overwhelming and too technical. Solutions for this were given, such as a table or graph to show the class imbalance, but also a digital and interactive solution for training data comparisons. AI experts already stressed the importance of providing information about the class imbalance before being shown the TRIPOD-AI example.

Later in the TRIPOD-AI document, the inclusion and exclusion graph for the training data, which is often provided in papers, was shown. Clinicians stressed that without understanding who was systematically left out of the training data, whether by design or by oversight, they felt the context to interpret the applicability lacked.

Finally, both clinicians and AI experts noted that information about the training data size was missing in the Model Facts example. This was, however, crucial information according to clinicians, since a great size of the training data is associated with better model performance.

Another key theme was the need for clear evidence of the performance and reliability of the model. Both groups stressed that for an AI-CDSS to be integrated into clinical practice, its performance must be demonstrably robust and easily interpretable. However, the manner in which the performance should be presented, as well as when it was acceptable, was not agreed upon by participants.

Clinicians, in particular, emphasized the importance of understanding what the reported performance metrics mean in practice. They noted that values like area under the curve (AUC), while technically informative, do not always translate into clinical insight without additional context. Some participants mentioned a desire for performance measures to align more closely with clinical outcomes, like sensitivity and specificity and their relevant thresholds. Clinicians and AI experts suggested that these measures can also easily be translated into statistical language such as false positives or negatives, which was thought to be more familiar to clinicians.

Experts mentioned that besides the performance measures, there should also be a focus on providing an uncertainty band to clinicians. This should give insights into when the model is handling an edge case. AI experts noted that in the AI development at the time of this writing, developers chase a high-performance score such as the AUC but might overlook clinical utility or model robustness by doing so.

Ultimately, while clinicians overall claimed to be familiar with performance metrics, they expressed that information on model validation often remains buried in dense technical language or tables. However, to be actionable in a clinical setting, they emphasized that information must be presented in a more structured and visually digestible manner, such as performance graphs and comparative visualizations. Besides presenting the performance metrics, clinicians expressed a desire to know when they could use an outcome to make a treatment decision, in other words, when it was “good enough” to rely on.

An important theme for both clinicians and experts was the need for clear warnings and boundaries for AI-CDSS use. Explicit indicators or limitations were desired, especially when the system may be applied in situations that fall outside its intended use or scope.

Participants praised that limitations and inappropriate uses were included in the presented reporting standards. However, many AI experts emphasized that these sections were often underused, poorly highlighted, and filled in too vaguely. AI experts also mentioned that biases and limitations of the model are difficult to formulate and fill in with only their background as developers. They mentioned that defining these biases and formulating necessary warnings for misuse of the model should be done in interdisciplinary settings. They also emphasized that misuses of the model can be identified during prototype testing or practice sessions.

Besides explicitly mentioning these limitations, participants also stressed that more active warnings should be included. AI experts mentioned that it would be safest to lock down the scope of the AI-CDSS within the software, such as limiting input variables or blocking predictions outside the scope. However, clinicians had varying opinions about this, with some arguing that they can decide on the model scope themselves. Another suggestion made by AI experts to warn clinicians about the correct use of an AI-CDSS was to provide pop-ups when using the model out of scope. However, opinions amongst clinicians were divided, with concerns regarding pop-up tiredness and the pop-up not being read or respected. Similarly, they thought that disclaimers were likely not to be read or forgotten about during the clinical use of the model.

Participants emphasized that how information is presented is just as critical as the content itself. Even when essential data about the system is available, it may be ignored or misunderstood in clinical settings if it is not accessible, clearly structured, or presented in a usable format.

First, the information needs to be available at all times while using the AI-CDSS, according to participants. They stressed that information should preferably be integrated within the software. However, it was also thought to be important to distribute information in different ways, such as presentations, workshops, and emails. According to participants, clinicians should also familiarize themselves with the model, which can be done through e-learning and practice sessions, which is important for retention of their understanding of the model.

Second, participants stressed the importance of the information being conveyed in an understandable manner. AI experts often spoke about the “language barrier” they experience when communicating about AI with clinicians, and clinicians confirmed this gap when they remarked on developer jargon. Participants suggested communicating in statistical language, such as P values or false positives, which is generally more in line with clinicians’ education and academic training. Others suggested shrinking this “language barrier” by providing general AI training for clinicians, for example, from the hospital or during medical training. Another familiar way of providing information would be in the form of papers, which clinicians are accustomed to reading. The TRIPOD-AI example was well received by participants, as its resemblance to the layout of traditional academic papers appeared to create a sense of familiarity. However, some clinicians stressed that they then might as well read the entire paper, while AI experts showed significant information was still missing due to word count restraints.

Third, clinicians often labeled themselves as being intrigued by AI and intrinsically motivated to gather information about new technologies. However, many clinicians also stressed that they themselves or their coworkers would not have enough time to read up about a model. This shows a difference in needs between clinicians, which calls for customizable or layered information provision. An example of how this works out in practice would be through providing information blocks or pop-ups on demand or linking to (validation) papers and other useful resources.

Finally, of all the examples, the Model Card example was perceived as the least desired method by the participants, while the first page of the Model Facts was the most appreciated. However, according to participants, Model Facts still missed essential information blocks that were provided in the TRIPOD-AI, such as class imbalance, inclusion and exclusion criteria, and population size. Participants recommended that this information be integrated as much as possible within the software to use the AI-CDSS.