To address these questions, we developed a test set of feedback messages that a software application could generate. We formatted these as brief email messages, but designed them as “least common denominator” content that could also be delivered via other channels for feedback, such as clinical quality dashboards.

In the absence of instruments with validity evidence for assessing health care professional feedback preferences, we created an experimental design to elicit preferences that would expose participants, who were anesthesia providers, to contrasting message versions. To enable measurement validity assessment, we developed performance scenarios in which the same motivating performance information and display characteristics could be repeated in contrasting messages.

This study was approved by the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board (IRB-HSBS HUM00167426). All participants provided consent to participate and were informed about the ability to opt out of the study. No participant identifiers were collected with the research data for this study, preventing the linking of participants’ responses with their identities. No incentives for participation were provided. We offered participants an opportunity to receive a copy of the study results upon completion.

We developed the email message test set iteratively in three phases: (1) knowledge modeling, (2) display format development, and (3) message set development (Figure 1).

We designed a within-subjects, repeated measures study of anesthesia provider preferences for precision feedback using a test set of prototype email messages printed on paper. We created 2 versions of the test set with alternate display formats for each message (group A vs group B) to enable randomization of the pairing of display format with motivating information. We created a document containing all of the email messages in the test set (Multimedia Appendix 2). We printed paper copies of the messages and organized them into packets in varying order for a paper card selection task. Based on our experience, we estimated that a sample of more than 30 participants would provide adequate power to detect meaningful differences in summary statistics and internal consistency of preferences.

We recruited anesthesia providers from a single academic medical center in the midwestern United States. Anesthesiologists (physicians) and certified registered nurse anesthetists (CRNAs) were eligible to participate. A member of the study team recruited anesthesia provider participants by email. All participants received monthly anesthesia provider feedback emails from MPOG.

Upon enrollment, we scheduled a 15-minute proctored video call with each participant and sent them a paper packet with email prototypes before the call. Participants were randomized to receive a paper packet of messages from either group A or group B of the message test set, each of which contained 16 email messages grouped in 4 packets of 4. Each packet of 4 messages contained alternate message formats for 1 of 4 performance scenarios, with balanced message formats and performance information across the 4 scenarios. We created a questionnaire to collect data from participants about their preferred emails using Qualtrics (Qualtrics, Inc). We created 2 versions of the questionnaire (A and B), 1 for each message group to be used based on the participant’s random assignment at the time of enrollment. At the time of enrollment, we also instructed participants to have a desk space or table available for placing printed email messages in front of them, and to wait to open the packets until asked to do so during the video call.

At the start of the proctored video call, a research team member introduced the study, confirmed the participant’s preparation, and provided a link to the questionnaire. During the completion of the questionnaire, the participant repeated a preferred email message selection task 4 times, following the instructions in their packet, once for each performance data scenario. The questionnaire software randomized the scenario presentation order. We described the scenarios as hypothetical performances that the participant could imagine as being their own. At the start of each scenario, participants were asked to find the corresponding set of emails, identified with a cover sheet. Participants were then asked to lay out all 4 of the printed email messages for that scenario in front of them. Next, participants read each message and selected their preferred message. After selecting a preferred message, participants responded to the following statement: “I gained information from this email that would benefit my practice.” We adapted this question from an instrument with good validity evidence for assessing the usability of feedback displays [39]. Responses were collected on a 5-point Likert scale, ranging from strongly disagree to strongly agree. The survey questions did not ask directly about preference for information content or display format. Instead, participants’ preferences were inferred through the types of content and display format that the selected message contained. After participants completed the questionnaire, we conducted brief interviews and collected qualitative data that were analyzed separately and will be reported elsewhere.

To identify preferences, we analyzed 2 characteristics of the selected messages: motivating information (including temporality type and comparator type) and display format. We summed the selected messages with each type of motivating information and display format and calculated descriptive statistics for these sums (Q1). To investigate the consistency of participants’ preferences, we calculated the Cronbach α for each preference characteristic in participants’ selected messages across the 4 performance scenarios (Q2). We used descriptive statistics to assess relationships between participants’ preferences and the characteristics of the 4 performance scenarios, including performance level (high vs low) and trend presence (present vs absent). Similarly, we considered relationships between participants’ preferences and their professional background using descriptive statistics (Q3).

To understand participants’ perceptions of the potential benefit of precision feedback to their clinical practice, we analyzed ratings of perceived benefit for selected messages using descriptive statistics (Q4). We conducted analyses using R and Google Sheets (Google LLC).

An overall preference for multiple time intervals (ie, time series) was apparent, with 110 of 140 (79%) messages being selected over those with a single time interval (ie, time point) (Q1a). Preferences for display format were highly varied, with selected messages being equally distributed between bar charts versus other formats (Table 4) (Q1c). Preferred messages were also highly varied in their comparators, with 74 of 140 (53%) preferred cards containing explicit target comparators (ie, organizational goals not dependent on population performance) (Q1b), but our assessment of the consistency suggests that the comparator result was not reliable as a preference characteristic (see Q2 below).

Participants’ preferences were consistent for temporality (α=.85) and display format (α=.80). For performance comparators, participants’ selected messages were negatively correlated (α=−.40), indicating an absence of consistency, perhaps from an incorrect measurement model [40]. We consider this result to be an artifact of the study design, given that our message test set balanced several characteristics and created opportunities to select them in combination. We anticipate that comparators were not salient for participants, relative to the visual display and temporality characteristics; therefore, we are unable to draw conclusions about preferences for comparators.

Participant preferences for temporality and display format did not appear to depend on the messages’ performance level, with relatively similar means for the selection of each type of message content. Similarly, these preferences did not appear to vary with the presence or absence of performance trends (Table 5).

Preferences for temporality and display format varied with participants’ professional background (Table 6). Some professional role-based differences in means were apparent, such as a higher preference for time point messages among CRNAs than anesthesiologists (mean message characteristics preference 1.18, SD 1.59 vs mean message characteristic preference 0.56, SD 1.10). However, a majority of CRNAs preferred time series messages, and all message characteristics were repeatedly observed in selections by participants from both professional background–based groups.

Participants’ ratings of perceived benefit from all precision feedback messages were positive, with a mean rating of 4.27 (SD 0.77). Although positive overall, the anesthesiologists’ ratings were lower than the CRNAs’ ratings (mean rating 4.08, SD 0.85 vs mean rating 4.47, SD 0.61). Ratings for messages did not appear to vary across performance levels or with trends (Table 7). Average ratings of perceived benefit were similar across message content characteristics. One exception to this was for explicit target comparators, which appeared to receive slightly higher ratings (mean rating 4.38, SD 0.7) over benchmark comparators (mean rating 4.15, SD 0.83).

In this study, we found that anesthesia provider preferences for motivating information and display format varied, which suggests that individual difference characteristics may represent a barrier to improving the effectiveness of feedback interventions. Across a set of 4 diverse performance scenarios, we observed preference variability that precision feedback could better address than one-size-fits-most feedback in this anesthesia provider population.

We observed consistency in participant preferences for the temporality of motivating information and for display format. Even though a large majority of participants preferred messages with time-series information, the participants who preferred time-point messages reliably selected them. The consistency of preferences for display format was similar, and also more varied, with exactly half of participants choosing bar charts over other visual displays. We also did not observe differences in preferences associated with performance scenario characteristics or professional background that could be used to design one-size-fits-most feedback interventions.

While participants exhibited diverse preferences, their ratings of the benefit of the messages were consistently high across performance scenarios. These findings suggest that anesthesia providers would welcome the enhancement of feedback interventions with precision feedback that prioritizes motivating information. These findings are important because they point to a possible approach for improving audit and feedback that can leverage both high and low performance, as well as increasing or decreasing trends, to prioritize performance feedback.

To our knowledge, this is the first quantitative study of preferences for clinical performance feedback. As an exploratory study, the findings primarily demonstrate the existence of differences in preferences for feedback, rather than speaking to the significance of their role in the success of clinical performance feedback. Our findings are related to CP-FIT, which recognizes that health professional knowledge and skills for engaging with feedback can be important factors for the success of feedback [38]. Differences in feedback preferences could be driven by differences in health care professionals’ knowledge and skills related to the interpretation of performance data. For example, participants’ variable and consistent selection of messages could be related to their graph literacy skills [41,42]. Precision feedback could be used to accommodate these and other individual differences by enabling health professionals to configure their feedback delivery and display, which further holds potential to increase feelings of ownership of feedback. By prioritizing motivating information according to recipients’ preferences, precision feedback could be a strategy for reducing the cognitive load required by health professionals to recognize and assess the priority of learning opportunities. Precision feedback has also potential to improve feedback cycle completion by delivering information that is more likely to be perceived and accepted, resulting in increased formation of intentions to sustain or improve performance. In terms of CP-FIT, precision feedback can be understood as an approach for prioritization of feedback messages that are more likely to result in successful completion of the feedback cycle.

Our findings are aligned with the idea that positive feedback can be effective for learning and improvement [13], as well as sustainment of high performance. It is noteworthy that participants rated precision feedback messages as beneficial even when performance was high, such as the messages “you are a top performer” or “you reached the goal.” This finding points to the possibility that a key function of feedback may be to motivate recipients through appreciation of accomplishments [43], including recognition of high performance, in addition to motivating recipients to learn to improve.

As an exploratory study for a novel type of feedback intervention, there are several important limitations for this study. The poor consistency of preferences demonstrated for performance comparators suggests that participants did not meaningfully differentiate between peer-based benchmarks and explicit targets, as presented in the message test set. This may be a function of the labels used for the comparators message test set, and during the study we discovered that some of the printed messages contained the abbreviation “ave” instead of “avg” for the peer average comparator. Competing explanations are that (1) anesthesia providers equated the value of both comparator types or did not perceive them as fundamentally different, and (2) that this characteristic was less salient than the others, such that its significance was negligible.

Using performance scenarios based on synthetic performance data may have introduced bias in participants’ responses. However, the consistency of participant preferences for temporality of motivating information and display format suggests that this bias was not significant. Nevertheless, our study design assessed preferences within types of motivating information (eg, high and improving performance or low and worsening performance) that were presented with unambiguous motivating information, such as trends showing marked improvement or worsening. As such, our results do not address the appropriateness of using performance scenarios to elicit the strength of anesthesia provider preferences directly; rather, they primarily demonstrate the existence of individual differences as an exploration of factors that may moderate the influence of feedback on health care professional learning and improvement.

We asked participants to rate the perceived benefit of messages that they had already selected as their preferred message, which may have resulted in positively biased ratings. Furthermore, we used a single performance measure for all messages (avoiding postoperative nausea and vomiting) that may not be representative of other performance measures, both in terms of perceived benefit and preferences for motivating information. We did not evaluate feedback about clinical outcome measures, which may have resulted in a different preference profile across this population. We also did not evaluate participants’ skills or knowledge to engage effectively in feedback, which is a recognized factor [38] that may have resulted in further insight into participant preferences.

Additional limitations include the context and nature of the preference elicitation task, which was done in a video call with paper prototypes and thus differed from the context of email use in health care organizations. When designing this study, we chose to use email messages printed on paper because we could not identify a remote, video call–proctored approach that would allow participants to consider 4 different messages types in the same field of view on their personal or work computer without a risk of technical complications from participants’ particular computer monitor and device configurations.

Our model of preferences in this study was linear and static and assumed that available information was complete, but anesthesia provider preferences may be nonlinear, dynamic, and depend on missing information that we did not consider. When designing the test set of messages, we paired the text-only display format consistently with time-point information, and line charts with the time-series format. As such, preferences for line charts and text-only display formats were not independent from temporality. We recruited anesthesia providers from a single academic institution whose population is not necessarily representative of other anesthesia provider populations. We did not recruit any anesthesia providers who identified as Black or Hispanic, increasing the likelihood that our results are racially and ethnically biased toward the perspectives of anesthesia providers who identify as White and non-Hispanic. In spite of all of these limitations, we note that the variability that we observed demonstrates that preferences were nonuniform in this small population, which suggests that a one-size-fits-all solution may be inadequate for feedback reporting to anesthesia providers more generally.

We anticipate that preference clusters may exist and may be identifiable in studies that are better powered to detect such differences. Such clusters could be used to develop profiles for precision feedback, such as profiles for anesthesia providers who prefer text-only messages about low performance or those who prefer visualization of performance changes (ie, trends) using time-series displays in line charts. Future studies may be able to detect preference clusters to better understand the diversity of preferences for performance feedback across a larger anesthesia provider population that is more racially, ethnically, and geographically diverse. Furthermore, we would welcome studies that aim to better understand the diversity of anesthesia provider preferences in association with additional anesthesia provider characteristics, such as duration of professional experience, clinical setting, and organization type.

Clinical performance feedback to health care professionals has potential to support continuous learning and influence practice, but this potential is frequently not achieved. By prioritizing motivating performance information based on the preferences and needs identified for a health care professional population, precision feedback may increase the effectiveness of clinical performance feedback for health care professionals’ continuous learning and resulting quality improvement. Among a sample of anesthesia providers, preferences for precision feedback were varied, yet consistent within participants. Furthermore, participants’ perceived benefits of precision feedback messages were observed to be high across a diverse set of performance scenarios. Based on these findings, it appears that precision feedback holds potential to improve support for health care professionals’ continuous learning.