This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.
Despite clear evidence that antibiotics do not cure viral infections, the problem of unnecessary prescribing of antibiotics in ambulatory care persists, and in some cases, prescribing patterns have increased. The overuse of antibiotics for treating viral infections has created numerous economic and clinical consequences including increased medical costs due to unnecessary hospitalizations, antibiotic resistance, disruption of gut bacteria, and obesity. Recent research has underscored the importance of collaborative patient-provider communication as a means to reduce the high rates of unnecessary prescriptions for antibiotics. However, most patients and providers do not feel prepared to engage in such challenging conversations.
The aim of this pilot study was to assess the ability of a brief 15-min simulated role-play conversation with virtual humans to serve as a preliminary step to help health care providers and patients practice, and learn how to engage in effective conversations about antibiotics overuse.
A total of 69 participants (35 providers and 34 patients) completed the simulation once in one sitting. A pre-post repeated measures design was used to assess changes in patients’ and providers’ self-reported communication behaviors, activation, and preparedness, intention, and confidence to effectively communicate in the patient-provider encounter. Changes in patients’ knowledge and beliefs regarding antibiotic use were also evaluated.
Patients experienced a short-term positive improvement in beliefs about appropriate antibiotic use for infection (
This pilot study provided preliminary evidence on the efficacy of the use of simulated conversations with virtual humans as a tool to improve patient-provider communication (ie, through increasing patient confidence to actively participate in the visit and physician attitudes about SDM) for engaging in conversations about antibiotic use. Future research should explore if repeated opportunities to use the 15-min simulation as well as providing users with several different conversations to practice with would result in sustained improvements in antibiotics beliefs and knowledge and communication behaviors over time. The results of this pilot study offered several opportunities to improve on the simulation in order to bolster communication skills and knowledge retention.
The economic and clinical consequences of antibiotic overuse are numerous and can lead to increased medical costs due to unnecessary hospitalizations [
Patients and health care providers often express frustration engaging in conversations about challenging or sensitive topics such as the overuse of antibiotics for treating viral infections within the clinic encounter. A review of the evidence shows that most antibiotics for viral infections are not prescribed as the result of clinical evidence but rather given in response to patient demands or lack of training in the appropriate guidelines among health care providers [
Building on this evidence, this pilot study examined whether a 10-15 min simulated role-play conversation with a virtual human, one for providers and one for patients, can facilitate the development of collaborative communication skills, knowledge, and confidence of patients and providers to effectively manage conversations regarding overprescribing of antibiotics for viral infections. Specifically, we hypothesized that use of the simulation would result in improvements in (1) patients’ knowledge and attitudes toward antibiotic usage; (2) patient activation; (3) patients’ and providers’ attitudes toward and preference for SDM; (4) providers’ perception of patient engagement in their self-management; and (5) patients’ and providers’ confidence, preparedness, and behavioral intention to engage in conversations about antibiotics.
This pilot study used a single group repeated measures design. Patients were recruited from the Bellevue Ambulatory Care practice, a New York City-based public hospital-based primary care practice that serves predominately low-income minority patient populations. Patients were recruited through their previous participation in completed trials with one of the study authors at Bellevue Hospital. Patients were sent letters inviting them to participate in the study and a telephone number to call for more information. Study staff also called patients inviting them to participate. Patients were excluded from the study if they (1) were unable to give informed consent, (2) refused to participate, (3) were unable to speak and read in English, or (4) age <18 years. Primary care providers were affiliated with NYU Langone Medical Center, providing care across four health care facilities: Bellevue Hospital, Gouverneur Health, Veterans Affairs NY Harbor Health Care System’s New York Campus, and the NYU Faculty Group Practice. An email was sent to providers inviting them to participate in the study. All patients and providers provided written informed consent approved by the Institutional Review Board of New York University Langone Medical Center.
The 15-min simulation was developed by Kognito in collaboration with a group of experts in motivational interviewing, patient engagement, medical education, and antibiotics. In addition, over 25 patients and providers provided feedback during the development phase before the beta version was piloted in this study.
For this study, both patients and health care providers engaged in a simulated conversation aimed at the overarching goal of improving collaborative patient-physician communication and SDM for antibiotic use. When patients accessed the simulation, they assume the role of Laura (the virtual patient) and engage in a conversation with Dr Wei (the virtual provider). Health care providers enter the simulation taking on the role of Dr Wei, who has to manage the conversation with the patient, Laura. At the beginning of the learning experience, participants view a brief movie that explains the backstory and their goals in the conversation. For example, participating providers are told that they will play the role of Dr Wei and conduct an office visit with Laura, a mom who has been coughing for a week and believes that antibiotics can help her get better quickly. Their goals in the conversation are to engage Laura in a conversation about her condition and health goals, and then to collaborate with her on a treatment plan that she understands and is motivated to follow all while expressing empathy, using plain language, checking understanding, and managing her repeating requests for antibiotics. Study patients who choose to play the role of the virtual patient are told that they will act as Laura in the conversation and decide what to say to the virtual physician, Dr Wei. Their goals in the conversation are to provide Dr Wei with a clear understanding of Laura’s illness, ask Dr Wei questions so that they understand everything he says, learn about the proper use of antibiotics, and to work with Dr Wei on a plan they both are satisfied with (
At the end of the 15-min simulation, users view a brief movie where the virtual coach provides them with feedback on the decisions they made in the conversation. Then, they are provided with a performance dashboard that includes more detailed feedback on their performance. The information in the dashboard is based on the exact dialog decisions made by the learner during the conversation (
The simulation was designed using the Kognito Conversation Platform, an innovative group of development, delivery, application programming interface (API), data collection, and analytic technologies that integrates a behavior change model that employs the principles of neuroscience, social cognition, adult learning, applied game mechanics, and storytelling [
In each simulation, a learner enters a risk-free practice environment, assumes a role (ie, health care provider, patient), and engages in a conversation with intelligent, fully animated, and emotionally responsive virtual characters that model human behavior. Virtual humans are coded to possess an individual personality and memory, and adapt their behaviors to the decisions of the learner throughout the conversation. Learners communicate with the virtual human by selecting from a dynamic menu of dialog options. Each option represents a specific conversation tactic based on communications skills that may be more or less effective or ineffective in accomplishing the learner’s goal (see
The relationship between dialog decisions made by the learner and the response of the emotionally responsive virtual human is controlled by a set of mathematical behavioral models and algorithms specifically designed to simulate real interactions with “types” of people presenting particular personality traits or conditions (ie, cold or cough). The algorithms ensure that learners are repeatedly exposed to target conversations and behavioral patterns as a way to develop skills.
Dr Wei talking with his patient Laura.
Performance dashboard provided to users who played the role of the provider. RWJF: Robert Wood Johnson Foundation.
Self-report measures were administered at the baseline (presimulation), immediate postsimulation, and 1-month follow-up to participating patients and providers. The measures were designed to assess key aspects of patient-provider communication targeted in the simulation (ie, SDM), patient activation, patient antibiotic knowledge and beliefs, behavioral intentions, preparedness, and confidence to engage in challenging conversations, and satisfaction with the simulated encounter. In addition, exit interviews were conducted postsimulation to determine acceptability of the approach.
Descriptive statistics were generated for baseline patient and provider characteristics. Generalized Linear Models (GLM) using repeated measures analysis were used to analyze the pre-, post- and 1-month follow-up survey measures. Analyses were first run for the total sample and then repeated for the subset of participants, who were in the lower PAM levels at the baseline visit (presimulation) (PAM level 1 or 2; n=13). Independent
We recruited a total of 69 participants (35 providers and 34 patients); with a retention rate of 99% (68/69) (one patient was lost at follow-up). As shown in
At the presurvey assessment, the mean score on the beliefs subscale was 1.85 (range: 1-4; lower scores indicate more accurate beliefs). Immediately after completing the simulation, patients were significantly more likely to possess accurate beliefs about antibiotic use (mean change −0.11,
The mean PAM score for the total patient sample was 63.60 (SD 15.39) at the presurvey assessment, 62.61 (SD 13.35) at the immediate postsurvey assessment, and 62.83 (SD 14.57) at the 1-month follow-up. Results of the repeated measures analysis showed no significant differences in PAM score across time (
At the previsit assessment, patients not only tended to prefer their doctor make the final decisions about their care (mean 2.66, range 1-4;
After completing the simulation, patients reported feeling prepared to actively participate in a future medical visit with their provider (mean 3.35, range 1-4), which did not significantly change at the 1-month follow-up (
Participant characteristics.
Characteristics | Mean (SDa) or n (%) | |||
Patient characteristics | ||||
Age | 57.62 (14.57) | |||
Male | 16 (47) | |||
7th to 8th grade | 11 (32) | |||
High school | 23 (68) | |||
Less than high school | 1 (3) | |||
high school/GEDb/technical | 9 (27) | |||
Some college | 8 (24) | |||
College degree | 15 (46) | |||
Private | 9 (27) | |||
Medicare | 10 (29) | |||
Medicaid | 10 (29) | |||
None | 2 (6) | |||
Other | 3 (9) | |||
Provider characteristics | ||||
Age | 40.34 (9.44) | |||
Male | 19 (54) | |||
Physician/family medicine | 12 (34) | |||
Physician/internal medicine | 22 (63) | |||
Nurse practitioner | 1 (3) | |||
Attending | 26 (74) | |||
Resident | 9 (26) | |||
MDc/DOd | 34 (97) | |||
NPe | 1 (3) | |||
Mean years at clinic | 7.48 (6.53) | |||
A little | 8 (42) | |||
Some | 7 (37) | |||
Quite a bit | 4 (21) |
aSD: standard deviation.
bGED: General Educational Development.
cMD: Doctor of Medicine.
dDO: Doctor of Osteopathy.
eNP: nurse practitioner.
Comparisons of patient survey responses across all three time points.
Patient measures | Response range | Presimulation |
Postsimulation |
1-month follow-up |
||
PAMb score | 0-100 | 63.60 (15.39) | 62.61 (13.35) | 62.83 (14.57) | 1.86 | .18 |
Antibiotic beliefs | 1-4c | 1.85 (0.42) | 1.74 (0.41) | 1.76 (0.48) | 14.10 | .001 |
Antibiotic knowledge | 1-4 | 3.01 (0.42) | 3.26 (0.43) | 3.08 (0.46) | 31.16 | <.001 |
Decision-making preference | 1-4c | 2.66 (0.73) | 2.85 (0.73) | 2.83 (0.78) | 1.94 | .17 |
Patient-provider orientation: shared power | 1-4c | 2.06 (0.44) | 2.01 (0.40) | 2.02 (0.41) | 1.86 | .18 |
Preparedness to act | 1-4 | - | 3.35 (0.59) | 3.25 (0.53) | 0.74 | .47 |
Behavioral intent | 1-4 | - | 3.24 (0.57) | 3.31 (0.52) | 0.81 | .43 |
Confidence to act | 1-4 | - | 3.32 (0.61) | 3.34 (0.57) | −1.62 | .12 |
aSD: standard deviation.
bPAM: Patient Activation Measure.
cLower scores indicate more accurate beliefs about antibiotics and shared power in the clinic encounter.
Results of the subanalysis showed that patients with low PAM scores demonstrated similar improvements in accurate beliefs regarding antibiotic use at the postsurvey (mean 2.04) and 1-month follow-up (mean 2.01;
Patients with low PAM scores also agreed that they felt better prepared to ask their doctor questions, express their concerns, and discuss treatment options after completing the simulation (mean 3.24) and at the 1-month follow-up (mean 3.33). There was no significant differences in the mean scores across time (
Comparisons of low PAM patient survey responses across all three time points (n=13).
Patient measures | Response range | Presimulation |
Postsimulation |
1-month follow-up |
||
PAMb score | 0-100 | 39.00 (5.66) | 40.90 (6.79) | 40.90 (5.72) | 0.32 | .58 |
Antibiotic beliefs | 1-4c | 2.40 (0.20) | 2.04 (0.27) | 2.01 (0.44) | 10.88 | .01 |
Antibiotic knowledge | 1-4 | 2.86 (0.25) | 3.23 (0.39) | 2.89 (0.23) | 28.53 | <.001 |
Decision-making preference | 1-4 | 2.58 (0.49) | 2.69 (0.48) | 2.64 (0.71) | 0.45 | .52 |
Patient-provider orientation: shared power | 1-4c | 2.40 (0.20) | 2.27 (0.16) | 2.17 (0.30) | 17.19 | .002 |
Preparedness to act | 1-4 | - | 3.24 (0.71) | 3.33 (0.27) | −0.55 | .60 |
Behavioral intent | 1-4 | - | 3.25 (0.42) | 3.50 (0.32) | −2.24 | .08 |
Confidence to act | 1-4 | - | 2.93 (0.52) | 3.33 (0.50) | −2.34 | .07 |
aSD: standard deviation.
bPAM: Patient Activation Measure.
cLower scores indicate more accurate beliefs and shared power in the clinic encounter.
Before engaging with the simulation, providers held high positive beliefs about patient’s involvement in their self-management (mean 78.19, range 0-100). These ratings remained high (mean 76.47) immediately after completing the simulation as well as at the 1-month follow-up (mean 77.15). There were no significant differences across time (
Immediately after completing the simulation, participating providers felt that doctors would be better prepared to have an effective conversation with patients (mean 3.45), actively engage patients in the conversation (mean 3.48), and feel confident in their abilities to engage and respond to patients’ biomedical and psychosocial concerns (mean 3.33). Similar to the postsimulation results, providers continued to agree that doctors would be better prepared, confident, and able to effectively engage in conversations about antibiotics, respond to patient emotion, and invite patients to be active participants in the medical encounter. All providers also felt doctors would be more prepared to have an effective conversation about antibiotics with patients (
Before completing the simulation, providers felt that they already engaged patients in the shared decision-making process (mean 3.24, range 1-4) and that decision-making process should be a collaborative process (mean 1.82, range 1-4; lower scores indicate more collaboration). Immediately after completing the simulation, there was a significant increase in providers’ attitudes about patients’ collaborative involvement in the shared decision-making process (mean 1.62,
Comparisons of provider survey responses across all three time points.
Provider measures | Response range | Presimulation |
Postsimulation |
1-month follow-up |
||
CS-PAMb | 1-100 | 78.19 (13.02) | 76.47 (13.34) | 77.15 (14.44) | 0.11 | .74 |
Shared decision-making | 1-4 | 3.24 (0.38) | - | 3.35 (0.38) | 1.61 | .21 |
Patient-provider orientation: shared power | 1-4c | 1.82 (0.37) | 1.62 (0.32) | 1.82 (0.39) | 8.03 | .01 |
Satisfaction | 1-4 | - | 3.25 (0.28) | - | - | - |
Preparedness to act | 1-4 | - | 3.45 (0.50) | 3.34 (0.44) | −0.36 | .73 |
Behavioral intent | 1-4 | - | 3.48 (0.47) | 3.42 (0.44) | −0.84 | .42 |
Confidence to act | 1-4 | - | 3.33 (0.60) | 3.31 (0.41) | −1.30 | .22 |
aSD: standard deviation.
bCS-PAM: Clinician Support for Patient Activation Measure.
cLower scores indicate shared power in the clinic encounter.
In cross-sectional analysis, comparing the data by provider rank (resident vs attending), there were no significant differences between the groups before completing the simulation. At the 1-month follow-up, attendings were more likely to agree that patients should be actively involved in the shared decision-making process (mean 3.38 vs 3.28,
This pilot study provided a unique opportunity to evaluate a brief 15-min simulated role-play conversation with a virtual patient or provider designed to promote effective communication and collaborative decision-making between health care providers and patients in order to improve intermediary health outcomes, including over-prescribing of antibiotics. Although there were not changes in activation scores for patients, the findings indicate that patients’ experienced short-term positive benefit on beliefs about antibiotic use and a positive, albeit intermediate, impact on patients’ knowledge about antibiotics. Patients with lower levels of activation, in particular, exhibited positive, short-term benefits in increased intent and confidence to discuss their needs and ask questions in the clinic visit and attitudes about engaging in shared decision-making with their provider. In particular, 79% of patients who saw their doctor after completing the simulation reported that it helped them in talking with their doctor. The results also suggest small immediate gains in providers’ attitudes about shared decision-making. Providers also felt that doctors would be better prepared and confident to have collaborative conversations with patients as well as create an environment conducive to active patient involvement in the encounter after completing the simulation. In particular, 77% of providers reported that the simulation had a positive impact on the way they now communicate with patients, 65% indicated that it helped them have a conversation with a patient about antibiotics, and 94% said they intent to further invite patients to ask questions and participate.
These findings support the role of utilizing simulated role-play conversations with virtual humans for the purpose of improving communication and relational (ie, empathy) skills in a variety of domains. Specifically, previous research has identified needed skill frameworks, training, practices, and elements of effective relationships that can be integrated in digital interventions to improve social emotional and communication skills, and drive positive behaviors [
A strength of this study was the use of an evidence-based simulation that leveraged virtual humans to improve users’ social emotional skills, empathy, motivational interviewing, and the use of sound communication tactics (ie, shared decision-making) that have been linked to sustained behavior change [
Several limitations of the study are worth noting. First, this was a single-group pre-post study. The lack of a control group limits our ability to attribute changes in participant’s behavioral intentions, attitudes, and perceptions of communication exclusively to the simulation. Moreover, it is possible that increased awareness from completing the presimulation assessments diminished our ability to detect significant changes in the postsimulation assessments. However, the primary focus of this pilot study was to establish the preliminary efficacy of this approach and not statistical significance. Relatedly, changes in scores from postsimulation to the 1-month follow-up may reflect a decay effect over time and not long-term change. The knowledge gained from this project will be used to develop the evidence for a larger randomized control trial. Second, the small sample size prohibits making any statistical inference generalizations about the study measures reliability (ie, alpha scores) and requires replication in a larger sample. Third, since the primary focus of this study was the use of a tablet-based simulation, a selection bias may be present whereby patients with low levels of computer literacy or poor vision may be underrepresented. To mitigate this risk, we implemented several strategies to increase the generalizability of this approach to all patient populations including the use of audio for the dialog and ensuring that the text was written at or below a 6th-grade reading level. Moreover, only 20% of individuals (13 patients and 4 providers) contacted to participate in the study declined, of which there were no differences in demographics between participants and nonparticipants; the most common reason for both patients and providers was lack of time. Fourth, participants were only permitted one opportunity to practice a one 10-15 min role-play conversation. Normally users have unlimited opportunities to practice multiple different conversations within a single simulation as well as opportunities to engage in these practice over time. Another important limitation is that the study design neither allows for definitive conclusions about whether the simulation affected patients’ actual level of engagement in their care nor whether shared decision-making as opposed to patient engagement was the primary communication strategy through which change occurred. Future studies should seek to disaggregate patient engagement from shared decision-making to elucidate the specific elements of communication that are associated with changes in patients’ knowledge and beliefs about antibiotic use. Moreover, future research should determine which elements of shared decision-making (ie, adequate information-exchange, taking into account patients’ values and preferences) are needed to improve patient outcomes. Preliminary results from this study suggest that patient-provider communication does not necessarily need to include patient participation in the final decision-making in order to be effective.
Finally, the external validity of our findings may be limited as a high percentage of the study participants (82%) were highly activated (as determined by PAM scores) at baseline (presimulation), even though the target audience for the simulation content was individuals with lower activation scores. This left little room for growth and could offer a plausible explanation for any nonsignificant findings. It is also plausible that the lack of significant findings was due to a baseline effect due to high levels of awareness about the problems with the overuse of antibiotics by patients and providers at the presimulation assessment.
In conclusion, this pilot study provided preliminary evidence on the efficacy of a simulation to improve patient-provider communication for engaging in collaborative conversations and decision-making on short-term improvements in patients’ knowledge and beliefs about antibiotic use. Future research should examine whether repeated opportunities for patients to use the simulation and practice the skills being taught may lead to sustained improvements in knowledge, beliefs, and behaviors. Moreover, although providers of all levels derived some benefits from the simulation, residents and medical students may experience the greatest gains in improving their communication skills for challenging conversations and attitudes about patient-centered care.
America Board of Internal Medicine
application programming interface
Clinician Support for Patient Activation Measure
Generalized Linear Models
Medical Communication Competence Scale
Patient Activation Measure
Provider Orientation Scale
Robert Wood Johnson Foundation
standard deviation
shared decision-making
This research was funded by the Robert Wood Johnson Foundation (PI: Goldman). We would like to acknowledge Amy O’Neal, Seth Bleecker, Sutton King, Matthew Chin, Josh Harrison, Victoria Degtyareva, and Diana Rosenthal for their assistance in implementing the study.
AS acquired, analyzed, and interpreted data, and drafted the manuscript as well. GA acquired data and critically reviewed the manuscript. JH interpreted data and critically reviewed the manuscript. RG developed the simulation platform and contributed to all critical revisions of the manuscript.
Ron Goldman is a Co-Founder and CEO of Kognito. Glenn Albright is a Co-Founder and Director of Research at Kognito. Judith Hibbard is a consultant to and an equity stakeholder in Insignia Health.