Assessing Pharmacists’ Use and Perception of AI Chatbots in Pharmacy Practice: Cross-Sectional Survey Study

Introduction

Artificial Intelligence Chatbots

In November 2022, the artificial intelligence (AI)–based large language model chatbot ChatGPT was launched [1]. It quickly gained popularity, amassing over 100 million monthly users in just 2 months [2]. Since the release of ChatGPT, several other AI chatbots have debuted, including Gemini, Microsoft Bing AI, and Copilot. This expansion in AI chatbots has led to the diversification of their use in fields outside of the technology industry. ChatGPT has been explored for use in health care practice, education, and research, and specialized chatbots such as OpenEvidence and Dougall GPT have emerged, catering specifically to clinicians and health care workers [3-6].

Use of Artificial Intelligence in Healthcare

A survey evaluating the use and perceptions of ChatGPT in health care professionals was conducted at Northwestern University [7]. In addition to gathering information regarding applications of ChatGPT in health care, the survey focused on perceptions related to use in health care research, education, and practice. Participants reported uncertainty about the use of ChatGPT due to its recent release, although many respondents indicated an interest in future use. Similarly, in a survey of health care workers in Saudi Arabia, participants expressed concerns about ChatGPT use in health care, including lack of credibility and concerns about inaccurate medical information [8]. Despite these concerns, most respondents still anticipated a positive impact of ChatGPT on the future of health care practice, including medical decision-making, patient and family support, and medical research appraisal. Studies conducted in Jordan, Saudi Arabia, and the United Arab Emirates have reported similar findings among pharmacists [9-11].

These findings were reinforced in a 2023 systematic review that examined 60 studies focusing on the applications of and concerns about ChatGPT in health care practice, education, and research [6]. Potential uses for ChatGPT in practice included clinical documentation, creation of personalized care plans, improved health literacy, and provision of patient education. In education, ChatGPT was used in the creation of personalized learning tools and writing clinical cases. In research, ChatGPT was found to be beneficial in improving writing efficiency. However, in nearly all the studies, there were reported concerns with ChatGPT, including ethical, legal, and copyright issues; inaccuracy; and limited reproducibility. Specific to pharmacy, the use of ChatGPT has been explored with respect to drug information, medication therapy management, patient education, and adverse drug reaction assessment, with the general consensus that it has potential as a supplementary tool but is not yet able to handle complex problems [12-15].

On the basis of the findings from previously conducted surveys and research studies, it is evident that there is current and future potential for the use of AI technology in health care. However, information about the opinions and perceptions of pharmacists regarding the utility of AI chatbots in practice is limited. Pharmacists have been poorly represented in most surveys of health professionals conducted to date, and most studies regarding pharmacist perceptions have been conducted outside of the United States. Understanding the current perceptions, use, and barriers to use of chatbots will help inform and identify future roles for chatbots in pharmacy practice. Pharmacy preceptors are at the forefront of practice and education in the United States; therefore, this study aimed to assess current perceptions and use of AI chatbots in pharmacy practice from the perspective of pharmacist preceptors and determine factors that may influence the use of AI chatbots according to the Technology Acceptance Model Edited to Assess ChatGPT Adoption (TAME-ChatGPT) tool [16].

Methods

Overview

A cross-sectional survey was conducted using a convenience sample of pharmacy practice preceptors from Purdue University College of Pharmacy, University of Michigan College of Pharmacy, Wayne State University Eugene Applebaum College of Pharmacy and Health Sciences, University of Illinois Chicago Retzky College of Pharmacy, and Butler University College of Pharmacy and Health Sciences. Current pharmacist preceptors were recruited through their respective offices of experiential education. Pharmacists were excluded from the study if they had not precepted students in the previous year given the recent introduction of AI chatbots or if they did not practice in one of the affiliated states of Illinois, Indiana, or Michigan. Participants were instructed to only take the survey once.

Questionnaire Design and Administration

The survey instrument was built in Qualtrics XM (Qualtrics International Inc) and was primarily based on the TAME-ChatGPT survey tool, a validated tool adapted from the technology acceptance model (TAM) for assessing health care students’ attitudes toward ChatGPT [16]. Before the first survey item, respondents were asked a screening question to ensure that they met the study inclusion criteria. The first section gathered information regarding pharmacists’ practice setting and current use of AI chatbots. The second section contained questions from the TAME-ChatGPT that were adapted to detect an association with different factors of the TAM (perceived ease of use, perceived risk, technology or social influence, anxiety, perceived usefulness, and behavior) and the use of AI chatbots. The survey concluded with questions regarding demographic information and tasks in which preceptors would recommend the use of AI. The instrument was pilot-tested by 3 faculty members to ensure adequate formatting, comprehension, clarity, and completeness of the survey. The final survey instrument is available in Multimedia Appendix 1.

An email invitation to complete the survey instrument was sent to 1877 pharmacy preceptors on February 12, 2024. The survey link was accessible until April 22, 2024, and 3 reminder emails were sent out before the survey closed. Only fully completed surveys were included in the final analysis.

Data Analysis

Data were described using means and SDs for continuous variables and medians and frequencies for nominal variables. Univariable analysis was conducted using the student 2-tailed t test for continuous variables and chi-square test for nominal variables. Normality was assessed using visual inspection of Q-Q plots.

An exploratory factor analysis was conducted to assess construct validity of the TAME-ChatGPT instrument. First, correlation matrices, the Bartlett test, and the Kaiser-Meyer-Olkin measure of sampling adequacy were assessed to ensure that proceeding with factor analysis was appropriate. Multicollinearity was assessed by examining the determinant of the correlation in the correlation matrix with a <0.00001 cutoff. If multicollinearity was present, items with r values greater than 0.8 were considered for removal. The number of factors included was based on performing parallel analysis in which an eigenvalue cutoff was determined from randomly generated correlation matrices and then compared with eigenvalues from the data. Factors were retained if the dataset eigenvalue was greater than the corresponding random eigenvalue. Promax rotation was used to allow for correlation of factors. The Cronbach α was calculated to assess internal consistency.

Multivariable analysis was conducted using logistic regression to assess the association between preceptor characteristics and the components of the TAME-ChatGPT with the outcome of use of AI or future use of AI. Variables included the TAME-ChatGPT constructs and factors that significantly predicted current or future AI use. The number of variables included in the model was limited based on the number of participants who responded that they used AI. Goodness of fit was evaluated using the Hosmer-Lemeshow test (P>.05 indicating model fit) along with −2 log likelihood ratio, with smaller values indicating improved fit. As a secondary analysis, the model’s ability to discriminate between those who reported AI use and those who did not was evaluated using a receiver operating characteristic curve. The area under the curve was used as a summary measure of the model’s discrimination. SPSS (version 29; IBM Corp) was used to conduct the data analysis. A P value of <.05 was considered statistically significant.

Ethical Considerations

This study was determined to be exempt research by the institutional review boards at all participating institutions—Butler University, Purdue University, University of Illinois Chicago, University of Michigan, and Wayne State University. The survey responses were anonymous and confidential, and all responses were stored without any identifiers. After survey completion, respondents were directed to an optional link to receive a US $10 Amazon gift card as compensation for participation.

Results

Overview

A total of 235 responses were received. Responses were excluded from the analysis if they were incomplete (n=10, 4.3%); if the respondent practiced outside of Indiana, Illinois, or Michigan (n=4, 1.7%); or if the respondent had not been a preceptor in the previous year (n=27, 11.5%). The final response number was 194, which represented an overall response rate of 10.3% (194/1877). Respondent demographics can be found in Table 1, and they were generally representative of pharmacy preceptors with the exception of years and area of practice based on data from the American Association of Colleges of Pharmacy 2024 preceptor survey [17]. Of the 194 responses that met the inclusion criteria, 59 (30.4%) indicated that the respondents had used an AI chatbot before. A total of 51.5% (100/194) of the respondents indicated that they would continue or plan to start using chatbots in the future.

Among those who had used AI chatbots before and those who had not, demographics were similar with respect to age, gender, and years of practice (Table 2). Most respondents (146/194, 75.3%) indicated that they were unlikely to make a patient care decision based on information from a chatbot. Of the respondents who had used AI chatbots (59/194, 30.4%), there was a substantially higher percentage who practiced in academia and a lower percentage who practiced in the community setting than among those who had not used AI chatbots. Additionally, respondents who had coworkers who used AI chatbots or an institutional AI policy were more likely to have used an AI chatbot themselves.

ChatGPT was the most frequently reported chatbot used (55/59, 93%), followed by Bing Chat (10/59, 17%) and Google Bard, now Gemini (7/59, 12%). The most common uses for AI chatbots in practice included summarizing information (31/59, 53%), letter of recommendation writing (20/59, 34%), and obtaining disease state information (14/59, 24%). Among respondents who had not used AI chatbots in practice before (135/194, 69.6%), 63% (85/135) selected not knowing how to use them effectively as the reason for disuse. Other common reasons for disuse included preference for other resources (80/135, 59.3%), lack of credibility or trust (63/135, 46.7%), and concerns of plagiarism (43/135, 31.9%).

All 194 respondents received the survey including questions about the respondents’ recommendations for use of AI chatbots by pharmacists and pharmacy students in practice. The top recommendations for use for pharmacists included administrative purposes (92/194, 47.4%), summarizing information (90/194, 46.4%), creating meeting agendas (74/194, 38.1%), letter of recommendation writing (64/194, 33%), and obtaining disease state information (63/194, 32.5%); 15.5% (30/194) of respondents indicated that they would not recommend that pharmacists use AI chatbots in practice. In comparison, the top uses recommended for students were summarizing information (61/194, 31.4%), creating meeting agendas (45/194, 23.2%), obtaining disease state information (42/194, 21.6%), administrative purposes (42/194, 21.6%), and conducting literature searches (38/194, 19.6%). In total, 40.2% (78/194) of the respondents indicated that they would not recommend that students use AI chatbots in practice.

Factors Associated With AI Use

The first 13 items of the TAME-ChatGPT were included in the primary exploratory factor analysis. These items included all survey respondents, with an overall rate of AI use of 30.4% (59/194). The overall data were appropriate for conducting factor analysis (Bartlett test χ²₇₈=1288.7; P<.001), and the Kaiser-Meyer-Olkin value (0.82) indicated that sampling was adequate. There was no concern for multicollinearity, and parallel analysis identified 2 factors as the optimal number, with an eigenvalue cutoff of 1.31. The eigenvalues for the 2 factors were 4.49 and 1.97, which explained 49.7% of the cumulative variance. These 2 factors were classified as attitude toward the technology and perceived risk. Descriptive statistics for these constructs are provided in Table 3, and the pattern matrix is shown in Table 4. Including a third factor (eigenvalue=1.155), similar to the original TAME-ChatGPT validation study, resulted in an explanation of 57.1% of the cumulative variance. However, factors 2 and 3 were largely correlated (r=0.57), and many of the items were correlated with both factors 2 and 3. Cronbach α values for the 2 constructs were good at 0.86 for both comfort with technology and perceived risk.

In the secondary exploratory analysis, all items were included for the 30.4% (59/194) of respondents who had previously used AI. The overall data were appropriate for conducting factor analysis (Bartlett test χ²₅₂₈=1301.5; P<.001), and the Kaiser-Meyer-Olkin value (0.70) indicated that sampling was adequate. There was concern for multicollinearity, and therefore, items 4, 14, and 31 were removed. Parallel analysis identified 4 factors as the optimal number, with an eigenvalue cutoff of 2.08. The eigenvalues for the 4 factors were 9.33, 4.36, 2.96, and 2.26, which explained 57.3% of the cumulative variance. The 4 factors identified included attitude toward the technology, perceived usefulness, perceived risk, and ease of use. Descriptive statistics for these constructs are provided in Table 3, and the pattern matrix is shown in Table 5. Cronbach α values for the 4 constructs were very reliable for attitude toward the technology (0.92), perceived usefulness (0.92), perceived risk (0.84), and ease of use (0.83).

Logistic regression was conducted to predict current and future use of AI (Table 6). Factors that remained significant in the model that predicted current AI use included positive attitude toward technology, coworker use of AI, and the respondent practicing in academia. Factors that predicted future use of AI included perceived risk, positive attitude toward technology, and coworker use.

Discussion

Approximately one-third of pharmacy preceptors (59/194, 30.4%) reported use of an AI chatbot, with approximately half (100/194, 51.5%) indicating that they planned to start or would continue using chatbots in the future. Consistent with findings from other studies, we found that most respondents were unlikely to make patient care decisions based on information provided by an AI chatbot. However, they did report use for administrative tasks such as summarizing information and writing letters of recommendation. To our knowledge, this is the first study using the TAME-ChatGPT assessment tool among pharmacists. The findings show that this tool is valid and reliable for assessing pharmacists’ attitudes toward chatbots and their use in pharmacy practice. Pharmacists’ attitudes toward chatbots were largely influenced by their attitude toward the technology and their perceived risk related to use of the technology. Furthermore, among pharmacists who had used chatbots, attitudes toward use of chatbots were affected by the same 2 factors plus their perceived usefulness and ease of use. Positive attitudes toward technology, having coworkers who use AI, and working in academia predicted current use of AI chatbots, whereas factors predicting future use of the technology included perceived risk, positive attitudes toward technology, and coworker use of AI chatbots. Our results affirm that, when adopting ChatGPT and other AI chatbots, it is important to consider perceptions of risk, usefulness, and ease of use, as well as the users’ attitudes toward technology.

Significantly more pharmacists practicing in academia have used chatbots than pharmacists in other practice settings, perhaps because these pharmacists are often at the forefront of exploring new technologies and their applications in pharmacy practice and may be more comfortable with adopting new technologies. While academic pharmacists may be at the forefront of chatbot adoption, the use of AI chatbots is expected to increase across all pharmacy practice settings as the technology becomes more refined and its benefits become more apparent. Since the time of our data collection, more advanced AI chatbots have become available, such as OpenEvidence, which may provide more targeted information for pharmacists to use in practice.

However, most pharmacists surveyed (135/194, 69.6%) had not used an AI chatbot, with nearly two-thirds of them (85/135, 63%) stating that their reason for not using chatbots was that they did not know how to use them effectively; 46.7% (63/135) cited lack of credibility or trust in chatbots as the reason for lack of use. AI chatbots are relatively new technologies in health care, and many pharmacists may not have had sufficient exposure or training to use them effectively. Furthermore, there is currently no standardized approach to training pharmacists or other health care providers on the use of AI chatbots; thus, knowledge and skills across the profession may be inconsistent. Knowledge on how to use AI chatbots will likely improve as they become more prevalent in health care settings. Pharmacists will gain hands-on experience with these tools, become more comfortable using them, and learn to integrate them into their daily workflows. Preceptors are in an ideal position to help guide students who are just learning about AI capabilities and limitations on the optimal use of this new technology. Health care organizations, educational institutions, and postgraduate training programs should consider including training on the appropriate use of AI and risks of inappropriate use.

Our findings differ somewhat from those of the work by Sallam et al [16], who validated the TAME-ChatGPT in a sample of Jordanian health care students and identified factors affecting their attitudes toward ChatGPT and use of ChatGPT. Attitudes of Jordanian students toward ChatGPT were influenced by an additional third factor, anxiety related to fear of ChatGPT; attitudes toward use of ChatGPT were affected by behavior as a fourth factor instead of attitudes toward technology [16]. The differences are likely due to differences in the populations studied; US pharmacists and Jordanian students represent very different populations based on culture, age, practice experience, and other characteristics. Different approaches to factor analysis may also contribute to differences.

Risk perception is known to be a key factor affecting decision-making, and perception of risks associated with chatbots significantly influenced pharmacists’ attitudes toward chatbots and their use [18]. The credibility of AI chatbots in health care is a complex and evolving issue. Reliability and accuracy can vary significantly between chatbots, and not all AI chatbots are reliable sources of information. Inaccuracies, hallucinations, potential for biased responses, and the inability of chatbots to provide nuanced or context-specific information have been documented [12,19-21]. These potential barriers highlight the need for health care professional scrutiny and oversight of chatbot responses. As the technologies continue to develop, ongoing research and validation will be crucial to establish and maintain the credibility of AI chatbots in health care applications and minimize their potential for harm.

Some of the limitations of this study include the sample size as only pharmacist preceptors in the Midwest who were affiliated with the participating colleges were surveyed, so the results may not be representative of pharmacy preceptors across the United States. However, based on comparison with the 2024 American Association of Colleges of Pharmacy preceptor survey, outside of years and area of practice, the surveyed sample seems to be fairly representative of preceptors in the United States [17]. Our response rate was low, which may be due to technology-related factors (ie, the email going to the spam folder) and may have resulted in selection bias in the sense that preceptors with experience using AI may have been more likely to respond. As the survey was anonymous, we could not explore differences between respondents and nonrespondents. It is possible that the use of AI at the time of our survey overestimated use in the target population. Additionally, the AI and chatbot space is constantly evolving, and although the survey was conducted recently, perceptions can change rapidly as more people are exposed to chatbots and start using them in their daily work. Future studies could expand the surveyed population to include pharmacist preceptors nationwide or include all pharmacists, along with following up on respondents over time to evaluate changes in responses. Additionally, the validated survey can be readministered following training and education on AI chatbots to determine how perceptions of AI technology have changed.

At the time of our study, it had been almost 2 years since AI chatbots were introduced, and pharmacist preceptors were still hesitant to use the new technology, with only approximately one-third of respondents (59/194, 30.4%) indicating that they had used a chatbot in practice. Pharmacist preceptors were hesitant to use the technology for clinical decisions and were uncertain about their place in practice. This study also demonstrated that the TAME-ChatGPT survey is a reliable and validated tool that can be used to assess pharmacists’ attitude toward and use of chatbots. Constructs from the TAM, including attitude toward technology, perceived usefulness, ease of use, and perceived risk of use, as well as practice setting and coworker use, can determine and predict pharmacist use of AI chatbots. Future studies with this validated tool can be used to guide the implementation of chatbots into pharmacy practice and help inform policymakers and organization leaders on the education and training needed to promote the safe and effective use of AI chatbots in pharmacy practice.