Utilization of, Perceptions on, and Intention to Use AI Chatbots Among Medical Students in China: National Cross-Sectional Study

Introduction

The rapid advancements in artificial intelligence (AI) have significantly transformed various sectors, including health care. Among these advancements, AI chatbots have emerged as a promising tool with potential applications in medical education [1]. These intelligent systems use natural language processing and machine learning algorithms to engage in human-like dialogues, providing information in an understandable, efficient, interactive, and scenario-specific format, such as ChatGPT, Claude, Google Bard, and Bing’s AI [2]. The chatbots can assist medical students in medical research support, personalized learning, comprehending complex medical topics, developing clinical decision-making skills, and so forth [1,3]. A recent study demonstrated the efficacy of AI chatbots in answering complex medical questions and providing valuable medical educational support [4].

In China, integrating AI technology into medical education is particularly important, given the country’s substantial investment in AI development and its growing emphasis on innovative educational methodologies [5,6]. AI chatbots would facilitate personalized learning experiences when facing the situation of rigorous curricula and high student to teacher ratios in China. Medical students are a crucial target group for AI chatbot technology, as they are early adopters of technology and future health care providers who will play a vital role in shaping the future of health care. While research on AI chatbot applications in medical students has emerged [7-10], the utilization of, perceptions on, and intention to use AI chatbots among Chinese medical students are still unknown.

The adoption and effective utilization of AI chatbots among medical students depend on various factors, including their perceptions, attitudes, and behavioral intentions. The Unified Theory of Acceptance and Use of Technology (UTAUT), developed by Venkatesh et al [11], provides a comprehensive framework to understand the determinants of technology acceptance and usage, which is widely used in health care. Applying the UTAUT model in the context of AI chatbots can yield valuable insights into the factors that drive or hinder their adoption among medical students.

This study aims to explore the utilization of, perceptions on, and intention to use generative AI chatbots among medical students in China, using the UTAUT framework. By conducting a national cross-sectional survey, we seek to identify the key determinants that influence medical students’ acceptance of AI chatbots, thereby providing a basis for enhancing their integration into medical education. Understanding these factors is crucial for educators, policy makers, and technology developers to design and implement effective AI-driven educational tools that align with the needs and expectations of future health care professionals.

Methods

Participants and Procedure

The target population was medical students enrolled in Chinese medical colleges or universities. An electronic survey was developed through a web-based survey platform named Wenjuanxing Questionnaire Star (Ranxing Technology Corp.), and the survey link was distributed via WeChat (Tencent Holdings Ltd) to medical college students across the country. Using a convenience sampling method, the questionnaire was posted on WeChat Moments and sent to WeChat groups from the research team’s WeChat accounts. We identified relevant WeChat groups that consisted of medical students across various regions in China. These groups were selected based on their active participation in medical education discussions and their membership of medical students from diverse backgrounds and institutions. The research team directly contacted a total of 15 WeChat groups. To further enhance the reach, we used a snowball sampling method by requesting initial respondents to forward the survey link to other medical students in their network. Questionnaires that were considered valid included only the following: (1) each account responded only once, and (2) the total response time for completing the questionnaire was more than 300 seconds. Participants were recruited between June 2023 and July 2023.

To ensure adequate statistical power and precision for the intended analyses, we conducted a sample size calculation using G*Power software (version 3.1.9.7) [12]. The calculation was based on the following parameters: a small effect size (f₂=0.05) was chosen for the multivariable regression analysis; the number of predictors was set at 15; the desired statistical power was set at 0.95; and the significance level was set at .05 (2-tailed). Based on these parameters, the minimum required sample size for the multivariable regression analysis was calculated to be 566 participants. However, to account for 20% missing data and increase the generalizability of the findings, we aimed to recruit 680 participants. Ultimately, we were able to collect 715 questionnaires across China, with 693 determined valid, representing a 96.9% final response rate.

The Theoretical Framework

The study used the UTAUT as a theoretical framework for the research. The UTAUT describes 4 key independent variables: performance expectancy (PE), effort expectancy (EE), social influence (SI), and facilitating conditions (FC). In this study, PE measured the participants’ expectation that an AI chatbot will be useful for the study; EE measured the expectation that an AI chatbot is user-friendly and easy to use; SI measured the degree to which a user perceives that important others believe that he or she should use the new technology; and FC measured the degree to which a user believes that an organizational and technical infrastructure exists to support AI chatbot use [13]. The dependent variable BI was determined by PE, EE, SI, and FC. BI measured participants’ intention to use the AI chatbot in their future study.

The intention was used as an outcome instead of the actual use of an AI chatbot, because the application of AI chatbot services has not been widely commercialized in China. Most medical students may not have experience with an AI chatbot in their study. Also, BI is a good representation of actual behavior [14]. Previous studies confirmed that these 4 variables (PE, EE, SI, and FC) have a positive influence on the intention to use the AI technology [15-17]. The original UTAUT validation study found that the UTAUT model is robust in explaining a high degree of variance (70%) in BI [13]. Moderating effects of age, gender, and experience were not tested in this study.

In addition, we added perceived risk (PR), resistance bias (RB), and personal innovativeness (PI) as 3 variables to the original UTAUT model. PR is defined as the potential for loss in the pursuit of the desired outcome of using a technology and identified for 7 facets of PR [18]. Here, PR was measured for performance risk, time risk, and privacy risk. RB is resistance to change, referring to people’s attempts to maintain previous behaviors or habits that are connected to their past experiences when facing changes [16]. PI was designed to measure an individual’s willingness to try out any new information technology [19]. Since AI chatbots are an emerging technology in health care, a user’s inherent innovativeness may impact his or her intention to adopt this innovation, and some users may be accompanied by concerns and resistance to change when embracing the new technology. Previous studies found that PR and RB have been regarded as major barriers to health care information technology adoption [20,21], and PI has been statistically significant in predicting the BI of the user [22].

Questionnaire and Instrument

The developed questionnaire consisted of 3 parts (see questionnaire in Multimedia Appendix 1): (1) participants’ sociodemographic information, such as age, gender, and grade level; (2) participants’ cognition of, attitude toward, and experience with AI chatbots (these items were designed as categorical variables and were derived through a comprehensive process, including literature review and expert consultation, to ensure their relevance and clarity); and (3) the scale of the research model. The model covered 7 constructs with 29 questionnaire items (Table 1). The items of the survey were ordered such that items measuring each construct were grouped. The responses were recorded using a 5-point Likert scale (ranging from 1=totally disagree to 5=totally agree) in which the higher score values indicated a higher level of a construct and a higher score of the outcome (BI) indicated greater intention to use the AI chatbot.

In the third part of the questionnaire, each item in the scale was sourced from relevant literature related to new technology acceptance research. The main modifications to the original instrument were made to fit the context of an AI chatbot used for medical students, such as changing the word “system” to “AI Chatbot.” The items that assessed PE, EE, SI, FC, and BI were adopted from the original instrument developed by Venkatesh et al [11]. The original survey was validated and applied to previous studies based on the UTAUT model [16,23,24]. The items that assessed PR and RB were adopted from the validated questionnaire developed by Zhai et al [16]. The reliability of the items’ scales was tested by Cronbach α coefficient analysis. The results of Cronbach α are considered to have acceptable reliability (Table 1), as the generally accepted rule is that α values of 0.6‐0.7 indicate an acceptable level of reliability, and 0.8 or greater is a very good level [25].

After we developed the questionnaire and before implementing the survey, we conducted a consensus panel of 5 experts to review the questionnaire and ensure clarity of the survey and content validity. We then conducted a pilot study of 20 students to clarify phrasing and eliminate items that were not identifiable in the questionnaire.

Data Analysis

The statistical software SPSS 25.0 (IBM Corp) was used to calculate the Cronbach α coefficient. Data analysis was carried out using descriptive statistics, such as means, frequencies, and percentages, as well as inferential statistics, such as multiple linear regression, to explore the relationships between the dependent variable (BI) and the set of predictors (PE, EE, SI, FC, PR, RB, and PI). The α level was set at .05 for all analyses. Data analysis was performed using Stata (version 17.0; StataCorp LLC).

Ethical Considerations

Ethical approval was obtained from the Ethics Committee on Biomedical Research, West China Hospital of Sichuan University (approval number: 2023‐834). The research purpose; methods; and participants’ rights, including that they could cease participation at any point without penalty, were explained. All the participants read and signed the electronic informed consent before completing the questionnaire. The detailed information on the informed consent form is given in the questionnaire in Multimedia Appendix 1. This survey was anonymous and voluntary. To promote survey completion and ensure an adequate response rate, postsurvey gifts were randomly raffled as an incentive.

Results

Participants’ Information

A total of 693 participants were from 57 universities covering 21 provinces or municipalities. The sample distribution is shown in Figure 1. The demographic characteristics of the participants are shown in Table 2. The majority of participants (251/693, 63.78%) were female, while 36.22% (442/693) were male. The average age was 22.6 (SD 5.2) years, and more than half of the participants were in the 20‐ to 24-year age range (413/693, 59.60%). The majority (543/693, 78.35%) were undergraduate students. The mean self-reported academic score was 73.7 (SD 14.8), and the most common self-reported score range was 80‐89 (247/693, 35.64%).

‎

Participants’ Perception of AI Chatbots

Table 3 presents the participants’ cognition, attitudes, usage behavior, and willingness to pay for AI chatbots. While only 24.68% (171/693) of participants reported being fairly familiar with AI chatbots and 4.47% (31/693) were very familiar with AI chatbots, 59.88% (415/693) agreed or strongly agreed with using them for study or work purposes. Of the 28.72% (199/693) who have used AI chatbots for studying, mainly ChatGPT (129/693, 18.61%), 50.25% (100/199) reported occasional usage as needed. Among nonusers, 55.06% (272/494) expressed willingness to learn AI chatbot usage, with the main reasons for unwillingness being no need (15/29, 51.72%) and no interest (15/29, 51.72%). In addition, 36.45% (242/664) preferred to use AI chatbots without charge.

Table 4 summarizes the participants’ purposes for using AI chatbots and their perceived advantages and disadvantages. The primary purposes were quickly obtaining medical information and knowledge (631/693, 91.05%) and increasing learning efficiency (594/693, 85.71%). Perceived advantages included effectively helping medical students learn (631/693, 91.05%) and providing fast and accurate medical information (624/693, 90.04%). However, data privacy breaches (635/693, 91.63%) and risks of misdiagnosis or underdiagnosis (619/693, 89.32%) were predominant concerns.

Descriptive Statistics of the UTAUT Constructs

Descriptive statistics (mean [SD]) were reported to explain and describe the UTAUT constructs (Table S1 in Multimedia Appendix 1). The value of each construct ranges from 1 to 5 (1=strongly disagree, 5=strongly agree). As shown in Figure 2, the mean for PE, EE, SI, FC, PR, and PI were higher than 3 and the mean for RB was <3. The highest score was PE at 3.66, followed by EE at 3.56. The mean of BI was 3.26, which shows a higher level of intention to use AI chatbots among Chinese medical students.

‎

Determinant Factors of Intention to Use AI Chatbots

A multiple linear regression analysis was conducted to identify factors influencing medical students’ intentions to use AI chatbots (Table 5). Utilization behavior ($\beta$=.27; P<.001), SI ($\beta$=.32; P<.001), FC ($\beta$=.29; P<.001), PR ($\beta$=.27; P<.001), and PI ($\beta$=.35; P<.001) were significantly positively associated with BI. PI had the largest positive regression coefficient ($\beta$=.35) compared with the other significant variables. PE ($\beta$=.09; P=.12), EE ($\beta$=.03; P=.60), and RB ($\beta$=−0.04; P=.32) did not significantly affect BI.

Discussion

Principal Findings

In this study, we examined the perceptions of Chinese medical students toward Natural Language Processing–based AI chatbots and investigated the factors that may influence their intention to use such technology based on the UTAUT model. This research yielded several key findings. First, the medical students demonstrated positive perceptions and expressed a high BI to use AI chatbots. Second, among the factors considered, SI and FC emerged as more influential in the adoption of AI chatbots among medical students than PE and EE. However, PE and EE were not found to have a significant relationship with BI. Third, PR and PI positively influenced BI, while RB did not show a significant association with BI.

This study revealed that although most medical students have limited knowledge about AI chatbots at an early time, they hold positive perceptions and demonstrate a strong intention to use this innovative technology. The overall sample displayed high BIs, with a mean score of 3.26 out of 5.00. Furthermore, 81.63% (1697/2079) of participants rated their intention as 3 or higher, indicating their plans to use the technology within the next 2 months. These findings align with previous research indicating that while medical students may lack knowledge about AI and its applications, they maintain a favorable view of AI in the medical field and are willing to adopt it [26,27]. The majority of participants believe that AI chatbots have the potential to enhance their study or work performance, improve efficiency, and provide fast and accurate medical information, among other benefits. However, limited availability and coverage of AI chatbots in China have resulted in less than one-third of participants actually using these tools and only a few using them on a daily basis. This indicates a gap between intention to use and actual adoption. Practical barriers, such as inadequate technical infrastructure and lack of support, may hinder the actual implementation and use of AI chatbots. In addition, our regression analysis revealed that utilization behavior significantly influences medical students’ intentions to use AI chatbots. User experience may impact their perceptions of the technology from multiple aspects, thereby affecting their usage intentions.

This study found that SI and FC have a stronger impact on BI than PE and EE. This finding aligns with research examining the perceptions of Chinese radiation oncologists toward adopting AI-assisted contouring technology [16]. However, it contradicts some prior studies that have established a positive and significant relationship between PE and EE with students’ BI to use AI-assisted learning environments [28] or chatbots [29]. This suggests that factors such as PE and EE may be less critical for the population in this study, although they rated PE and EE higher than other dimensions. It is possible that as medical students are still in training, they rely more on the experiences of their peers and the infrastructure provided by their educational institution to guide their technology use. Thus, demonstrating adoption and endorsement from fellow students, professors, and the academic medical system may be more influential in persuading them to use AI chatbots than emphasizing use and usability. Ensuring accessibility within the educational context appears to shape students’ willingness to use AI chatbots more than their individual perceptions of performance and efficiency.

Interestingly, this study found that PR and PI positively influenced the intentions of medical students to use AI chatbots, while RB was not found to be a significant factor. This suggests that concerns regarding the risks associated with adopting AI chatbots were outweighed by the students’ openness to embracing new technologies. Those with a greater inclination toward innovation recognized the potential benefits despite the potential risks involved. This finding aligns with previous research indicating that perceived usefulness can override PR when it comes to determining acceptance of technology [18]. It also reflects a growing understanding that AI systems present both opportunities and risks, necessitating ethical analysis and oversight [30], including privacy breaches and the possibility of misinformation. Notably, we found that PI emerged as a key determinant of user behavior intentions, which is consistent with a similar study [31]. However, RB did not negatively impact intentions, suggesting that medical students may have fewer biases against AI than health care professionals in hospitals who may fear a loss of professional autonomy and challenges in integrating AI into clinical workflows [32]. Encouraging PI while addressing risk concerns through testing and regulation may further bolster the adoption of AI chatbots.

Implications for Practice

Based on our findings, we recommend the following specific strategies for educational institutions and AI chatbot developers to enhance the adoption rate among medical students. First, medical schools and health care organizations should prioritize efforts to improve FC and leverage SI to drive the adoption of AI chatbots, rather than solely focusing on performance benefits. Providing integrated access, training, and IT support and sharing peer usage examples can help translate positive intentions into actual usage behaviors. In addition, demonstrating value through pilot studies and addressing valid risk concerns will promote responsible and open adoption of the technology. Targeted training in AI competencies can further equip students to become champions of safe and effective adoption. The key lies in creating optimal environments and processes to enable the proficient use of AI systems such as chatbots as students transition into practice.

Strengths and Limitations

This study was the first to use the UTAUT theoretical framework to analyze medical students’ intention to use AI chatbots. It possesses several strengths, including the robust technology adoption model used, the focus on an important user population, and the identification of key variables influencing intentions. However, there are some limitations that need to be addressed in future studies. First, the unbalanced research sample primarily from Sichuan province may limit the generalizability of the findings, potentially overrepresenting specific regional experiences. Although we distributed the survey widely, future studies should use stratified sampling for better regional representation. Second, the cross-sectional design offers only a snapshot of adoption, which may change over time as participants accumulate knowledge and experience. Future research should consider longitudinal designs to track these changes. Third, it is crucial to acknowledge that the field of AI chatbots is rapidly evolving, and our findings capture perceptions and attitudes at a specific point in time. As AI chatbot capabilities continue to advance, the external validity of our findings may need to be reevaluated.

Future studies with larger samples using longitudinal methods would enhance our understanding of actual perceptions and usage patterns over time. For example, a longitudinal study could follow a cohort of medical students from their entry into medical school until graduation, periodically assessing their perceptions, intentions, and actual usage of AI chatbots. This longitudinal approach would capture how their adoption and experiences with AI chatbots evolve as they progress through their medical education and gain more exposure to clinical settings. Furthermore, mixed methods designs, combining quantitative surveys with qualitative interviews or focus groups, could provide more in-depth insights into specific barriers, challenges, and facilitators influencing AI chatbot adoption among medical students. Overall, this study lays the foundation for a wide range of future research, which can deepen knowledge and generate evidence to guide the implementation of AI in education and health care.

Conclusions

This study offers valuable insights into medical students’ utilization of, perceptions on, and intention to use AI chatbots in health care. The results indicate that these medical students have positive perceptions and strong intentions to use chatbots, primarily influenced by SI and FC rather than PE and EE. However, despite these intentions, there remains a gap between intention and actual adoption, signaling the need for strategies that improve access, training, and support and provide peer usage examples to enhance the realization of the potential benefits of chatbots. While concerns about risks exist, the students’ general openness to innovation suggests that the integration of AI with proper oversight is well received. As future health care professionals, students serve as early adopters who can shape wider acceptance if barriers to adoption are actively addressed. This research provides a foundation for understanding the technology needs and motivations of this important user population in order to guide the successful implementation of AI.