ChatGPT May Improve Access to Language-Concordant Care for Patients With Non–English Language Preferences

Introduction

Chat Generative Pre-trained Transformer (ChatGPT) is an artificial intelligence (AI) language model based on natural language processing techniques, developed by OpenAI, that is capable of generating new texts, responding to user input with conversational responses, and summarizing and translating text [1]. The model is trained on large data sets to mimic human language. While it can produce correct responses to queries, ChatGPT is not specifically trained to produce accurate information and may generate false answers that sound plausible.

Prior work evaluating its uses in the medical domain found that ChatGPT could achieve a greater than 60% threshold on NBME (National Board of Medical Examiners) Step 1 questions and provide justification to support most answers [2]. Applications in supporting clinical practice and scientific writing have also been proposed [3]. However, there is a need to explore novel uses to improve patient outcomes and address disparities. Written education materials are critical for communicating information to patients; however, they are not widely available in languages other than English. In 2020, the American Academy of Dermatology website offered information on 6% (5/83) of its listed dermatologic conditions in Spanish [4]. We aimed to evaluate ChatGPT’s ability to translate patient education materials from English to Spanish, Mandarin, and Russian.

Methods

English-language resources, each fewer than 500 words, used at the University of California, San Francisco, covering 5 dermatology topics, including seborrheic keratosis, sun protection, tinea pedis, warts, and wound care, were translated using the prompt, “Translate this into <target language>.” Excerpts from each topic and the corresponding translations provided by ChatGPT in Mandarin, Spanish, and Russian are shown in Table 1.

Topics were chosen based on their frequency of discussion in clinics. Resource materials addressed etiology, symptoms, diagnosis, and treatment. In addition, one document addressed expectations and care recommendations for wounds with dissolvable sutures. Board-certified dermatologists fluent in the target languages reviewed the translations for accuracy and suitability for clinical use. One dermatologist was assigned for each target language.

Reviewers completed a 3-question survey indicating the number of instances of misinformation and grammatical errors or misspellings. Misinformation was defined as translation errors that distorted the concept presented in the source document. Reviewers indicated the frequency by selecting one of four options: (1) <3, (2) 3‐5, (3) 6‐10, and (4) >10. Reviewers also indicated their level of agreement with the statement, “I could provide this to a patient speaking this language,” on a 5-point Likert scale: (1) strongly agree, (2) agree, (3) neutral, (4) disagree, and (5) strongly disagree.

Ethical Considerations

The study adhered to institutional ethical guidelines. The study did not require approval from an institutional review board or research ethics board because the study did not involve identifiable private information or clinical intervention involving humans subjects. Informed consent was not applicable for this study.

Results

The results of the survey are shown in Table 2. Misinformation was prevalent in materials translated into Mandarin, with 60% (3/5) of translations containing between 3 and 5 instances. Grammatical and spelling errors were identified for all languages. For Mandarin, 60% (3/5) had 6-10 errors, and 40% (2/5) had between 1 and 5 errors. For Russian, 20% (1/5) had 6-10 errors, and 80% (4/5) had more than 10 errors. For Spanish, 40% (2/5) documents had 3-5 errors, and 60% (3/5) had 6-10 errors. When asked whether they would provide the translation to patients, the reviewers responded “agree” for 80% (4/5) of Russian and Spanish documents and 20% (1/5) of Mandarin documents. Moreover, 60% (3/5) of Mandarin documents received a “disagree” response.

Discussion

This study’s findings indicate that ChatGPT may be an effective tool for translating patient education materials into Spanish and Russian. Despite some errors, most translations were deemed clinically usable. Mandarin translations, however, were less reliable and deemed insufficient for patient use. ChatGPT may be a promising tool for expanding access to language-concordant care; however, further investigation is necessary, given that its proficiency appears to vary by language.

Previous studies have demonstrated ChatGPT’s ability to answer medical licensing examination questions in Spanish and Chinese, underscoring its potential applications in medical education in non–English languages [5,6]. However, one study concluded that its performance in Chinese is suboptimal for medical education and clinical decision-making compared to English [7]. The linguistic complexity of Mandarin poses challenges; differences in sentence structure and terminology between Mandarin and English likely contribute to this suboptimal performance, highlighting the need for caution when using AI translation for languages with significant linguistic differences from English.

Our study is limited by its focus on dermatologists’ expertise for evaluating translations and the use of existing materials without exploring multiple prompts, thus potentially underestimating the effects of errors on comprehension and credibility. Ultimately, any implementation of translated documents should be preceded by careful proofreading to ensure their suitability given variable efficacy by language.