In accordance with an international guideline [15], we translated the original ATTARI-12 into Japanese. First, the first author (HF) asked the creator of the original ATTARI-12 and corresponding author of the article that reported it to allow us to develop the Japanese version. The creator willingly provided permission for our translation of the scale. Second, 2 translators (HF and KK) conducted forward translations independently. Both translators were familiar with Western and Japanese cultures and had rich experience in developing translated versions of scales in the field of health profession education. In particular, KK is a fluent speaker of Japanese and English. Third, the translators performed a synthesis of their translations. Discrepancies were resolved through discussion (version 1). Fourth, HF asked professional bilingual translators who were not involved in our study to translate the Japanese text back into English. HF and KK then compared the back-translated and original English versions item by item and then revised the Japanese version (version 2). Fifth, an expert review of version 2 was conducted by an AI expert (YY) and a health profession education expert (YN) at HF’s request. These experts concluded that no amendment was required. Sixth, HF contacted the creator of the original scale again and asked him to check version 2. The creator concluded that no revision was necessary. Seventh, pilot testing was performed with 2 medical trainees, which indicated no problematic items. Finally, version 2 was confirmed as the finalized Japanese version of the ATTARI-12 (J-ATTARI-12).

To investigate the structural validity of the instrument, we performed exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). Because conducting these 2 types of factor analysis on the same dataset would potentially raise concerns [16], we adopted the split-half validation approach, in which the participants were randomly divided into 2 groups, half (group A) for EFA and the other half (group B) for CFA. As mentioned above, we aimed to develop the scale in a manner that was culturally adapted to the Japanese medical education context. Consequently, EFA was conducted first, followed by CFA.

To determine the appropriateness of EFA, we checked the Kaiser-Meyer-Olkin (KMO) coefficient and Bartlett sphericity test. Running EFA requires a KMO value over 0.80 and a significant result in the Bartlett test [17,18]. We applied EFA to the responses of group A to explore the factor structure of the J-ATTARI-12 using maximum likelihood estimation and Promax rotation. We determined the final factor solution using the results of parallel analysis and the factor loading values (cutoff value=0.30).

We subsequently conducted CFA on group B to confirm the model obtained in EFA. CFA was performed using the maximum likelihood estimator method. Model fitness of CFA is commonly conducted using indexes, including the comparative fit index (CFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR), with higher CFI (>0.900), lower RMSEA (<0.080), and lower SRMR (<0.080) values indicating a good fit [19-21]. In this study, we compared these indexes between the 2 models, namely, the model suggested by the EFA (2-factor model; described in detail below) and the 1-factor model.

Convergent validity was evaluated through hypothesis testing. On the basis of a previous finding that ATTARI-12 scores were positively linked with specific attitudes toward robots [10], we investigated this relationship using Pearson correlation coefficients. With reference to previous studies [10,22], attitudes toward robots were assessed using 3 items (eg, “Robots are necessary as they can do jobs that are too hard or too dangerous for people”), each of which was answered on a 4-point Likert scale (Multimedia Appendix 2). The scores were computed by averaging the responses to the 3 items (Cronbach α=0.78; mean 3.25, SD 0.55 in our dataset), with higher scores indicating a more positive attitude toward robots. The Pearson correlation coefficients are deemed meaningful if they are greater than 0.30 [23].

We used Cronbach α values to evaluate the internal consistency reliability of the scale. In this study, we computed the values using the responses of the entire sample (N=326; described in detail below). Cronbach α values above 0.70 are acceptable [24].

The possible influence of participant gender and year group on ATTARI-12 scores was explored using descriptive statistics and comparison analyses (independent 2-tailed t test or 1-way ANOVA). With regard to gender, as outlined later, the “Others” category comprised only 4 participants, rendering it inadequate for inclusion in a 1-way ANOVA. Given the high likelihood of unstable estimates and heterogeneity of variance associated with such a small group, we compared scores between only the 2 larger groups (man and woman) via independent t test. The undergraduate medical curriculum in Japan has traditionally consisted of 2 phases: the first 4 years of preclinical education and the subsequent 2 years of clinical education (ie, clinical clerkship) [25]. Accordingly, medical trainees were divided into preclinical medical students (ie, first- to fourth-year students), clinical medical students (ie, fifth- and sixth-year students), and medical residents. All statistical analyses were performed using R (version 4.5.1; R Foundation for Statistical Computing) and SPSS (version 30.0; IBM Corp), with 2-sided P values of <.05 considered statistically significant.

This study was conducted according to the ethical standards and principles of the Declaration of Helsinki and was approved by the ethics committee of Juntendo University Faculty of Medicine (E25-0028). All participants checked the consent box at the beginning of the questionnaire to indicate their informed consent to take part in the study. To ensure confidentiality, all participant data were anonymized before analysis. Participants were entered into a draw for 1 of 10 ¥5000 (approximately US $30) gift cards.

In total, 9.2% (326/3551) of the eligible participants responded to the survey. There were no missing data. Table 1 shows the participants’ characteristics, and Table 2 shows the responses to each item.

We performed EFA on group A using maximum likelihood estimation with Promax rotation (154/326, 47.2% of the participants). The KMO value was 0.83, and the Bartlett test was significant (P<.001). Table 3 shows the results of the EFA. EFA suggested a 2-factor solution. After discussion among the team members, we named the factors as follows: factor 1 was AI anxiety and aversion, and factor 2 was AI optimism and acceptance.

We then performed CFA on group B (the remaining 172/326, 52.8% of the participants) using the maximum likelihood estimator method. The 2-factor model suggested by the EFA yielded better goodness-of-fit results (CFI=0.914; RMSEA=0.075; SRMR=0.056) than the 1-factor model (CFI=0.804; RMSEA=0.113; SRMR=0.078). Multimedia Appendix 3 shows the CIs for the RMSEA. Figure 1 shows the path diagram. Therefore, we adopted the 2-factor model.

We computed the Pearson correlation coefficient between the ATTARI-12 scores and the attitude toward robots scores. The coefficient value was 0.52 (P<.001), which indicated a positive correlation between these 2 scores.

Table 4 shows the descriptive statistics with the Cronbach α values. The Cronbach α for all 12 items was 0.84. The Cronbach α values for factors 1 and 2 were 0.80 and 0.76, respectively. These values were above the cutoff (0.70).

Table 5 shows descriptive data and comparison by gender and year group. The comparison analyses did not show any significant differences in ATTARI-12 scores.

In this study, we translated the ATTARI-12, originally developed for the general population, into Japanese in accordance with an international guideline [15] and then validated its structural and convergent validity and internal consistency reliability for medical trainees. Applying this scale in the context of medical trainees in Japan has the potential to stimulate future research and educational interventions. Such initiatives would serve to effectively assess and enhance the integration of AI in clinical practice and the medical education system.

Our study found that the internal consistency reliability of the J-ATTARI-12 was good. The findings were consistent with those of the original ATTARI-12 developmental study [10]. The original study reported Cronbach α values of 0.93 for a US sample and 0.90 for a German sample [10], which suggested that the ATTARI-12 is likely a helpful measure with good internal consistency reliability across countries.

Factor analysis indicated that the J-ATTARI-12 had a 2-factor structure, in contrast to the original English-language version’s unidimensional structure. The original ATTARI-12 was conceptualized as a unidimensional scale. The original article by Stein et al [10] suggested that the developers intentionally balanced positively and negatively worded items. This design makes it unsurprising that the Japanese version yielded 2 factors corresponding to negative (factor 1: AI anxiety and aversion) and positive (factor 2: AI optimism and acceptance) item valence. Therefore, we should acknowledge that, although 2 factors emerged, they simply reflect positive versus negative wording and that, for the purpose of international comparison, using a total score based on a 1-factor assumption may remain preferable. At the same time, the content of the 2 factors (ie, AI anxiety and aversion vs AI optimism and acceptance) may reflect attitudinal domains that are influenced by broader cultural characteristics, such as uncertainty avoidance, technophilia, or collectivism. To determine whether a 2-factor structure offers conceptual or psychometric advantages beyond a unidimensional model and whether this pattern is observed across different cultural contexts, future cross-cultural validation studies with larger and more diverse samples are required.

We note a couple of potential limitations of our study. First, the response rate was relatively low. In addition, the survey was conducted using convenience sampling and included only 14 institutions, and it is likely that participating institutions or individuals had a greater interest in AI. It is increasingly challenging to obtain high response rates to online surveys, and rates frequently drop to 10% [26,27]. Nevertheless, the literature suggests that the response rate to our survey may have been sufficient to provide reliable data [28,29]. Additionally, it should be noted that, despite the absence of accurate statistics on the demographic variables of medical trainees across Japan, recent reports have indicated that the proportion of female physicians aged ≤29 years stands at approximately 40%, with a steady upward trend observed in recent years [30]. Thus, the finding that 43.6% (142/326) of the respondents were women does not generate concerns regarding the representativeness of the sample. Second, in this study, although we evaluated the structural validity, convergent validity, and internal consistency reliability of the scale, other determinants of validity (eg, discriminant validity and predictive validity) and reliability (eg, test-retest reliability) have yet to be examined. Future studies should examine these psychometric properties. Third, the use of a gift card lottery can raise concerns about bias. Nevertheless, this method is commonly used and appears to be acceptable [31-34]. Fourth, our results may indicate a ceiling effect. For example, 41.4% (135/326) of the respondents selected the highest score (5) for item 5. However, according to the original ATTARI-12 paper, responses tend to be skewed toward a score of 4, suggesting that the scale inherently elicits generally positive attitudes toward AI. Therefore, these findings do not represent an unusual deviation but are consistent with the response patterns reported in the original study.

Despite these limitations, this study produced the first Japanese version of the ATTARI-12, which is likely to be used as a novel measure for assessing AI attitudes among medical students and residents. Two outcomes are expected. First, by incorporating the J-ATTARI-12 into medical curricula, medical educators will be able to develop a more comprehensive understanding of the trainees’ readiness for AI adoption. This will likely be advantageous in the design of customized educational interventions using AI based on the level of each trainee’s attitude toward AI [35] and will support curriculum development by enabling medical educators to identify learners who may need more foundational AI exposure or targeted support. Second, the J-ATTARI-12 will facilitate medical education research in Japan. The scale allows researchers to examine the impact of specific educational interventions on attitude change concerning AI, and it is suitable for repeated administration in longitudinal studies to track how attitudes evolve over the course of training. In addition, the availability of the J-ATTARI-12 will promote cross-cultural comparison of attitudes toward AI and its possible outcomes internationally. For these reasons, we expect that the J-ATTARI-12 will play a pivotal role in facilitating data-informed curriculum development and contribute to the expanding body of medical education research in this era of AI.

In this study, we developed the J-ATTARI-12, originally developed for the general population, in accordance with an international guideline. A validation survey revealed that the structural and convergent validity, as well as the internal consistency reliability, were good for medical trainees in Japan. The developed measure can be used for customized educational initiatives using AI based on the level of each trainee’s attitude toward AI. It will also provide helpful information to medical education researchers in Japan in this era of AI.