JME JMIR Med Educ JMIR Medical Education 2369-3762 JMIR Publications Toronto, Canada v6i1e19285 32602844 10.2196/19285 Review Review Artificial Intelligence Education and Tools for Medical and Health Informatics Students: Systematic Review Eysenbach Gunther Virgona Thomas Jacobowitz William Frontoni Emanuele Liao ChienHung Sapci A Hasan MD 1
Adelphi University Nexus Building 1 South Avenue Garden City, NY, 11530 United States 1 5168338156 sapci@adelphi.edu
https://orcid.org/0000-0001-5212-904X Sapci H Aylin MD https://orcid.org/0000-0003-0239-6830 Adelphi University Garden City, NY United States Corresponding Author: A Hasan Sapci sapci@adelphi.edu Jan-Jun 2020 30 6 2020 6 1 e19285 12 4 2020 1 5 2020 6 5 2020 14 6 2020 ©A Hasan Sapci, H Aylin Sapci. Originally published in JMIR Medical Education (http://mededu.jmir.org), 30.06.2020. 2020

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.

 Background

The use of artificial intelligence (AI) in medicine will generate numerous application possibilities to improve patient care, provide real-time data analytics, and enable continuous patient monitoring. Clinicians and health informaticians should become familiar with machine learning and deep learning. Additionally, they should have a strong background in data analytics and data visualization to use, evaluate, and develop AI applications in clinical practice.

 Objective

The main objective of this study was to evaluate the current state of AI training and the use of AI tools to enhance the learning experience.

 Methods

A comprehensive systematic review was conducted to analyze the use of AI in medical and health informatics education, and to evaluate existing AI training practices. PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols) guidelines were followed. The studies that focused on the use of AI tools to enhance medical education and the studies that investigated teaching AI as a new competency were categorized separately to evaluate recent developments.

 Results

This systematic review revealed that recent publications recommend the integration of AI training into medical and health informatics curricula.

 Conclusions

To the best of our knowledge, this is the first systematic review exploring the current state of AI education in both medicine and health informatics. Since AI curricula have not been standardized and competencies have not been determined, a framework for specialized AI training in medical and health informatics education is proposed.

 artificial intelligence education machine learning deep learning medical education health informatics systematic review Introduction Overview

Artificial intelligence (AI) is one of the most disruptive innovations in health care, and the topic has attracted the attention of physicians, clinicians, researchers, and medical device industry professionals. Recent advancements in machine learning (ML) and deep learning (DL) algorithms and cloud computing have increased the adoption of AI. Consequently, applications that can handle a large number of unstructured data sets and solve complex problems have become a part of daily clinical practice.

Most AI applications process data and run self-learning algorithms behind the scenes. Although some AI applications provide data-driven recommendations to clinicians, others may not offer an option to accept, reject, or modify the output. The recommendations AI applications provide through statistical correlations may not be the best option because human-made AI algorithms may be flawed. To use and screen AI-based decisions, clinicians and health informaticians who develop AI applications should have an excellent understanding of the underlying AI concepts. This paper will focus on the emerging need for formal AI education in medicine and health informatics.

 Background

Intelligence requires the capacity to perceive contexts, associate contexts to actions, and act. Even though the concept of machines that imitate intelligent human behavior is not new, AI has recently become a topic of interest [1]. As an academic discipline, the Dartmouth College Artificial Intelligence Conference that was organized by John McCarthy in 1956 was considered the birth of this field [2].

AI, ML, and DL are closely related, and the absence of universal definitions might be confusing; however, the difference between AI, ML, and DL is simple. AI is defined as “the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages” [3]. AI-based devices can perceive the environment, simulate human intelligence, and solve problems. Their ability to adapt through progressive learning algorithms is what differentiates AI technologies from robotic and hardware-driven automation. In other words, computers can mimic human intelligence using AI techniques [4]. ML is the subset of AI that allows systems to learn from data and develop self-learning algorithms. ML applications can learn from data without being explicitly programmed; make predictions and recommendations using various tools; and enable computer applications to improve their performance [5]. DL is a subfield within ML that allows machines to use algorithms inspired by the structure of neural networks. A computer can learn how to classify images and how to assign labels to words in a sentence (semantic labeling) by using DL algorithms [5]. DL programming uses large quantities of unstructured data, calculates complex statistical models, and predicts outcomes without being explicitly programmed. Virtual assistants, chatbots, and facial recognition algorithms are some other practical examples of DL.

The main purpose of this study was to investigate peer-reviewed publications focused on AI education and to determine objective assessment methods for AI skills and competency training for medical and health informatics professionals. The impact of AI on the learning experience was evaluated to assess the need for AI education. As medical education, clinical informatics education, and health informatics education are closely related, medical and health informatics education trends were analyzed together. The American Medical Informatics Association has been working closely with the Commission on Accreditation for Health Informatics and Information Management Education to determine health informatics competencies; clinical informatics became a medical subspecialty in 2011 [6]. Although clinical and health informatics programs are designed for students who plan to pursue different career pathways, they use similar competencies. The implementation of ML in health care could result in unintended challenges and biased decisions, depending on the algorithms, data sources, and methodologies used [7]. Physicians who are not familiar with the evidence standards for AI might not be able to use the right approaches to integrate AI into clinical care. Even though there are several studies that explored how AI algorithms were helping enhance education [8-12], the number of peer-reviewed publications that focused on artificial intelligence education in medicine is limited (Figure 1).

The number of citations that appear on PubMed by year for the following search terms: ("Medical Education" OR "Medical Training") AND ("Artificial Intelligence" OR "Machine Learning" OR "Deep Learning").

 Methods

Using a replicable systematic search strategy, a full-text review was performed between November 2019 and February 2020. PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols) systematic review methodology introduced by Moher et al [13] was used to identify and analyze reliable literature. The PRISMA-P method uses a structured procedure that consists of a 17-item checklist to facilitate systematic review protocols.

The combination of five groups of keywords was used to search PubMed, IEEE (Institute of Electrical and Electronics Engineers) Xplore Digital Library, CINAHL (Cumulative Index to Nursing and Allied Health Literature) Plus, and ScienceDirect databases: (1) medical education, (2) medical training, (3) artificial intelligence, (4) machine learning, and (5) deep learning (Figure 2 and Table 1). Overall, 2082 articles matched the search criteria. After removing duplicate studies and performing an abstract review, 76 full-text articles were selected for the review. All search results were entered into EPPI-Reviewer 4 text mining software (the EPPI-Centre, University of London), and the studies that met the inclusion criteria were identified. Two researchers performed the extraction independently and assessed quality.

Venn diagram for the PubMed search. A total of 363 results were found with the following search terms: ("Medical Education" OR "Medical Training") AND ("Artificial Intelligence" OR "Machine Learning" OR "Deep Learning").

Literature sources and keywords.

Search query and literature sources Search in Return value
(“Medical Education” OR “Medical Training”) AND (“Artificial Intelligence” OR “Machine Learning” OR “Deep Learning”)
PubMed All fields 363
IEEE (Institute of Electrical and Electronics Engineers) Xplore Full text and metadata 60
ProQuest Central Full text and peer reviewed 6271
CINAHL (Cumulative Index to Nursing and Allied Health Literature) Plus All text (TX) 68
ScienceDirect Title, abstract, author-specified keywords 1588

Based on the following inclusion and exclusion criteria, the selection process was applied. Peer-reviewed research articles, review papers, conference papers, case reports, correspondences, discussions, viewpoint papers, editorials, mini-reviews, and short communications papers that focused on AI tools to enhance the learning experience in medical and health informatics education or teach AI as a new competency published after 1990 were included (Table 1). Book chapters, news, and extended abstracts published before 1990 in languages other than English were excluded. To establish validity, disagreements were discussed until a consensus was reached. Overall, 26 articles matched the inclusion criteria of this research (Figure 3).

Search methodology. CINAHL: Cumulative Index to Nursing and Allied Health Literature; IEEE: Institute of Electrical and Electronics Engineers.

 Results

One of the goals of this systematic literature review was to evaluate existing studies and determine the current state of AI education. The selected papers were used to identify the answers to three research questions.

The first research question is the following: what topics are discussed in peer-reviewed publications that focus on medical and health informatics education and AI? To answer this question, the selected publications were classified based on their education foci and the characteristics of included studies were summarized in Table 2. The publications that focused on the use of AI applications in medical and health informatics education were categorized as Category 1. These studies used various AI-based tools to enhance the learning experience. Several case studies and new initiatives to teach specific AI skills, such as ML programming languages and big data analytics software, were also identified. The publications that evaluated AI education were classified as Category 2. Category 1 studies discussed different AI applications to enhance education and summarized the impact of AI on medical and health informatics education, while Category 2 studies focused on the teaching of AI concepts.

Characteristics of included studies.

Author(s), year, and reference Country Title or objective Categorya Level of evidenceb Study objective Comments and knowledge gap
Winkler-Schwartz et al, 2019 [14] Canada Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation 1 IV The authors developed a checklist to assess surgical expertise in virtual reality simulation. The study provided a general framework only. The authors emphasized the need to add further elements.
Chan and Zary, 2019 [15] Singapore Applications and Challenges of Implementing Artificial Intelligence in Medical Education: Integrative Review 1 IV This review evaluated current applications of AIc in medical education and highlighted the main challenges. The authors acknowledged that a low number of studies were reviewed and stated that conclusions might be inconsequential.
Lillehaug and Lajoie, 1998 [16] Sweden AI in medical education—another grand challenge for medical informatics 1 IV This comprehensive review discussed the potential use of AI to enhance medical informatics education. This article was published before the discovery of high-performance computing processors and recent advancements in data recording technology.
Frize and Frasson, 2000 [17] Canada Decision-support and intelligent tutoring systems in medical education 1 V This study evaluated the use of intelligent tutoring systems in medical education. This article discusses the potential use of decision support tools but emphasizes the need for further research to validate their usefulness.
Zhao et al, 2018 [18] China Research on Application of Artificial Intelligence in Medical Education 1 V This article analyzed the application of AI in medical education. This study evaluated the effect of AI technology on traditional medical education with a focus on personalized learning.
Chary et al, 2018 [19] United States A Review of Natural Language Processing in Medical Education 1 IV This study reviewed the application of NLPd to medical education and identified concepts from NLP used in those applications. The authors investigated the integration of NLP to medical education resources using published manuscripts and stated the potentially biased representation of the scope.
Caudell et al, 2003 [20] United States Virtual patient simulator for distributed collaborative medical education 1 IV The study investigated the feasibility of using a real-time AI simulation engine in medical school curricula. The study described an ongoing project and did not provide any data about the difference between problem-based learning using virtual patient simulators and standard paper case tutorials.
Guimarães et al, 2017 [21] Portugal Rethinking Anatomy: How to Overcome Challenges of Medical Education's Evolution 1 IV This literature review evaluated the integration of complementary technology-based methodologies to medical instruction. The authors discussed the potential of AI in learning analytics-oriented systems to predict behavior but did not make any recommendations about new research studies.
Bowyer et al, 2008 [22] United States Immersive Virtual Environments for Medical Training 1 IV This study highlighted the role of advanced virtual environments and surgical simulators as a training platform for medical training. The paper described various virtual reality environments where students can interact with AI-based simulators.
Sitterding et al, 2019 [23] United States Using Artificial Intelligence and Gaming to Improve New Nurse Transition 1 IV This research discussed the preliminary pilot study data from a virtual reality simulation education intervention that compared virtual reality, augmented reality, serious gaming, and gamification. The sample size and pending postintervention findings were stated as the limitations of the preliminary findings.
Boulet and Durning, 2019 [24] United States What we measure … and what we should measure in medical education 1 V This paper focused on the validity of assessment scores and discusses the application of AI to automate the assessment process. The authors recommended developing new competency assessment practices and highlighted the importance of the application of AI. They did not provide supporting evidence about AI's potential to eliminate the need for human ratings.
Conde et al, 2009 [25] United States Telehealth Innovations in Health Education and Training 1 V This discussion paper indicated the potential of telehealth technologies for health education and training. The authors recommended the development of AI applications for patient simulation and the integration of telehealth applications in health education, but the paper did not provide any evidence.
Kabassi et al, 2008 [26] Greece Specifying the personalization reasoning mechanism for an intelligent medical e-learning system on Atheromatosis: An empirical study 1 IV The objective of this empirical study was to incorporate intelligent techniques in web-based medical education. The authors described the specification of an intelligent medical learning system for atheromatosis that can interact with students. The design was based on the results of empirical data and the authors did not compare the e-learning system with traditional methods.
Klar and Bayer, 1990 [27] Germany Computer-assisted teaching and learning in medicine 1 IV This article provided a comprehensive discussion of computer-assisted instruction systems and discussed expert systems' contribution to software for medical learning. This paper was published before AI impacted multiple fields but the authors successfully envisioned how AI would transform decision making, simulation, and medical education.
Yang et al, 2019 [28] Taiwan An expert-led and artificial intelligence (AI) system-assisted tutoring course increase confidence of Chinese medical interns on suturing and ligature skills: prospective pilot study 1 IV This paper examined the impact of an AI system tutoring course on clinical training. This study compared regular, expert-led, and expert-led+AI groups and found an increased improvement in the expert-led+AI tutoring group. Authors recommended AI-assisted tutoring for novice medical interns.
Alonso-Silverio et al, 2018 [29] Mexico Development of a Laparoscopic Box Trainer Based on Open Source Hardware and Artificial Intelligence for Objective Assessment of Surgical Psychomotor Skills 1 IV This study evaluated the effect of a laparoscopic trainer system that uses an AI algorithm. The authors described the development of a low-cost intelligent simulator to improve laparoscopic skills and proposed the training as a validated training tool for surgical education programs.
Kolachalama and Garg, 2018 [30] United States Machine learning and medical education 2 V This perspective article discussed the lack of student access to machine learning content and makes some suggestions to instructors. This perspective paper only provided an outline and did not provide any evidence.
Park et al, 2019 [31] Korea What should medical students know about artificial intelligence in medicine? 2 IV This short review emphasized the lack of direct access to machine learning education for clinicians and recommended the inclusion of focused content. The review emphasized the need to identify correct information about AI.
Wartman and Combs, 2018 [32] United States Medical Education Must Move From the Information Age to the Age of Artificial Intelligence 2 V This article discussed the need to develop new curricular components to teach the use of AI tools. This commentary article summarized the authors' perspective and did not provide supporting evidence.
Wartman and Combs, 2019 [33] United States Reimagining Medical Education in the Age of AI 2 V This paper indicated the need for a more sophisticated mathematical understanding of analytics. The authors proposed a new curriculum that will include the skill sets required to use AI effectively.
Beregi, 2018 [34] France Artificial intelligence and medical imaging 2018: French Radiology Community white paper 2 IV This review discussed current applications of AI in medical imaging and recommended AI education for radiology residents. This position paper summarized AI principles, provided an update on research in the area of AI, and described radiologists' role in providing education about AI.
Tang et al, 2018 [35] Canada Canadian Association of Radiologists White Paper on Artificial Intelligence in Radiology 2 IV This paper assessed the educational needs of radiologists and medical students, and provided recommendations. The AI working group recommended the integration of health informatics and computer science courses to analyze the opportunities and challenges associated with new AI tools.
Masters, 2019 [36] Oman Artificial intelligence in medical education 2 V This review highlighted the demand to learn how to work with AI systems and emphasized the need for AI training. The authors identified new AI applications in medicine and recommended changes to medical curricula.
Chin-Yee and Upshur, 2017 [37] Canada Clinical judgement in the era of big data and predictive analytics 2 IV This article explored different approaches to clinical judgment. Authors indicated that data-driven and AI-based applications move medicine away from virtue-based approaches to clinical reasoning and recommended an integrative approach.
Santos et al, 2019 [38] Germany Medical students' attitude toward artificial intelligence: a multicenter survey 2 IV This study investigated undergraduate medical students' attitudes toward AI. The authors designed a survey to explore students' familiarity with AI concepts in radiology and concluded that they did not have an understanding of the basic technical principles underlying AI.
Paranjape et al, 2019 [39] Netherlands Introducing Artificial Intelligence Training in Medical Education 2 IV This paper summarized the state of medical education and recommended a framework to include AI education. This viewpoint paper suggested different AI-related content for different stages of medical education.

aCategory 1: the use of AI applications in medical and health informatics education; Category 2: AI education.

bEvidence levels were as described by the Oxford Centre for Evidence-Based Medicine Levels of Evidence [40].

cAI: artificial intelligence.

dNLP: natural language processing.

Of 26 publications, 16 (61%) investigated the use of AI applications in medical education (Category 1) and 10 publications (39%) evaluated AI education in medicine (Category 2; Figure 4).

The first publications about the use of AI for medical applications were published in the early 1990s, and the capabilities of AI applications were restricted by technological limitations at that time. Klar and Bayer's paper [27], published in 1990, was one of the first publications about the application of AI, and they discussed the integration of expert knowledge into computer-assisted teaching in medicine. Lillehaug and Lajoie [16] proposed greater integration of AI in their 1998 paper, and they were among the first researchers who advocated for intelligent decision support systems and AI-based applications for medical education. Frize and Frasson [17] examined the role of decision support and intelligent tutoring systems in medical education, and recommended multidisciplinary studies in 2000.

Classification of selected AI publications. AI: artificial intelligence; ML: machine learning.

Most Category 1 studies explored innovative applications designed to improve the learning experience. For example, Chan and Zary [15] evaluated existing AI applications in medicine, determined that the primary reason to use AI in medical education was to provide feedback, and identified that significant challenges included the assessment of effectiveness and management of technical difficulties. Another systematic review conducted by Chary et al [19] identified 30 articles that assessed the application of natural language processing (NLP) to medical education. NLP is a subfield of AI and refers to intelligent communication methods using natural languages. The study revealed the benefits of NLP training in residency education and recommended strategies for its application.

Simulation-based learning has evolved over the last decade, and the virtual environment has become essential for education. Our review identified multiple case studies about advanced virtual environments. For instance, Winkler-Schwartz et al [14] analyzed virtual reality simulators that use AI, and developed a checklist to assess studies using ML algorithms to evaluate technical skills. This study concluded that the checklist had the potential to decrease the knowledge gaps in the use of AI in surgical education. Similarly, Zhao et al [18] concluded that virtual patient systems and other distance education systems that used AI increased the efficiency of medical education. Another case study that focused on virtual patient simulators identified educational and technical challenges to enhancing the learning process with AI virtual reality applications [20]. The case study published by Bowyer et al [22] described the role of advanced virtual environments in surgical training. Moreover, Conde et al [25] recommended the use of AI applications for training and education when simulated human patients were not an option.

Augmented reality is another form of virtual reality, and this technology superimposes images on top of the video viewer. Sitterding et al [23] described the differences between virtual reality, augmented reality, serious gaming, and gamification, and shared the preliminary findings of their pilot study, which determined that the simulation experience was similar to a real-life environment.

In our research, we noted different innovative AI applications for special learning activities. In their 2017 article, Guimarães et al [21] reviewed current education models for anatomy education, and recommended the use of AI analytic tools to personalize the learning process. Boulet and Durning [24] discussed the application of AI to replace human ratings and assess medical education competencies. Another study that used augmented reality and AI algorithms evaluated a case study about the development of a laparoscopic box trainer to assess surgical psychomotor skills, and concluded the proposed system had potential benefits [29].

Incorporating intelligent techniques into an adaptive e-learning system was another research group's focus. Kabassi et al [26] designed a web-based educational system for medical education, incorporated intelligent algorithms to individualize the learning experience, and shared the findings. The authors recommended further studies with more participants. Yang and Shulruf [28] designed a pilot study to demonstrate the value of AI-assisted tutoring, and determined that this additional tutoring significantly enhanced the performance of medical students.

Our second research question involved determining the highest level of evidence of current research. The articles were classified according to the Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence ranking scheme [40]. Of 26 articles, 18 (69%) were classified as Level IV evidence (case series), and 8 (31%) were classified as Level V evidence (expert opinions). The majority of the publications had Level IV and V evidence, which are considered poor reference standards. Hence, these findings emphasize the need to design new research studies.

Finally, our third research question was the following: what is the status of AI education in medicine and health informatics? In our study, 10 articles discussing AI education were identified. Since this topic is a relatively new area in medical and health informatics education, our findings were consistent with recent developments.

In their review, Paranjape et al [39] summarized multiple initiatives for AI in medical education, in which students worked with data experts and solved health care problems. The authors recommended familiarizing students with AI-based clinical applications, and introducing linear algebra, calculus, and probability during different stages of medical education. Similarly, Chin-Yee and Upshur [37] discussed the random error and biased data generated by AI and ML applications, and emphasized the effect of medical education for appraising clinical judgments.

A review conducted by Park et al [31] emphasized the importance of understanding AI to be able to validate the clinical accuracy of AI algorithms. Furthermore, multiple publications stated the need to move beyond traditional medical education, suggested a reform to align education with new practice requirements, and emphasized the role of academics and teachers in the development of appropriate AI application skills [32,33,36].

As discussed earlier, ML is an AI technique to process massive amounts of data and make predictions using computers. Kolachalama and Garg [30] proposed the integration of ML-related content in medical student, resident, and fellow education. The authors recommended the integration of real-world clinical examples into ML courses as well as practical guidelines for choosing the right tools.

The radiology community embraced AI and ML long before other medical specialties, and pioneered the usage of AI algorithms in advanced imaging applications. A multicenter survey of undergraduate medical students determined students' optimistic views about the implications of AI applications for radiology [38]. The French Radiology Community developed principles to regulate the use of AI tools, and recommended specific education to evaluate AI technologies [34]. Similarly, the Canadian Association of Radiologists recommended the integration of computer science, health informatics, and statistics training during residency education [35].

 Discussion Overview

The technological advancements in computer and software technologies; digitization of health care data; and methodological developments in information science, philosophy, mathematics, linguistics, and psychology disciplines accelerated medical research programs that focus on ML and commercialization. Maturation of AI technologies changed the roles of clinicians, and novel decision-making processes in medical settings and innovative AI-based protocols have the potential to provide diagnostic and treatment decisions by analyzing complex data sets [41].

Over the last decade, many AI researchers have concentrated on developing a proof of concept system for clinicians and patients. There is an increase in the number of studies that evaluate the effectiveness of intelligent reasoning. Intelligent monitoring technologies require new algorithms to detect anomalies, predict patterns, and make decisions.

A recent independent report prepared for the UK Secretary of State for Health and Social Care explored how the health care workforce could be prepared to use digital technology. The report emphasized the skills gap in the workforce, and made some recommendations about the integration of digital health care technologies, AI, and data analytics in undergraduate curricula [42,43].

Clinicians should have a realistic view of AI, and become familiar with the right tasks for AI in health care. Formal training for medical and health informatics students should enable them to develop AI algorithms, use AI technologies in a competent manner, and keep bias out of AI tools. Any new AI technology might encounter something new for which it has no experience, and therefore the physician should be able to assess problematic decisions and take necessary precautions when needed. Recent discussions about the need for medical education reform emphasize the shortcomings of the current model of education [44]. Overreliance on ML and AI technologies might have unintended severe adverse consequences, such as failure to recognize invalid test results [45].

While machine learning and AI algorithms are able to handle high-dimensional data classification problems and medical image interpretation, their success rates in risk prediction and diagnosis are lower. Consequently, there is a need to determine the most appropriate application areas for AI in health care [46].

There are different ways to implement AI in clinical practice, and clinicians and health informaticians need formal training to use the right approaches. Health informaticians and physician champions who design and develop AI-based protocols need to have a good understanding of complex algorithms, methodologies for data quality assessment, probabilistic forecasting, and comparative model assessment to work with engineers and develop reliable AI applications. Moreover, clinicians who use AI applications should become familiar with potential challenges.

AI-based relational time pattern analysis replaced simple threshold-based diagnostic rules. Current medical education and health informatics curricula still do not provide the ability to understand AI communications and necessary skill sets to develop AI systems that can detect and analyze relational time patterns [41].

The ability to interpret AI algorithms' mistakes and formulate the best strategies to correct these applications requires specialized training. Consequently, medical and health informatics education must emphasize algorithm-based platforms, and include relevant data analytics and AI topics in their curricula. Moreover, computer science and health informatics programs should consist of health care–focused digital skills training.

To the best of our knowledge, no previous research has investigated the use of AI tools to enhance the learning experience and AI education of medical and health informatics students. The main findings of this systematic review are as follows: (1) Although there are several recommendations on the integration of AI into medical and health informatics curricula and some academic institutions implemented experimental training programs, AI and ML education are not a part of traditional medical and health informatics curricula yet. (2) Current medical education and health informatics accreditation standards do not require AI training, and AI competencies have not been determined. (3) Using the OCEBM Levels of Evidence classification table, the majority of studies were classified as Level IV and V, which indicates poor reference standards.

 Limitations

Several efforts were made to design an optimal systematic review process; however, there were still many limitations. It is probable that some studies might not be listed in the peer-reviewed academic literature databases or might be published in a non-English language. Although this is a systematic review of the field, AI is a new technical discipline, particularly in medicine, and therefore the number of articles that met the inclusion criteria was limited.

 Future Directions

Overall, the selected publications did not provide specific details about different jobs' requirements and curriculum needs. The emergence of intelligent systems in health care requires new learning modalities. Even though several organizations, agencies, and work groups such as the International Medical Informatics Association [47], the Commission on Accreditation for Health Informatics and Information Management Education [48,49], the Health Informatics Society of Australia [50], the TIGER (Technology Informatics Guiding Education Reform) Initiative [51], and the Association of American Medical Colleges [52] published skill recommendations for health informatics curricula, they have not determined specific skill sets for AI education. Furthermore, a recent study evaluated health informatics students' skills in developing AI apps and emphasized the need to develop new competencies [53].

A specialized AI education framework for various professional fields would be useful; using the results of this systematic review, we propose a framework for specialized AI training for different domains (Figure 5). Medical students need to become familiar with clinical AI applications and predictive modeling techniques to assess biased data and evaluate innovative AI technologies. Health informatics students should become familiar with the application of appropriate ML algorithms and development of innovative clinical informatics systems. Furthermore, they should gain the hands-on skills required to extract data, manage large data sets to perform sophisticated data analytics, and develop innovative AI systems. Computer science students need specialized skill sets to work with data scientists and should become familiar with Python, R, and SQL programming languages, and data analytics tools (Figure 5).

Proposed framework for specialized AI training for different professional fields. AI: artificial intelligence; DL: deep learning; ML: machine learning.

 Abbreviations AI

artificial intelligence

 CINAHL

Cumulative Index to Nursing and Allied Health Literature

 DL

deep learning

 IEEE

Institute of Electrical and Electronics Engineers

 ML

machine learning

 NLP

natural language processing

 OCEBM

Oxford Centre for Evidence-Based Medicine

 PRISMA-P

Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols

 TIGER

Technology Informatics Guiding Education Reform

None declared.

 Mayor A Gods and robots: myths, machines, and ancient dreams of technology Princeton: Princeton University Press 2018 Princeton, NJ, USA Princeton University Press Moor J The Dartmouth College Artificial Intelligence Conference: The Next Fifty Years AI Magazine 2006 27 4 87 91 10.1609/aimag.v27i4.1911 Artificial intelligence Oxford Reference 2020-06-22 https://www.oxfordreference.com/view/10.1093/oi/authority.20110803095426960 Poole DL Mackworth AK Goebel R Computational intelligence: a logical approach 1998 New York, NY, USA Oxford University Press 1 558 LeCun Y Bengio Y Hinton G Deep learning Nature 2015 05 28 521 7553 436 44 10.1038/nature14539 26017442 nature14539 Clinical Informatics Becomes a Board-certified Medical Subspecialty Following ABMS Vote 2011 2020-03-01 https://www.amia.org/news-and-publications/press-release/ci-is-subspecialty Char DS Shah NH Magnus D Implementing Machine Learning in Health Care — Addressing Ethical Challenges N Engl J Med 2018 03 15 378 11 981 983 10.1056/nejmp1714229 McArthur D Lewis M Bishary M The Roles Of Artificial Intelligence In Education: Current Progress And Future Prospects JET 2005 03 15 1 4 42 80 10.26634/jet.1.4.972 Popenici SAD Kerr S Exploring the impact of artificial intelligence on teaching and learning in higher education Res Pract Technol Enhanc Learn 2017 11 23 12 1 22 10.1186/s41039-017-0062-8 30595727 62 PMC6294271 Bayne S Teacherbot: interventions in automated teaching Teaching in Higher Education 2015 04 16 20 4 455 467 10.1080/13562517.2015.1020783 Petit dit Dariel OJ Raby T Ravaut F Rothan-Tondeur M Developing the Serious Games potential in nursing education Nurse Educ Today 2013 12 33 12 1569 75 10.1016/j.nedt.2012.12.014 23332500 S0260-6917(12)00411-X Nye BD Intelligent Tutoring Systems by and for the Developing World: A Review of Trends and Approaches for Educational Technology in a Global Context Int J Artif Intell Educ 2014 10 9 25 2 177 203 10.1007/s40593-014-0028-6 Moher D Shamseer L Clarke M Ghersi D Liberati A Petticrew M Shekelle P Stewart LA PRISMA-P Group Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement Syst Rev 2015 01 01 4 1 1 10.1186/2046-4053-4-1 25554246 2046-4053-4-1 PMC4320440 Winkler-Schwartz A Bissonnette V Mirchi N Ponnudurai N Yilmaz R Ledwos N Siyar S Azarnoush H Karlik B Del Maestro RF Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation J Surg Educ 2019 11 76 6 1681 1690 10.1016/j.jsurg.2019.05.015 31202633 S1931-7204(19)30106-0 Chan KS Zary N Applications and Challenges of Implementing Artificial Intelligence in Medical Education: Integrative Review JMIR Med Educ 2019 06 15 5 1 e13930 10.2196/13930 31199295 v5i1e13930 PMC6598417 Lillehaug S Lajoie SP AI in medical education—another grand challenge for medical informatics Artificial Intelligence in Medicine 1998 3 12 3 197 225 10.1016/s0933-3657(97)00054-7 Frize M Frasson C Decision-support and intelligent tutoring systems in medical education Clin Invest Med 2000 23 266 269 10.1007/3-540-45108-0 Zhao H Li G Feng W Research on Application of Artificial Intelligence in Medical Education 2018 08 10 International Conference on Engineering Simulation and Intelligent Control (ESAIC) 2018 Changsha, China 340 342 10.1109/esaic.2018.00085 Chary M Parikh S Manini A Boyer E Radeos M A Review of Natural Language Processing in Medical Education West J Emerg Med 2019 01 12 20 1 78 86 10.5811/westjem.2018.11.39725 30643605 wjem-20-78 PMC6324711 Caudell TP Summers KL Holten J Hakamata T Mowafi M Jacobs J Lozanoff BK Lozanoff S Wilks D Keep MF Saiki S Alverson D Virtual patient simulator for distributed collaborative medical education Anat Rec B New Anat 2003 01 13 270 1 23 9 10.1002/ar.b.10007 12526063 Guimarães B Dourado L Tsisar S Diniz JM Madeira MD Ferreira MA Rethinking Anatomy: How to Overcome Challenges of Medical Education's Evolution Acta Med Port 2017 02 27 30 2 134 140 10.20344/amp.8404 28527481 Bowyer MW Streete KA Muniz GM Liu AV Immersive Virtual Environments for Medical Training Seminars in Colon and Rectal Surgery 2008 6 19 2 90 97 10.1053/j.scrs.2008.02.005 Sitterding MC Raab DL Saupe JL Israel KJ Using Artificial Intelligence and Gaming to Improve New Nurse Transition Nurse Leader 2019 04 17 2 125 130 10.1016/j.mnl.2018.12.013 Boulet JR Durning SJ What we measure … and what we should measure in medical education Med Educ 2019 01 14 53 1 86 94 10.1111/medu.13652 30216508 Conde JG De S Hall RW Johansen E Meglan D Peng GC Telehealth innovations in health education and training Telemed J E Health 2010 02 16 1 103 6 10.1089/tmj.2009.0152 20155874 PMC2937346 Kabassi K Virvou M Tsihrintzis GA Vlachos Y Perrea D Specifying the personalization reasoning mechanism for an intelligent medical e-learning system on Atheromatosis: An empirical study IDT 2008 10 07 2 3 179 190 10.3233/idt-2008-2304 Klar R Bayer U Computer-assisted teaching and learning in medicine International Journal of Bio-Medical Computing 1990 7 26 1-2 7 27 10.1016/0020-7101(90)90016-n Yang YY Shulruf B Expert-led and artificial intelligence (AI) system-assisted tutoring course increase confidence of Chinese medical interns on suturing and ligature skills: prospective pilot study J Educ Eval Health Prof 2019 04 10 16 7 10.3352/jeehp.2019.16.7 30986892 jeehp.2019.16.7 PMC6517322 Alonso-Silverio GA Pérez-Escamirosa F Bruno-Sanchez R Ortiz-Simon JL Muñoz-Guerrero R Minor-Martinez A Alarcón-Paredes A Development of a Laparoscopic Box Trainer Based on Open Source Hardware and Artificial Intelligence for Objective Assessment of Surgical Psychomotor Skills Surg Innov 2018 08 29 25 4 380 388 10.1177/1553350618777045 29809097 Kolachalama VB Garg PS Machine learning and medical education NPJ Digit Med 2018 9 27 1 1 54 10.1038/s41746-018-0061-1 31304333 61 PMC6550167 Park SH Do KH Kim S Park JH Lim Y What should medical students know about artificial intelligence in medicine? J Educ Eval Health Prof 2019 07 03 16 18 10.3352/jeehp.2019.16.18 31319450 jeehp.2019.16.18 PMC6639123 Wartman SA Combs CD Medical Education Must Move From the Information Age to the Age of Artificial Intelligence Academic Medicine 2018 93 8 1107 1109 10.1097/acm.0000000000002044 Wartman SA Combs CD Reimagining Medical Education in the Age of AI AMA J Ethics 2019 02 01 21 2 E146 152 10.1001/amajethics.2019.146 30794124 amajethics.2019.146 SFR-IA Group CERF French Radiology Community Artificial intelligence and medical imaging 2018: French Radiology Community white paper Diagn Interv Imaging 2018 11 99 11 727 742 10.1016/j.diii.2018.10.003 30470627 S2211-5684(18)30246-8 Tang A Tam R Cadrin-Chênevert A Guest W Chong J Barfett J Chepelev L Cairns R Mitchell JR Cicero MD Poudrette MG Jaremko JL Reinhold C Gallix B Gray B Geis R Canadian Association of Radiologists (CAR) Artificial Intelligence Working Group Canadian Association of Radiologists White Paper on Artificial Intelligence in Radiology Can Assoc Radiol J 2018 05 01 69 2 120 135 10.1016/j.carj.2018.02.002 29655580 S0846-5371(18)30030-5 Masters K Artificial intelligence in medical education Medical Teacher 2019 04 21 41 9 976 980 10.1080/0142159x.2019.1595557 Chin-Yee B Upshur R Clinical judgement in the era of big data and predictive analytics J Eval Clin Pract 2018 06 13 24 3 638 645 10.1111/jep.12852 29237237 Pinto Dos Santos D Giese D Brodehl S Chon SH Staab W Kleinert R Maintz D Baeßler B Medical students' attitude towards artificial intelligence: a multicentre survey Eur Radiol 2019 04 6 29 4 1640 1646 10.1007/s00330-018-5601-1 29980928 10.1007/s00330-018-5601-1 Paranjape K Schinkel M Nannan Panday R Car J Nanayakkara P Introducing Artificial Intelligence Training in Medical Education JMIR Med Educ 2019 12 03 5 2 e16048 10.2196/16048 31793895 v5i2e16048 PMC6918207 Howick J Chalmers I Glasziou P Greenhalgh T Heneghan C Liberati A Moschetti I Phillips B Thornton H Goddard O Hodgkinson M OCEBM Levels of Evidence Working Group The Oxford 2011 Levels of Evidence Oxford Centre of Evidence-Based Medicine 2014 2020-03-15 https://www.cebm.net/wp-content/uploads/2014/06/CEBM-Levels-of-Evidence-2.1.pdf Lynn LA Artificial intelligence systems for complex decision-making in acute care medicine: a review Patient Saf Surg 2019 2 1 13 1 6 10.1186/s13037-019-0188-2 30733829 188 PMC6357484 Rimmer A Technology will improve doctors' relationships with patients, says Topol review BMJ 2019 02 11 364 l661 10.1136/bmj.l661 30745296 National Health Service The Topol Review: Preparing the healthcare workforce to deliver the digital future 2019 2020-03-01 https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf Mehta NB Hull AL Young JB Stoller JK Just Imagine Academic Medicine 2013 88 10 1418 1423 10.1097/acm.0b013e3182a36a07 Cabitza F Rasoini R Gensini GF Unintended Consequences of Machine Learning in Medicine JAMA 2017 08 08 318 6 517 518 10.1001/jama.2017.7797 28727867 2645762 Maddox TM Rumsfeld JS Payne PRO Questions for Artificial Intelligence in Health Care JAMA 2019 01 01 321 1 31 32 10.1001/jama.2018.18932 30535130 2718456 Mantas J Ammenwerth E Demiris G Hasman A Haux R Hersh W Hovenga E Lun KC Marin H Martin-Sanchez F Wright G Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics - First Revision ejbi 2011 10.24105/ejbi.2011.07.2.2 European Commission The EU Framework Programme for Research and Innovation Horizon 2020 2013 2020-03-15 https://www.uv.es/operuv/docs_h2020/InfoKit_UK_240214_Final.pdf EU US eHealth Work Consortium TIGER Initiative HIMSS Foundation 2018 2020-03-15 http://www.himss.org/professionaldevelopment/tigers-euus-ehealth-work-project Martin-Sanchez F Health Informatics Competencies Framework Health Informatics Society of Australia 2018 2020-03-15 http://www.healthinformaticscertification.com/wp-content/uploads/2016/02/CHIA-competencies-Framework_FINAL.pdf Gugerty B Delaney C TIGER Informatics Competencies Collaborative (TICC) Final Report TIGER Summit 2009 2020-03-15 https://tigercompetencies.pbworks.com/f/TICC_Final.pdf Souze A Fyfe-Mills K Bridging the Skills Gap - Workforce Development and the Future Work ATD Public Policy Advisory Group 2018 1 56 Sapci AH Sapci HA Teaching Hands-On Informatics Skills to Future Health Informaticians: A Competency Framework Proposal and Analysis of Health Care Informatics Curricula JMIR Med Inform 2020 01 21 8 1 e15748 10.2196/15748 31961328 v8i1e15748 PMC7001041