Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 30.06.20 in Vol 6, No 1 (2020): Jan-Jun

This paper is in the following e-collection/theme issue:

    Review

    Artificial Intelligence Education and Tools for Medical and Health Informatics Students: Systematic Review

    1Adelphi University, Garden City, NY, United States

    *all authors contributed equally

    Corresponding Author:

    A Hasan Sapci, MD

    Adelphi University

    Nexus Building

    1 South Avenue

    Garden City, NY, 11530

    United States

    Phone: 1 5168338156

    Email: sapci@adelphi.edu


    ABSTRACT

    Background: The use of artificial intelligence (AI) in medicine will generate numerous application possibilities to improve patient care, provide real-time data analytics, and enable continuous patient monitoring. Clinicians and health informaticians should become familiar with machine learning and deep learning. Additionally, they should have a strong background in data analytics and data visualization to use, evaluate, and develop AI applications in clinical practice.

    Objective: The main objective of this study was to evaluate the current state of AI training and the use of AI tools to enhance the learning experience.

    Methods: A comprehensive systematic review was conducted to analyze the use of AI in medical and health informatics education, and to evaluate existing AI training practices. PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols) guidelines were followed. The studies that focused on the use of AI tools to enhance medical education and the studies that investigated teaching AI as a new competency were categorized separately to evaluate recent developments.

    Results: This systematic review revealed that recent publications recommend the integration of AI training into medical and health informatics curricula.

    Conclusions: To the best of our knowledge, this is the first systematic review exploring the current state of AI education in both medicine and health informatics. Since AI curricula have not been standardized and competencies have not been determined, a framework for specialized AI training in medical and health informatics education is proposed.

    JMIR Med Educ 2020;6(1):e19285

    doi:10.2196/19285

    KEYWORDS



    Introduction

    Overview

    Artificial intelligence (AI) is one of the most disruptive innovations in health care, and the topic has attracted the attention of physicians, clinicians, researchers, and medical device industry professionals. Recent advancements in machine learning (ML) and deep learning (DL) algorithms and cloud computing have increased the adoption of AI. Consequently, applications that can handle a large number of unstructured data sets and solve complex problems have become a part of daily clinical practice.

    Most AI applications process data and run self-learning algorithms behind the scenes. Although some AI applications provide data-driven recommendations to clinicians, others may not offer an option to accept, reject, or modify the output. The recommendations AI applications provide through statistical correlations may not be the best option because human-made AI algorithms may be flawed. To use and screen AI-based decisions, clinicians and health informaticians who develop AI applications should have an excellent understanding of the underlying AI concepts. This paper will focus on the emerging need for formal AI education in medicine and health informatics.

    Background

    Intelligence requires the capacity to perceive contexts, associate contexts to actions, and act. Even though the concept of machines that imitate intelligent human behavior is not new, AI has recently become a topic of interest [1]. As an academic discipline, the Dartmouth College Artificial Intelligence Conference that was organized by John McCarthy in 1956 was considered the birth of this field [2].

    AI, ML, and DL are closely related, and the absence of universal definitions might be confusing; however, the difference between AI, ML, and DL is simple. AI is defined as “the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages” [3]. AI-based devices can perceive the environment, simulate human intelligence, and solve problems. Their ability to adapt through progressive learning algorithms is what differentiates AI technologies from robotic and hardware-driven automation. In other words, computers can mimic human intelligence using AI techniques [4]. ML is the subset of AI that allows systems to learn from data and develop self-learning algorithms. ML applications can learn from data without being explicitly programmed; make predictions and recommendations using various tools; and enable computer applications to improve their performance [5]. DL is a subfield within ML that allows machines to use algorithms inspired by the structure of neural networks. A computer can learn how to classify images and how to assign labels to words in a sentence (semantic labeling) by using DL algorithms [5]. DL programming uses large quantities of unstructured data, calculates complex statistical models, and predicts outcomes without being explicitly programmed. Virtual assistants, chatbots, and facial recognition algorithms are some other practical examples of DL.

    The main purpose of this study was to investigate peer-reviewed publications focused on AI education and to determine objective assessment methods for AI skills and competency training for medical and health informatics professionals. The impact of AI on the learning experience was evaluated to assess the need for AI education. As medical education, clinical informatics education, and health informatics education are closely related, medical and health informatics education trends were analyzed together. The American Medical Informatics Association has been working closely with the Commission on Accreditation for Health Informatics and Information Management Education to determine health informatics competencies; clinical informatics became a medical subspecialty in 2011 [6]. Although clinical and health informatics programs are designed for students who plan to pursue different career pathways, they use similar competencies. The implementation of ML in health care could result in unintended challenges and biased decisions, depending on the algorithms, data sources, and methodologies used [7]. Physicians who are not familiar with the evidence standards for AI might not be able to use the right approaches to integrate AI into clinical care. Even though there are several studies that explored how AI algorithms were helping enhance education [8-12], the number of peer-reviewed publications that focused on artificial intelligence education in medicine is limited (Figure 1).

    Figure 1. The number of citations that appear on PubMed by year for the following search terms: ("Medical Education" OR "Medical Training") AND ("Artificial Intelligence" OR "Machine Learning" OR "Deep Learning").
    View this figure

    Methods

    Using a replicable systematic search strategy, a full-text review was performed between November 2019 and February 2020. PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols) systematic review methodology introduced by Moher et al [13] was used to identify and analyze reliable literature. The PRISMA-P method uses a structured procedure that consists of a 17-item checklist to facilitate systematic review protocols.

    The combination of five groups of keywords was used to search PubMed, IEEE (Institute of Electrical and Electronics Engineers) Xplore Digital Library, CINAHL (Cumulative Index to Nursing and Allied Health Literature) Plus, and ScienceDirect databases: (1) medical education, (2) medical training, (3) artificial intelligence, (4) machine learning, and (5) deep learning (Figure 2 and Table 1). Overall, 2082 articles matched the search criteria. After removing duplicate studies and performing an abstract review, 76 full-text articles were selected for the review. All search results were entered into EPPI-Reviewer 4 text mining software (the EPPI-Centre, University of London), and the studies that met the inclusion criteria were identified. Two researchers performed the extraction independently and assessed quality.

    Figure 2. Venn diagram for the PubMed search. A total of 363 results were found with the following search terms: ("Medical Education" OR "Medical Training") AND ("Artificial Intelligence" OR "Machine Learning" OR "Deep Learning").
    View this figure
    Table 1. Literature sources and keywords.
    View this table

    Based on the following inclusion and exclusion criteria, the selection process was applied. Peer-reviewed research articles, review papers, conference papers, case reports, correspondences, discussions, viewpoint papers, editorials, mini-reviews, and short communications papers that focused on AI tools to enhance the learning experience in medical and health informatics education or teach AI as a new competency published after 1990 were included (Table 1). Book chapters, news, and extended abstracts published before 1990 in languages other than English were excluded. To establish validity, disagreements were discussed until a consensus was reached. Overall, 26 articles matched the inclusion criteria of this research (Figure 3).

    Figure 3. Search methodology. CINAHL: Cumulative Index to Nursing and Allied Health Literature; IEEE: Institute of Electrical and Electronics Engineers.
    View this figure

    Results

    One of the goals of this systematic literature review was to evaluate existing studies and determine the current state of AI education. The selected papers were used to identify the answers to three research questions.

    The first research question is the following: what topics are discussed in peer-reviewed publications that focus on medical and health informatics education and AI? To answer this question, the selected publications were classified based on their education foci and the characteristics of included studies were summarized in Table 2. The publications that focused on the use of AI applications in medical and health informatics education were categorized as Category 1. These studies used various AI-based tools to enhance the learning experience. Several case studies and new initiatives to teach specific AI skills, such as ML programming languages and big data analytics software, were also identified. The publications that evaluated AI education were classified as Category 2. Category 1 studies discussed different AI applications to enhance education and summarized the impact of AI on medical and health informatics education, while Category 2 studies focused on the teaching of AI concepts.

    Table 2. Characteristics of included studies.
    View this table

    Of 26 publications, 16 (61%) investigated the use of AI applications in medical education (Category 1) and 10 publications (39%) evaluated AI education in medicine (Category 2; Figure 4).

    The first publications about the use of AI for medical applications were published in the early 1990s, and the capabilities of AI applications were restricted by technological limitations at that time. Klar and Bayer's paper [27], published in 1990, was one of the first publications about the application of AI, and they discussed the integration of expert knowledge into computer-assisted teaching in medicine. Lillehaug and Lajoie [16] proposed greater integration of AI in their 1998 paper, and they were among the first researchers who advocated for intelligent decision support systems and AI-based applications for medical education. Frize and Frasson [17] examined the role of decision support and intelligent tutoring systems in medical education, and recommended multidisciplinary studies in 2000.

    Figure 4. Classification of selected AI publications. AI: artificial intelligence; ML: machine learning.
    View this figure

    Most Category 1 studies explored innovative applications designed to improve the learning experience. For example, Chan and Zary [15] evaluated existing AI applications in medicine, determined that the primary reason to use AI in medical education was to provide feedback, and identified that significant challenges included the assessment of effectiveness and management of technical difficulties. Another systematic review conducted by Chary et al [19] identified 30 articles that assessed the application of natural language processing (NLP) to medical education. NLP is a subfield of AI and refers to intelligent communication methods using natural languages. The study revealed the benefits of NLP training in residency education and recommended strategies for its application.

    Simulation-based learning has evolved over the last decade, and the virtual environment has become essential for education. Our review identified multiple case studies about advanced virtual environments. For instance, Winkler-Schwartz et al [14] analyzed virtual reality simulators that use AI, and developed a checklist to assess studies using ML algorithms to evaluate technical skills. This study concluded that the checklist had the potential to decrease the knowledge gaps in the use of AI in surgical education. Similarly, Zhao et al [18] concluded that virtual patient systems and other distance education systems that used AI increased the efficiency of medical education. Another case study that focused on virtual patient simulators identified educational and technical challenges to enhancing the learning process with AI virtual reality applications [20]. The case study published by Bowyer et al [22] described the role of advanced virtual environments in surgical training. Moreover, Conde et al [25] recommended the use of AI applications for training and education when simulated human patients were not an option.

    Augmented reality is another form of virtual reality, and this technology superimposes images on top of the video viewer. Sitterding et al [23] described the differences between virtual reality, augmented reality, serious gaming, and gamification, and shared the preliminary findings of their pilot study, which determined that the simulation experience was similar to a real-life environment.

    In our research, we noted different innovative AI applications for special learning activities. In their 2017 article, Guimarães et al [21] reviewed current education models for anatomy education, and recommended the use of AI analytic tools to personalize the learning process. Boulet and Durning [24] discussed the application of AI to replace human ratings and assess medical education competencies. Another study that used augmented reality and AI algorithms evaluated a case study about the development of a laparoscopic box trainer to assess surgical psychomotor skills, and concluded the proposed system had potential benefits [29].

    Incorporating intelligent techniques into an adaptive e-learning system was another research group's focus. Kabassi et al [26] designed a web-based educational system for medical education, incorporated intelligent algorithms to individualize the learning experience, and shared the findings. The authors recommended further studies with more participants. Yang and Shulruf [28] designed a pilot study to demonstrate the value of AI-assisted tutoring, and determined that this additional tutoring significantly enhanced the performance of medical students.

    Our second research question involved determining the highest level of evidence of current research. The articles were classified according to the Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence ranking scheme [40]. Of 26 articles, 18 (69%) were classified as Level IV evidence (case series), and 8 (31%) were classified as Level V evidence (expert opinions). The majority of the publications had Level IV and V evidence, which are considered poor reference standards. Hence, these findings emphasize the need to design new research studies.

    Finally, our third research question was the following: what is the status of AI education in medicine and health informatics? In our study, 10 articles discussing AI education were identified. Since this topic is a relatively new area in medical and health informatics education, our findings were consistent with recent developments.

    In their review, Paranjape et al [39] summarized multiple initiatives for AI in medical education, in which students worked with data experts and solved health care problems. The authors recommended familiarizing students with AI-based clinical applications, and introducing linear algebra, calculus, and probability during different stages of medical education. Similarly, Chin-Yee and Upshur [37] discussed the random error and biased data generated by AI and ML applications, and emphasized the effect of medical education for appraising clinical judgments.

    A review conducted by Park et al [31] emphasized the importance of understanding AI to be able to validate the clinical accuracy of AI algorithms. Furthermore, multiple publications stated the need to move beyond traditional medical education, suggested a reform to align education with new practice requirements, and emphasized the role of academics and teachers in the development of appropriate AI application skills [32,33,36].

    As discussed earlier, ML is an AI technique to process massive amounts of data and make predictions using computers. Kolachalama and Garg [30] proposed the integration of ML-related content in medical student, resident, and fellow education. The authors recommended the integration of real-world clinical examples into ML courses as well as practical guidelines for choosing the right tools.

    The radiology community embraced AI and ML long before other medical specialties, and pioneered the usage of AI algorithms in advanced imaging applications. A multicenter survey of undergraduate medical students determined students' optimistic views about the implications of AI applications for radiology [38]. The French Radiology Community developed principles to regulate the use of AI tools, and recommended specific education to evaluate AI technologies [34]. Similarly, the Canadian Association of Radiologists recommended the integration of computer science, health informatics, and statistics training during residency education [35].


    Discussion

    Overview

    The technological advancements in computer and software technologies; digitization of health care data; and methodological developments in information science, philosophy, mathematics, linguistics, and psychology disciplines accelerated medical research programs that focus on ML and commercialization. Maturation of AI technologies changed the roles of clinicians, and novel decision-making processes in medical settings and innovative AI-based protocols have the potential to provide diagnostic and treatment decisions by analyzing complex data sets [41].

    Over the last decade, many AI researchers have concentrated on developing a proof of concept system for clinicians and patients. There is an increase in the number of studies that evaluate the effectiveness of intelligent reasoning. Intelligent monitoring technologies require new algorithms to detect anomalies, predict patterns, and make decisions.

    A recent independent report prepared for the UK Secretary of State for Health and Social Care explored how the health care workforce could be prepared to use digital technology. The report emphasized the skills gap in the workforce, and made some recommendations about the integration of digital health care technologies, AI, and data analytics in undergraduate curricula [42,43].

    Clinicians should have a realistic view of AI, and become familiar with the right tasks for AI in health care. Formal training for medical and health informatics students should enable them to develop AI algorithms, use AI technologies in a competent manner, and keep bias out of AI tools. Any new AI technology might encounter something new for which it has no experience, and therefore the physician should be able to assess problematic decisions and take necessary precautions when needed. Recent discussions about the need for medical education reform emphasize the shortcomings of the current model of education [44]. Overreliance on ML and AI technologies might have unintended severe adverse consequences, such as failure to recognize invalid test results [45].

    While machine learning and AI algorithms are able to handle high-dimensional data classification problems and medical image interpretation, their success rates in risk prediction and diagnosis are lower. Consequently, there is a need to determine the most appropriate application areas for AI in health care [46].

    There are different ways to implement AI in clinical practice, and clinicians and health informaticians need formal training to use the right approaches. Health informaticians and physician champions who design and develop AI-based protocols need to have a good understanding of complex algorithms, methodologies for data quality assessment, probabilistic forecasting, and comparative model assessment to work with engineers and develop reliable AI applications. Moreover, clinicians who use AI applications should become familiar with potential challenges.

    AI-based relational time pattern analysis replaced simple threshold-based diagnostic rules. Current medical education and health informatics curricula still do not provide the ability to understand AI communications and necessary skill sets to develop AI systems that can detect and analyze relational time patterns [41].

    The ability to interpret AI algorithms' mistakes and formulate the best strategies to correct these applications requires specialized training. Consequently, medical and health informatics education must emphasize algorithm-based platforms, and include relevant data analytics and AI topics in their curricula. Moreover, computer science and health informatics programs should consist of health care–focused digital skills training.

    To the best of our knowledge, no previous research has investigated the use of AI tools to enhance the learning experience and AI education of medical and health informatics students. The main findings of this systematic review are as follows: (1) Although there are several recommendations on the integration of AI into medical and health informatics curricula and some academic institutions implemented experimental training programs, AI and ML education are not a part of traditional medical and health informatics curricula yet. (2) Current medical education and health informatics accreditation standards do not require AI training, and AI competencies have not been determined. (3) Using the OCEBM Levels of Evidence classification table, the majority of studies were classified as Level IV and V, which indicates poor reference standards.

    Limitations

    Several efforts were made to design an optimal systematic review process; however, there were still many limitations. It is probable that some studies might not be listed in the peer-reviewed academic literature databases or might be published in a non-English language. Although this is a systematic review of the field, AI is a new technical discipline, particularly in medicine, and therefore the number of articles that met the inclusion criteria was limited.

    Future Directions

    Overall, the selected publications did not provide specific details about different jobs' requirements and curriculum needs. The emergence of intelligent systems in health care requires new learning modalities. Even though several organizations, agencies, and work groups such as the International Medical Informatics Association [47], the Commission on Accreditation for Health Informatics and Information Management Education [48,49], the Health Informatics Society of Australia [50], the TIGER (Technology Informatics Guiding Education Reform) Initiative [51], and the Association of American Medical Colleges [52] published skill recommendations for health informatics curricula, they have not determined specific skill sets for AI education. Furthermore, a recent study evaluated health informatics students' skills in developing AI apps and emphasized the need to develop new competencies [53].

    A specialized AI education framework for various professional fields would be useful; using the results of this systematic review, we propose a framework for specialized AI training for different domains (Figure 5). Medical students need to become familiar with clinical AI applications and predictive modeling techniques to assess biased data and evaluate innovative AI technologies. Health informatics students should become familiar with the application of appropriate ML algorithms and development of innovative clinical informatics systems. Furthermore, they should gain the hands-on skills required to extract data, manage large data sets to perform sophisticated data analytics, and develop innovative AI systems. Computer science students need specialized skill sets to work with data scientists and should become familiar with Python, R, and SQL programming languages, and data analytics tools (Figure 5).

    Figure 5. Proposed framework for specialized AI training for different professional fields. AI: artificial intelligence; DL: deep learning; ML: machine learning.
    View this figure

    Conflicts of Interest

    None declared.

    References

    1. Mayor A. Gods and robots: myths, machines, and ancient dreams of technology. In: Princeton: Princeton University Press. Princeton, NJ, USA: Princeton University Press; 2018.
    2. Moor J. The Dartmouth College Artificial Intelligence Conference: The Next Fifty Years. AI Magazine 2006;27(4):87-91. [CrossRef]
    3. Artificial intelligence. Oxford Reference.   URL: https://www.oxfordreference.com/view/10.1093/oi/authority.20110803095426960 [accessed 2020-06-22]
    4. Poole DL, Mackworth AK, Goebel R. Computational intelligence: a logical approach. New York, NY, USA: Oxford University Press; 1998:1-558.
    5. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015 May 28;521(7553):436-444. [CrossRef] [Medline]
    6. Clinical Informatics Becomes a Board-certified Medical Subspecialty Following ABMS Vote. 2011.   URL: https://www.amia.org/news-and-publications/press-release/ci-is-subspecialty [accessed 2020-03-01]
    7. Char DS, Shah NH, Magnus D. Implementing Machine Learning in Health Care — Addressing Ethical Challenges. N Engl J Med 2018 Mar 15;378(11):981-983. [CrossRef]
    8. McArthur D, Lewis M, Bishary M. The Roles Of Artificial Intelligence In Education: Current Progress And Future Prospects. JET 2005 Mar 15;1(4):42-80. [CrossRef]
    9. Popenici SAD, Kerr S. Exploring the impact of artificial intelligence on teaching and learning in higher education. Res Pract Technol Enhanc Learn 2017 Nov 23;12(1):22 [FREE Full text] [CrossRef] [Medline]
    10. Bayne S. Teacherbot: interventions in automated teaching. Teaching in Higher Education 2015 Apr 16;20(4):455-467. [CrossRef]
    11. Petit dit Dariel OJ, Raby T, Ravaut F, Rothan-Tondeur M. Developing the Serious Games potential in nursing education. Nurse Educ Today 2013 Dec;33(12):1569-1575. [CrossRef] [Medline]
    12. Nye BD. Intelligent Tutoring Systems by and for the Developing World: A Review of Trends and Approaches for Educational Technology in a Global Context. Int J Artif Intell Educ 2014 Oct 9;25(2):177-203. [CrossRef]
    13. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015 Jan 01;4(1):1 [FREE Full text] [CrossRef] [Medline]
    14. Winkler-Schwartz A, Bissonnette V, Mirchi N, Ponnudurai N, Yilmaz R, Ledwos N, et al. Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation. J Surg Educ 2019 Nov;76(6):1681-1690. [CrossRef] [Medline]
    15. Chan KS, Zary N. Applications and Challenges of Implementing Artificial Intelligence in Medical Education: Integrative Review. JMIR Med Educ 2019 Jun 15;5(1):e13930 [FREE Full text] [CrossRef] [Medline]
    16. Lillehaug S, Lajoie SP. AI in medical education—another grand challenge for medical informatics. Artificial Intelligence in Medicine 1998 Mar;12(3):197-225. [CrossRef]
    17. Frize M, Frasson C. Decision-support and intelligent tutoring systems in medical education. Clin Invest Med 2000;23:266-269. [CrossRef]
    18. Zhao H, Li G, Feng W. Research on Application of Artificial Intelligence in Medical Education. 2018 Aug 10 Presented at: International Conference on Engineering Simulation and Intelligent Control (ESAIC); 2018; Changsha, China p. 340-342. [CrossRef]
    19. Chary M, Parikh S, Manini A, Boyer E, Radeos M. A Review of Natural Language Processing in Medical Education. West J Emerg Med 2019 Jan 12;20(1):78-86 [FREE Full text] [CrossRef] [Medline]
    20. Caudell TP, Summers KL, Holten J, Hakamata T, Mowafi M, Jacobs J, et al. Virtual patient simulator for distributed collaborative medical education. Anat Rec B New Anat 2003 Jan 13;270(1):23-29 [FREE Full text] [CrossRef] [Medline]
    21. Guimarães B, Dourado L, Tsisar S, Diniz JM, Madeira MD, Ferreira MA. Rethinking Anatomy: How to Overcome Challenges of Medical Education's Evolution. Acta Med Port 2017 Feb 27;30(2):134-140 [FREE Full text] [CrossRef] [Medline]
    22. Bowyer MW, Streete KA, Muniz GM, Liu AV. Immersive Virtual Environments for Medical Training. Seminars in Colon and Rectal Surgery 2008 Jun;19(2):90-97. [CrossRef]
    23. Sitterding MC, Raab DL, Saupe JL, Israel KJ. Using Artificial Intelligence and Gaming to Improve New Nurse Transition. Nurse Leader 2019 Apr;17(2):125-130. [CrossRef]
    24. Boulet JR, Durning SJ. What we measure … and what we should measure in medical education. Med Educ 2019 Jan 14;53(1):86-94. [CrossRef] [Medline]
    25. Conde JG, De S, Hall RW, Johansen E, Meglan D, Peng GC. Telehealth innovations in health education and training. Telemed J E Health 2010 Feb;16(1):103-106 [FREE Full text] [CrossRef] [Medline]
    26. Kabassi K, Virvou M, Tsihrintzis GA, Vlachos Y, Perrea D. Specifying the personalization reasoning mechanism for an intelligent medical e-learning system on Atheromatosis: An empirical study. IDT 2008 Oct 07;2(3):179-190. [CrossRef]
    27. Klar R, Bayer U. Computer-assisted teaching and learning in medicine. International Journal of Bio-Medical Computing 1990 Jul;26(1-2):7-27. [CrossRef]
    28. Yang YY, Shulruf B. Expert-led and artificial intelligence (AI) system-assisted tutoring course increase confidence of Chinese medical interns on suturing and ligature skills: prospective pilot study. J Educ Eval Health Prof 2019 Apr 10;16:7 [FREE Full text] [CrossRef] [Medline]
    29. Alonso-Silverio GA, Pérez-Escamirosa F, Bruno-Sanchez R, Ortiz-Simon JL, Muñoz-Guerrero R, Minor-Martinez A, et al. Development of a Laparoscopic Box Trainer Based on Open Source Hardware and Artificial Intelligence for Objective Assessment of Surgical Psychomotor Skills. Surg Innov 2018 Aug 29;25(4):380-388. [CrossRef] [Medline]
    30. Kolachalama VB, Garg PS. Machine learning and medical education. NPJ Digit Med 2018 Sep 27;1(1):54 [FREE Full text] [CrossRef] [Medline]
    31. Park SH, Do KH, Kim S, Park JH, Lim Y. What should medical students know about artificial intelligence in medicine? J Educ Eval Health Prof 2019 Jul 03;16:18 [FREE Full text] [CrossRef] [Medline]
    32. Wartman SA, Combs CD. Medical Education Must Move From the Information Age to the Age of Artificial Intelligence. Academic Medicine 2018;93(8):1107-1109. [CrossRef]
    33. Wartman SA, Combs CD. Reimagining Medical Education in the Age of AI. AMA J Ethics 2019 Feb 01;21(2):E146-E152 [FREE Full text] [CrossRef] [Medline]
    34. SFR-IA Group, CERF, French Radiology Community. Artificial intelligence and medical imaging 2018: French Radiology Community white paper. Diagn Interv Imaging 2018 Nov;99(11):727-742 [FREE Full text] [CrossRef] [Medline]
    35. Tang A, Tam R, Cadrin-Chênevert A, Guest W, Chong J, Barfett J, Canadian Association of Radiologists (CAR) Artificial Intelligence Working Group. Canadian Association of Radiologists White Paper on Artificial Intelligence in Radiology. Can Assoc Radiol J 2018 May 01;69(2):120-135 [FREE Full text] [CrossRef] [Medline]
    36. Masters K. Artificial intelligence in medical education. Medical Teacher 2019 Apr 21;41(9):976-980. [CrossRef]
    37. Chin-Yee B, Upshur R. Clinical judgement in the era of big data and predictive analytics. J Eval Clin Pract 2018 Jun 13;24(3):638-645. [CrossRef] [Medline]
    38. Pinto Dos Santos D, Giese D, Brodehl S, Chon SH, Staab W, Kleinert R, et al. Medical students' attitude towards artificial intelligence: a multicentre survey. Eur Radiol 2019 Apr 6;29(4):1640-1646. [CrossRef] [Medline]
    39. Paranjape K, Schinkel M, Nannan Panday R, Car J, Nanayakkara P. Introducing Artificial Intelligence Training in Medical Education. JMIR Med Educ 2019 Dec 03;5(2):e16048 [FREE Full text] [CrossRef] [Medline]
    40. Howick J, Chalmers I, Glasziou P, Greenhalgh T, Heneghan C, Liberati A, OCEBM Levels of Evidence Working Group. The Oxford 2011 Levels of Evidence. Oxford Centre of Evidence-Based Medicine. 2014.   URL: https://www.cebm.net/wp-content/uploads/2014/06/CEBM-Levels-of-Evidence-2.1.pdf [accessed 2020-03-15]
    41. Lynn LA. Artificial intelligence systems for complex decision-making in acute care medicine: a review. Patient Saf Surg 2019 Feb 1;13(1):6 [FREE Full text] [CrossRef] [Medline]
    42. Rimmer A. Technology will improve doctors' relationships with patients, says Topol review. BMJ 2019 Feb 11;364:l661. [CrossRef] [Medline]
    43. National Health Service. The Topol Review: Preparing the healthcare workforce to deliver the digital future. 2019.   URL: https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf [accessed 2020-03-01]
    44. Mehta NB, Hull AL, Young JB, Stoller JK. Just Imagine. Academic Medicine 2013;88(10):1418-1423. [CrossRef]
    45. Cabitza F, Rasoini R, Gensini GF. Unintended Consequences of Machine Learning in Medicine. JAMA 2017 Aug 08;318(6):517-518. [CrossRef] [Medline]
    46. Maddox TM, Rumsfeld JS, Payne PRO. Questions for Artificial Intelligence in Health Care. JAMA 2019 Jan 01;321(1):31-32. [CrossRef] [Medline]
    47. Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W, et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics - First Revision. ejbi 2011. [CrossRef]
    48. European Commission. The EU Framework Programme for Research and Innovation. Horizon 2020. 2013.   URL: https://www.uv.es/operuv/docs_h2020/InfoKit_UK_240214_Final.pdf [accessed 2020-03-15]
    49. EU US eHealth Work Consortium. TIGER Initiative. HIMSS Foundation. 2018.   URL: http://www.himss.org/professionaldevelopment/tigers-euus-ehealth-work-project [accessed 2020-03-15]
    50. Martin-Sanchez F. Health Informatics Competencies Framework. Health Informatics Society of Australia. 2018.   URL: http:/​/www.​healthinformaticscertification.com/​wp-content/​uploads/​2016/​02/​CHIA-competencies-Framework_FINAL.​pdf [accessed 2020-03-15]
    51. Gugerty B, Delaney C. TIGER Informatics Competencies Collaborative (TICC) Final Report. TIGER Summit. 2009.   URL: https://tigercompetencies.pbworks.com/f/TICC_Final.pdf [accessed 2020-03-15]
    52. Souze A, Fyfe-Mills K. Bridging the Skills Gap - Workforce Development and the Future Work. ATD Public Policy Advisory Group 2018:1-56 [FREE Full text]
    53. Sapci AH, Sapci HA. Teaching Hands-On Informatics Skills to Future Health Informaticians: A Competency Framework Proposal and Analysis of Health Care Informatics Curricula. JMIR Med Inform 2020 Jan 21;8(1):e15748 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    AI: artificial intelligence
    CINAHL: Cumulative Index to Nursing and Allied Health Literature
    DL: deep learning
    IEEE: Institute of Electrical and Electronics Engineers
    ML: machine learning
    NLP: natural language processing
    OCEBM: Oxford Centre for Evidence-Based Medicine
    PRISMA-P: Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols
    TIGER: Technology Informatics Guiding Education Reform


    Edited by G Eysenbach; submitted 12.04.20; peer-reviewed by T Virgona, W Jacobowitz, E Frontoni, C Liao; comments to author 01.05.20; revised version received 06.05.20; accepted 14.06.20; published 30.06.20

    ©A Hasan Sapci, H Aylin Sapci. Originally published in JMIR Medical Education (http://mededu.jmir.org), 30.06.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.