Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 03.12.19 in Vol 5, No 2 (2019): Jul-Dec

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/16048, first published Sep 09, 2019.

This paper is in the following e-collection/theme issue:

    Viewpoint

    Introducing Artificial Intelligence Training in Medical Education

    1Amsterdam University Medical Center, Amsterdam, Netherlands

    2Department of Internal Medicine, Amsterdam University Medical Center, Amsterdam, Netherlands

    3Section Acute Medicine, Department of Internal Medicine, Vrije Universiteit University Medical Center, Amsterdam, Netherlands

    4Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore

    *these authors contributed equally

    Corresponding Author:

    Ketan Paranjape, MS, MBA

    Amsterdam University Medical Center

    De Boelelaan 1117, 1081 HV

    Amsterdam

    Netherlands

    Phone: 31 3174108035

    Email: ketanp@alumni.gsb.stanford.edu


    ABSTRACT

    Health care is evolving and with it the need to reform medical education. As the practice of medicine enters the age of artificial intelligence (AI), the use of data to improve clinical decision making will grow, pushing the need for skillful medicine-machine interaction. As the rate of medical knowledge grows, technologies such as AI are needed to enable health care professionals to effectively use this knowledge to practice medicine. Medical professionals need to be adequately trained in this new technology, its advantages to improve cost, quality, and access to health care, and its shortfalls such as transparency and liability. AI needs to be seamlessly integrated across different aspects of the curriculum. In this paper, we have addressed the state of medical education at present and have recommended a framework on how to evolve the medical education curriculum to include AI.

    JMIR Med Educ 2019;5(2):e16048

    doi:10.2196/16048

    KEYWORDS



    Global health care expenditure has been projected to grow from US $7.7 trillion in 2017 to US $10 trillion in 2022 at a rate of 5.4% [1]. This translates into health care being an average of 9% of gross domestic product among developed countries [2,3]. Some key global trends that have led to this include tax reform and policy changes in the United States that could impact the expansion of health care access and affordability (Affordable Care Act) [4], implications on the United Kingdom’s health care spend based on the decision to leave the European Union [5], population growth and rise in wealth in both China and India [6-8], implementation of socioeconomic policy reform for health care in Russia [9], attempts to make universal health care effective in Argentina [10], massive push for electronic health and telemedicine in Africa [11], and the impact of an unprecedented pace of population aging around the world [12].

    From clinicians’ perspective there are many important trends that are affecting the way they deliver care of which the growth in medical information is alarming. It took 50 years for medical information to double in 1950. In 1980, it took 7 years. In 2010, it was 3.5 years and is now projected to double in 73 days by 2020 [13]. This growth is posing a challenge to health care professionals to both retain and use it effectively to practice medicine.


    Rise of Artificial Intelligence in Health Care

    Artificial Intelligence in Health Care

    Artificial intelligence (AI) is a scientific discipline that focuses on understanding and creating computer algorithms that can perform tasks that are usually characteristics of humans [14]. AI is now gaining momentum in health care. From its early roots in Turing’s seminal paper, Computing Machinery and Intelligence [15], where he proposed the question “Can machines think?”, AI has come a long way. Examples of advances in AI include natural language processing (NLP) [16], speech recognition [17], virtual agents [18], decision management [19], machine learning [20], deep learning [21], and robotic process automation [22].

    Today, AI is being piloted in health care [23] for faster and accurate diagnosis, to augment radiology [24], reduce errors due to human fatigue, decrease medical costs [25], assist and replace dull, repetitive, and labor-intensive tasks [26], minimally invasive surgery [27], and reduce mortality rates [28].

    Challenges With Artificial Intelligence

    The rise of AI in health care and its integration into routine clinical practice is going to be a challenge. Along with changing the conventional ways physician work, the black box problem [29] and liability issues [30] are some of the most anticipated challenges.

    Black Box

    Researchers at Mount Sinai Hospital have created a deep learning algorithm that was trained on the data of 700,000 patients. This algorithm was able to predict onset of a disease such as schizophrenia with high accuracy [31]. This is even more impressive considering the fact that this condition is difficult to diagnose even for experts. The main problem with this algorithm is that there is no way to know how the system created this prediction and what factors were taken into consideration. This phenomenon is called the black box phenomenon. It would not be a precedent in medicine, nevertheless it is difficult to trust a system when there is no understanding on how it works. The physician needs to understand the inputs and the algorithm and interpret the AI-proposed diagnosis to ensure no errors are made. We also need to understand what the consequences or unintended side effects are of black box medicine, even when good outcomes can be demonstrated against a standard of care.

    Finally, many of the AI systems attempt to mimic aspects of human and animal central nervous systems that are, at large, still a black box. In a recent paper, Zador [32] argued that we have much more to learn from animal brains to unravel this phenomenon.

    Privacy and Control Over Data

    The development of AI algorithms almost as a rule requires data from a large number of patients. Google, for example, is using 46 billion data points collected from 216,221 adults’ deidentified data over 11 combined years from 2 hospitals to predict the outcomes of hospitalized patients [33,34]. This raises many concerns including relating to patient privacy and control. What happens if a patient does not want to participate in a study where their information is used in algorithm development? In the European Union, the Right to be Forgotten would allow personal data to be erased when the patient has withdrawn their consent [35]. In situations where patient data are limited, algorithm developers train the models on synthetic or hypothetical data, with the risk of generating unsafe and incorrect treatment recommendations [36]. Finally, AI systems are also vulnerable to cybersecurity attacks that could cause the algorithm to misclassify medical information [37].

    Lack of Standards for Use of Artificial Intelligence in Patient Care and Liability

    Another unresolved question related to the use of AI in health care is liability for the predictions of an algorithm. It is unclear who is liable when a patient experiences serious harm because of an inaccurate prediction. One could argue for any of the involved parties: the physician, the hospital, the company that developed the software, the person who developed the software, or even the person who delivered the data. Standards for use of AI in health care are still being developed [38,39]. New standards for clinical care, quality, safety, malpractice, and communication guidelines have to be developed to allow for greater use of AI. A recently launched AI system for autonomous detection of diabetic retinopathy carries medical malpractice and liability insurance [40,41].

    As use of AI and proactive use of tools such as chatbots [42] increases, physicians and patients will need to be aware of strengths and limitations of such technologies and be trained in how to effectively and safely use them [43,44].

    How Can Artificial Intelligence Address Today’s Physician Challenges?

    With medical information growing at a breakneck speed, physicians are having trouble keeping up. This is leading to information overload and creates pressure to memorize all this content to pass the United States Medical Licensing Examinations (USMLE) to qualify for residency positions. Physicians today are working longer hours and are also expected to deliver coordinated care [45,46] in an aging society with complex conditions and comorbidities where health care costs are increasing and regulations are putting an additional burden on administrative processes.

    AI could help physicians by amalgamating large amounts of data and complementing their decision-making process to identify diagnosis and recommend treatments. Physicians in turn need the ability to interpret the results and communicate a recommendation to the patient. In addition, AI could have an impact by alleviating the burden from physicians for performing day-to-day tasks [47]. Speech recognition could help with replacing the use of keyboards to enter and retrieve information [48]. Decision management can help with sifting enormous amounts of data and enable the physician to make an informed and meaningful decision [49,50]. Automation tools can help with managing regulatory requirements such as Protecting Access to Medicare Act and enable physicians to review the appropriate criteria before making a cost decision [51]. Finally, to help with the acute shortage of health care professionals, virtual agents could, in the future, help with some aspects of patient care and become a trusted source of information for patients [52].


    Artificial Intelligence Training in Medical Education

    State of Medical Education Today

    Physicians go through extensive periods of training before they can eventually register as specialists. Although medicine has seen major changes over the last decades, medical education is still largely based on traditional curricula [53]. The specific length of training differs between countries, but the core competencies of these curricula are globally similar [54]. After a core phase of preclinical didactics, training is mostly centered around practice-based learning [53]. Medical education is often based on 6 domains: patient care, medical knowledge, interpersonal and communication skills, practice-based learning and improvement, professionalism, and systems-based practice [55]. These fields were introduced by the Accreditation Council for Graduating Medical Education (ACGME). A large part of medical training focuses on consuming as much information as possible and learning how to apply this knowledge to patient care. This process is still largely memorization based [56]. Less time is spent on familiarizing medical students or residents with new technologies such as AI, mobile health care applications, and telemedicine [53,55,56]. In the United States, USMLE does not test on these subjects [57]. However, change seems inevitable since the 2018 annual meeting of the American Medical Association (AMA) saw the adoption of AMA’s first policy on augmented intelligence, encouraging research into how AI should be addressed in medical education [58]. In Table 1, several initiatives for incorporating AI in medical education are shown, as presented by the AMA [58].

    Table 1. Initiatives for artificial intelligence in medical education.
    View this table

    Another important technology-related aspect that is often overlooked in medical training is working with electronic health records (EHRs). EHRs have many benefits, such as improved patient safety, but also assist the implementation of AI in health care. AI algorithms use information from the EHR, and therefore, the knowledge on how to input unbiased data into the EHR is essential. Otherwise, the AI algorithm will likely be biased as well [59]. At present, training on use of EHRs for medical students and physicians is not commonly incorporated in the medical curriculum [60], resulting in the medical professional using the EHR as a replacement to capture information on paper without understanding the true potential of this technology [61]. Training on the use of EHRs usually consists of ad hoc brief introductory courses that just teach the basic skills to use the hospital’s system in practice. Quality of data and concerns on the impact of the computer on the patient-physician relationship are rarely addressed [60] and the USMLE does not test on these subjects either [57].

    How Clinical Practice Is Changing

    With the rapid digitization of health care, EHRs facilitate new ways to acquire and process valuable information that can be used to make an informed decision [62]. These advances and transitioning from an information age to the age of AI [56] change clinical practice and patient outcomes for the better. Physicians of the future will have to add to the armory of their skills and competencies, the ability to manage data, supervise AI tools, and use AI applications to make informed decisions.

    Physicians will have a crucial role in deciding which of these tools is best for their patients. In turn, this will likely change the physician-patient relationship [63]. When information processing is done mainly by computers, this highlights one of the major benefits of AI in medicine: it allows the physician to focus more on caring for and communicating with patients [64]. Finally, in the age of AI, “the physician should combine narrative, mechanistic and mathematical thinking in their training and consider the biopsycho-social model of the disease with the patient at its center.” “Computers will never substitute for self-reflective medical expert who is aware of the strengths and limitations of human beings and of an environment characterized by information overload” [65,66].

    What Will Be Asked From Physicians in the Future?

    Future physicians will need a broad range of skills to adequately use AI in clinical practice. Besides understanding the principles of medicine, physicians will also need to acquire satisfactory knowledge of mathematical concepts, AI fundamentals, data science, and corresponding ethical and legal issues. These skills will help them to use data from a broad array of sources, supervise AI tools, and recognize cases where algorithms might not be as accurate as expected [67]. Furthermore, communication and leadership skills as well as emotional intelligence will be more important than ever as AI-based systems will not be able to consider all the physical and emotional states of the patient [56]. These traits are hard to master for computers and will characterize a great physician in the age of AI.

    Practical Considerations

    Some of the time that was originally spent on memorizing medical information will now have to be devoted to other skills. This will have a major impact on the way students and residents will experience their training. The system has to change in such a way that competence will no longer be judged based on factual knowledge but rather on communication skills, emotional intelligence, and knowledge on how to use computers.

    With an overfull curriculum, there is limited interest in adopting new topics [68], although a 2016 survey by AMA shows that 85% of physicians perceive benefits from new digital tools [58]. The integration of AI-oriented education into the medical curriculum will take time as the technology evolves. A new infrastructure for learning has to be introduced, and new educators from disciplines such as computer sciences, mathematics, ethnography, and economics will need to be hired. At the moment, these subjects are not even covered by the core competencies of ACGME, but these competencies “are robust enough to adapt to changing knowledge” [69].

    To achieve a change in curriculum, many political and bureaucratic hurdles have to be overcome. Educational systems, program structures, and objectives have to change to create new learning outcomes [70]. A change can only be implemented when large amount of evidence is generated. We have not reached that stage of implementing changes for AI. Furthermore, many other fields within medicine argue that they have not received the attention they deserve [71,72]. AI needs to prove its benefits and also justify that it is an important topic for medical curriculum over other important subjects that lack adequate medical training at present.

    However, one of the most compelling arguments for the implementation of AI training in medical education is that this training will augment existing curriculum rather than replace existing coursework. When students are trained to use AI tools, focus should shift from acquiring basic knowledge on how to use the tool to a basic understanding of the underlying principles. This will enable the students to use this fundamental knowledge when current tools get outdated and new tools are introduced.

    Another practical problem is that traditional medical training revolves mainly around the interactions between an attending physician and the residents or medical students. When AI is increasingly introduced into clinical practice, this could be problematic. Many senior physicians have little to no experience with AI. AI training could be delivered via Continuing Medical Education (CME) programs and might need to be also taught by educators from outside the medical community. For example, a 2-credit CME course on AI and the Future of Clinical Practice is delivered by a computational biologist and business economists [73].


    Recommendations

    Framework

    The traditional medical curriculum, which is mostly memorization based, must follow the transition from the information age to the age of AI. Future physicians have to be taught competence in the effective integration and utilization of information from a growing array of sources [56]. To embed this knowledge into medicine, it is of the essence to start introducing these concepts from the beginning of training. In many countries, a Medical College Admission Test (MCAT) has to be taken to be admitted into medical school. The current US MCAT exam, for example, focuses on biology, chemistry, physics, psychology, sociology, and reasoning [78]. These exams could start testing on mathematical concepts such as basis of linear algebra and calculus. These concepts are vital to the elementary understanding of AI and will set the tone for the rest of the curriculum.

    In the core phase of preclinical didactics, time should be devoted to working with health data curation and quality [79], provenance [80], integration [81], and governance, working with EHRs [60], AI fundamentals, and ethics and legal issues with AI [82,83]. Course work in critical appraisal and statistical interpretation of AI and robotic technologies is also important [84]. First, these subjects could be taught in self-contained courses to teach about the fundamentals of these subjects that can be used even after current applications become outdated [68]. These self-contained courses could potentially replace and augment courses on medical informatics and statistics in the current curriculum. Second, they should also recur in clinical courses to familiarize students with the clinical applications of AI and work with EHRs in diverse settings [68]. An approach to introducing AI could be to incorporate this technology during courses such as Evidence Based Medicine [85]. As the student is taught to appraise evidence through databases such as PubMed or diagnostic tests or systematic reviews, this process could be augmented by applying concepts from data sciences, applying AI technologies such as NLP and analyzing scenarios to test them on questions of ethics and liability [86]. In addition, the students should also be trained in the fundamentals of computer and software engineering to understand the semantics behind real-world AI applications. For example, basics of hardware and software development and user experience design may also be valuable.

    During clinical rotations and residency, focus should shift toward relevant applications of AI in practice. With advancements in digital biomarkers [87] and digital therapeutics [88], students should also be trained in these technologies as they rely on AI. They have the potential to enable large-scale diagnostics and treatments in in-home environments in the near future [89]. At the end of training, the USMLE should include a substantial number of questions on data science and AI fundamentals in their final exams. Attendance of conferences on health care AI could be incentivized, so that health care professionals stay up-to-date with the latest developments. For attending physicians, extensive courses on AI and data science should be part of CME. See Table 2 for more details.

    Table 2. List of Continuing Medical Education programs on artificial intelligence in health care.
    View this table

    AI skills must also be balanced with nonanalytics and person-centered aspects of medicine to develop a more rounded doctor of the future. Other skills such as communications, empathy, shared decision making, leadership, team building, and creativity are all skills that will continue to gain importance for physicians. At the Dell Medical School at the University of Texas, Austin, the curriculum in basic sciences has been reduced in duration to accommodate training in soft skills such as leadership, creativity, and communication [63].

    To enable clinicians to think innovatively and create technology-enabled care models, multidisciplinary training is needed in implementation science, operations, and clinical informatics. The Stanford medical school has created such a program to train clinician-innovators for the digital future by introducing a human-centered design approach to graduate medical education [90]. At the Healthcare Transformation Laboratory at Massachusetts General Hospital in Boston, a 1-year fellowship is offered in health care innovation exposing resident trainees to topics in data sciences, machine learning, health care operations, services, design thinking, intellectual property, and entrepreneurship [91]. These projects are new developments and are the first steps taken to introduce AI in medical education.

    First Steps

    As not all of these interventions can be introduced simultaneously, we suggest a few first steps that will lay the foundation for the upcoming years. We suggest to start off by introducing questions on mathematical concepts into the MCAT similar to the mathematics section in the Graduate Record Examination. High quality Web-based courses on data sciences and AI fundamentals should be freely offered in the core phase of medical education. This might lead to students focusing on applications of these subjects more naturally in following years of training.

    For residents and medical students who have already finished this phase of training, courses on the fundamental subjects should be available and mandatory throughout the remaining part of their medical education. For students interested in creating new technology-enabled care models, dedicated training in health care innovation during a gap year during the clinical years or after residency should be encouraged. For attending physicians, introductory courses and refresher courses should also be made available. Extensive training is especially necessary for this group so that they can partly take back the task of educating medical students and residents on these subjects in the future. Table 3 lists suggested content that can be added to the various phases of medical education. Table 4 lists a small subset of rapidly evolving AI in health care conferences that physicians and trainees can attend to learn more about this technology and its applications in health care.

    Table 3. Recommendations per stage of medical education.
    View this table
    Table 4. List of artificial intelligence in health care conferences.
    View this table

    Conclusions

    Physicians and machines working in combination have the greatest potential to improve clinical decision making and patient health outcomes [114]. AI can curate and process more data such as medical records, genetic reports, pharmacy notes, and environment data and in turn retain, access, and analyze more medical information. However, it cannot replace the art of caring. As AI and its application become mainstream in health care, medical students, residents, fellows, and practicing physicians need to have knowledge of AI, data sciences, EHR fundamentals, and ethics and legal issues concerning AI. Medical schools will need to include them as part of the curriculum. A staged approach to educating the medical student through their journey is recommended.

    AI will enable faster and accurate diagnosis, augment radiology, reduce errors due to human fatigue, decrease medical costs, assist and replace dull, repetitive, and labor-intensive tasks, minimally invasive surgery, and reduce mortality rates.

    With the global health care expenditure projected to reach US $10 trillion by 2022, AI has the invaluable potential to advance the quadruple aim in health care—enhance the patient experience, improve population health, reduce costs, and improve the provider experience [115,116].

    Conflicts of Interest

    KP has written this paper as part of his PhD studies. He is a vice president at Roche. There is no conflict of interest with his employment at Roche. None of the rest of the authors declare any conflicts of interest.

    References

    1. World Health Organization. 2018. Current health expenditure (CHE) as percentage of gross domestic product (GDP)   URL: http://apps.who.int/gho/data/node.main.GHEDCHEGDPSHA2011 [accessed 2019-10-31]
    2. Committee for a Responsible Federal Budget. American Health Care: Health Spending and the Federal Budget   URL: https://www.crfb.org/papers/american-health-care-health-spending-and-federal-budget [accessed 2019-08-25]
    3. Commonwealth Fund. Health Care Spending in the United States and Other High-Income Countries   URL: https:/​/www.​commonwealthfund.org/​publications/​journal-article/​2018/​mar/​health-care-spending-united-states-and-other-high-income [accessed 2019-08-25]
    4. The US Department of Health and Human Services. About the Affordable Care Act   URL: https://www.hhs.gov/healthcare/about-the-aca/index.html [accessed 2019-07-27]
    5. Fahy N, Hervey T, Greer S, Jarman H, Stuckler D, Galsworthy M, et al. How will Brexit affect health and health services in the UK? Evaluating three possible scenarios. Lancet 2017 Dec 4;390(10107):2110-2118. [CrossRef] [Medline]
    6. Deloitte. China's healthcare provider market: Riding the waves of reform   URL: https:/​/www2.​deloitte.com/​cn/​en/​pages/​life-sciences-and-healthcare/​articles/​china-healthcare-provider-market.​html [accessed 2019-07-27]
    7. Global Burden of Disease Health Financing Collaborator Network. Trends in future health financing and coverage: future health spending and universal health coverage in 188 countries, 2016-40. Lancet 2018 May 5;391(10132):1783-1798 [FREE Full text] [CrossRef] [Medline]
    8. Horton R. Offline: The new politics of health in India. Lancet 2018 Sep 15;392(10151):902 [FREE Full text] [CrossRef] [Medline]
    9. Rand Corporation. The Crisis of Russian Health Care and Attempts at Reform   URL: https://www.rand.org/pubs/conf_proceedings/CF124/CF124.chap5.html [accessed 2019-07-27]
    10. Rubinstein A, Zerbino MC, Cejas C, López A. Making universal health care effective in Argentina: a blueprint for reform. Health Syst Reform 2018;4(3):203-213. [CrossRef] [Medline]
    11. The World Economic Forum. How new tech can propel Africa to the forefront of healthcare   URL: https://www.weforum.org/agenda/2018/01/africa-lead-healthcare-fourth-industrial-revolution/ [accessed 2019-07-27]
    12. World Health Organization. Ageing and health   URL: https://www.who.int/news-room/fact-sheets/detail/ageing-and-health [accessed 2019-10-14]
    13. Densen P. Challenges and opportunities facing medical education. Trans Am Clin Climatol Assoc 2011;122:48-58 [FREE Full text] [Medline]
    14. Schinkel M, Paranjape K, Nannan Panday RS, Skyttberg N, Nanayakkara P. Clinical applications of artificial intelligence in sepsis: A narrative review. Comput Biol Med 2019 Oct 7;115:103488 [FREE Full text] [CrossRef] [Medline]
    15. Turing AM. Computing Machinery and Intelligence. Mind 1950 Oct;LIX(236):433-460 [FREE Full text] [CrossRef]
    16. Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, et al. Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review. J Biomed Inform 2017 Sep;73:14-29 [FREE Full text] [CrossRef] [Medline]
    17. Gold B, Morgan N, Ellis D. Speech and Audio Signal Processing. Hoboken, NJ, USA: John Wiley & Sons, Inc; 2011.
    18. Shaked NA. Avatars and virtual agents - relationship interfaces for the elderly. Healthc Technol Lett 2017 Jul;4(3):83-87 [FREE Full text] [CrossRef] [Medline]
    19. Lockwood T. Medium – Get smarter about what matters to you. Artificial intelligence can now explain its own decision making   URL: https:/​/medium.​com/​datadriveninvestor/​artificial-intelligence-can-now-explain-its-own-decision-making-71fe14d2f53f [accessed 2019-07-28]
    20. Erickson BJ, Korfiatis P, Akkus Z, Kline TL. Machine learning for medical imaging. Radiographics 2017;37(2):505-515 [FREE Full text] [CrossRef] [Medline]
    21. Miotto R, Wang F, Wang S, Jiang X, Dudley J. Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform 2018 Nov 27;19(6):1236-1246 [FREE Full text] [CrossRef] [Medline]
    22. Alexovic M, Dotsikas Y, Bober P, Sabo J. Achievements in robotic automation of solvent extraction and related approaches for bioanalysis of pharmaceuticals. J Chromatogr B Analyt Technol Biomed Life Sci 2018 Aug 15;1092:402-421. [CrossRef] [Medline]
    23. Yu K, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng 2018 Oct;2(10):719-731. [CrossRef] [Medline]
    24. Dreyer KJ, Geis JR. When machines think: radiology's next frontier. Radiology 2017 Dec;285(3):713-718. [CrossRef] [Medline]
    25. Guo J, Li B. The application of medical artificial intelligence technology in rural areas of developing countries. Health Equity 2018;2(1):174-181 [FREE Full text] [CrossRef] [Medline]
    26. Robeznieks A. American Medical Association. 3 ways medical AI can improve workflow for physicians   URL: https:/​/www.​ama-assn.org/​practice-management/​digital/​3-ways-medical-ai-can-improve-workflow-physicians [accessed 2019-02-03]
    27. Wang Z, Majewicz Fey A. Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int J Comput Assist Radiol Surg 2018 Dec;13(12):1959-1970. [CrossRef] [Medline]
    28. Weng SF, Vaz L, Qureshi N, Kai J. Prediction of premature all-cause mortality: a prospective general population cohort study comparing machine-learning and standard epidemiological approaches. PLoS One 2019;14(3):e0214365 [FREE Full text] [CrossRef] [Medline]
    29. Castelvecchi D. Can we open the black box of AI? Nature 2016 Oct 6;538(7623):20-23. [CrossRef] [Medline]
    30. Thomas S. Bill of Health. Artificial Intelligence and Medical Liability (Part II)   URL: http:/​/blogs.​harvard.edu/​billofhealth/​2017/​02/​10/​artificial-intelligence-and-medical-liability-part-ii/​#more-20718 [accessed 2017-12-24]
    31. Miotto R, Li L, Kidd BA, Dudley JT. Deep Patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci Rep 2016 May 17;6:26094 [FREE Full text] [CrossRef] [Medline]
    32. Zador AM. A critique of pure learning and what artificial neural networks can learn from animal brains. Nat Commun 2019 Aug 21;10(1):3770 [FREE Full text] [CrossRef] [Medline]
    33. Gershgorn D. Quartz. 2018 Jan 27. Google is using 46 billion data points to predict the medical outcomes of hospital patients   URL: https:/​/qz.​com/​1189730/​google-is-using-46-billion-data-points-to-predict-the-medical-outcomes-of-hospital-patients/​ [accessed 2018-11-01]
    34. Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. NPJ Digit Med 2018;1:18 [FREE Full text] [CrossRef] [Medline]
    35. General Data Protection Regulation. Right to be Forgotten   URL: https://gdpr-info.eu/issues/right-to-be-forgotten/ [accessed 2019-10-15]
    36. Ross C. Stat News. IBM’s Watson supercomputer recommended ‘unsafe and incorrect’ cancer treatments, internal documents showd   URL: https://www.statnews.com/2018/07/25/ibm-watson-recommended-unsafe-incorrect-treatments/ [accessed 2019-08-10]
    37. Finlayson SG, Bowers JD, Ito J, Zittrain JL, Beam AL, Kohane IS. Adversarial attacks on medical machine learning. Science 2019 Mar 22;363(6433):1287-1289. [CrossRef] [Medline]
    38. Doshi-Velez F. Food and Drug Administration. Considerations for the Practical Impact of AI in Healthcare   URL: https://www.fda.gov/media/107792/download
    39. Zweig M, Evans B. Rock Health. 2018 Jun 11. How should the FDA approach the regulation of AI and machine learning in healthcare?   URL: https:/​/rockhealth.​com/​how-should-the-fda-approach-the-regulation-of-ai-and-machine-learning-in-healthcare/​ [accessed 2019-09-01]
    40. Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med 2018;1:39 [FREE Full text] [CrossRef] [Medline]
    41. reddit. 2018. AskScience AMA Series   URL: https://www.reddit.com/r/askscience/comments/9d5id2/askscience_ama_series_im_michael_abramoff_a/ [accessed 2018-10-21]
    42. Bates M. Health care chatbots are here to help. IEEE Pulse 2019;10(3):12-14. [CrossRef]
    43. Palanica A, Flaschner P, Thommandram A, Li M, Fossat Y. Physicians' perceptions of chatbots in health care: cross-sectional web-based survey. J Med Internet Res 2019 May 5;21(4):e12887 [FREE Full text] [CrossRef] [Medline]
    44. Vaidyam AN, Wisniewski H, Halamka JD, Kashavan MS, Torous JB. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry 2019 Jul;64(7):456-464. [CrossRef] [Medline]
    45. Weaver SJ, Che XX, Petersen LA, Hysong SJ. Unpacking care coordination through a multiteam system lens: a conceptual framework and systematic review. Med Care 2018 Mar;56(3):247-259. [CrossRef] [Medline]
    46. Elliott J, Stolee P, Boscart V, Giangregorio L, Heckman G. Coordinating care for older adults in primary care settings: understanding the current context. BMC Fam Pract 2018 Aug 7;19(1):137 [FREE Full text] [CrossRef] [Medline]
    47. Zimmerschied C. American Medical Association. 2017 May 24. AI, teamed with physicians’ intelligence, could improve care   URL: https:/​/www.​ama-assn.org/​practice-management/​digital/​ai-teamed-physicians-intelligence-could-improve-care [accessed 2019-07-28]
    48. Hodgson T, Coiera E. Risks and benefits of speech recognition for clinical documentation: a systematic review. J Am Med Inform Assoc 2016 May;23(e1):e169-e179 [FREE Full text] [CrossRef] [Medline]
    49. Obermeyer Z, Emanuel EJ. Predicting the future - big data, machine learning, and clinical medicine. N Engl J Med 2016 Oct 29;375(13):1216-1219 [FREE Full text] [CrossRef] [Medline]
    50. Levenson SA. The health care decision-making process framework. Md Med 2010;11(1):13-17. [Medline]
    51. Hentel KD, Menard A, Mongan J, Durack JC, Johnson PT, Raja AS, et al. What physicians and health organizations should know about mandated imaging appropriate use criteria. Ann Intern Med 2019;170(12):880. [CrossRef]
    52. Safavi K, Dare F. Harvard Business Review. 2018 Apr 3. Virtual Health Care Could Save the US Billions Each Year   URL: https://hbr.org/2018/04/virtual-health-care-could-save-the-u-s-billions-each-year [accessed 2019-07-28]
    53. Pfeifer CM. A progressive three-phase innovation to medical education in the United States. Med Educ Online 2018 Dec;23(1):1427988 [FREE Full text] [CrossRef] [Medline]
    54. Weggemans MM, van Dijk B, van Dooijeweert B, Veenendaal AG, Ten Cate O. The postgraduate medical education pathway: an international comparison. GMS J Med Educ 2017;34(5):Doc63 [FREE Full text] [CrossRef] [Medline]
    55. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system — rationale and benefits. N Engl J Med 2012 Mar;366(11):1051-1056. [CrossRef]
    56. Wartman SA, Combs CD. Medical education must move from the information age to the age of artificial intelligence. Acad Med 2018;93(8):1107-1109. [CrossRef]
    57. United States Medical Licensing Examination. 2019. USMLE Content Outline   URL: https://www.usmle.org/pdfs/usmlecontentoutline.pdf [accessed 2019-08-01]
    58. American Medical Association. 2016. Digital Health Study Physicians’ motivations and requirements for adopting digital clinical tools   URL: https://www.ama-assn.org/practice-management/digital [accessed 2019-08-01]
    59. American Medical Association. Augmented intelligence in health care   URL: https://www.ama-assn.org/system/files/2019-01/augmented-intelligence-policy-report.pdf [accessed 2019-07-15]
    60. Wald HS, George P, Reis SP, Taylor JS. Electronic health record training in undergraduate medical education. Acad Med 2014;89(3):380-386. [CrossRef]
    61. Evans RS. Electronic health records: then, now, and in the future. Yearb Med Inform 2016;25(Suppl 1):S48-S61. [CrossRef]
    62. Celi LA, Davidzon G, Johnson AE, Komorowski M, Marshall DC, Nair SS, et al. Bridging the health data divide. J Med Internet Res 2016 Dec 20;18(12):e325 [FREE Full text] [CrossRef] [Medline]
    63. Johnston SC. Anticipating and training the physician of the future: the importance of caring in an age of artificial intelligence. Acad Med 2018 Aug;93(8):1105-1106. [CrossRef] [Medline]
    64. Mintz LJ, Stoller JK. A systematic review of physician leadership and emotional intelligence. J Grad Med Educ 2014;6(1):21-31. [CrossRef]
    65. de Leon J. Teaching medical students how to think: narrative, mechanistic and mathematical thinking. Actas Esp Psiquiatr 2018 Jul;46(4):133-145 [FREE Full text] [Medline]
    66. de Leon J, De las Cuevas C. The Art of Pharmacotherapy. J Clin Psychopharmacol 2017;37(2):131-137. [CrossRef]
    67. Park SH, Do K, Kim S, Park JH, Lim Y. What should medical students know about artificial intelligence in medicine? J Educ Eval Health Prof 2019;16:18 [FREE Full text] [CrossRef] [Medline]
    68. Shortliffe EH. Biomedical informatics in the education of physicians. J Am Med Assoc 2010 Oct 15;304(11):1227-1228. [CrossRef] [Medline]
    69. DeJong C, Lucey CR, Dudley RA. Incorporating a new technology while doing no harm, virtually. J Am Med Assoc 2015 Dec 8;314(22):2351-2352. [CrossRef] [Medline]
    70. Rudhumbu N. DSpace Home. 2015. The role of academic middle managers in the planning and implementation of curriculum change in private higher education institutions in Botswana   URL: http://libdspace.ufh.ac.za/handle/20.500.11837/562 [accessed 2019-10-31]
    71. Mishra S. Do we need to change the medical curriculum: regarding the pain of others. Indian Heart J 2015;67(3):187-191 [FREE Full text] [CrossRef] [Medline]
    72. Mogre V, Stevens FC, Aryee PA, Amalba A, Scherpbier AJ. Why nutrition education is inadequate in the medical curriculum: a qualitative study of students' perspectives on barriers and strategies. BMC Med Educ 2018 Mar 12;18(1):26 [FREE Full text] [CrossRef] [Medline]
    73. Massachusetts Medical Society. Artificial Intelligence and the Future of Clinical Practice   URL: http:/​/www.​massmed.org/​Continuing-Education-and-Events/​Online-CME/​Courses/​Artificial-Intelligence/​Artificial-Intelligence-and-the-Future-of-Clinical-Practice/​ [accessed 2019-09-01]
    74. RSNA Education. Intro to AI and Machine Learning: Why All the Buzz? (2018) - On Demand Webinar   URL: http://education.rsna.org/diweb/catalog/item/eid/1008312688 [accessed 2019-09-01]
    75. eMedEvents. Current Applications and Future of Artificial Intelligence in Cardiology 2019   URL: https:/​/www.​emedevents.com/​c/​medical-conferences-2019/​current-applications-and-future-of-artificial-intelligence-in-cardiology [accessed 2019-09-01]
    76. UPMC Physician Resources. Artificial Intelligence & Machine Learning: Application in the Care of Children   URL: https:/​/www.​upmcphysicianresources.com/​cme-courses/​artificial-intelligence-machine-learning-application-in-the-care-of-children [accessed 2018-07-15]
    77. Stanford Center for Continuing Medical Education. Artificial Intelligence in Healthcare: The Hope, The Hype, The Promise, The Peril   URL: https://stanford.cloud-cme.com/default.aspx?P=0&EID=34240 [accessed 2019-09-01]
    78. Association of American Medical Colleges. The MCAT Essentials for Testing Year 2019   URL: https:/​/aamc-orange.​global.ssl.fastly.net/​production/​media/​filer_public/​66/​4c/​664c14ab-f87e-435a-82ff-d112a89cc3f9/​essentials_2019_final_10262018.​pdf [accessed 2019-07-01]
    79. van Ooijen PM. Quality and curation of medical images and data. In: Ranschaert ER, Morozov S, Algra PR, editors. Artificial Intelligence In Medical Imaging: Opportunities, Applications And Risks. Switzerland: Springer; 2019:247-255.
    80. Roemerman S. Forbes. Four Reasons Data Provenance Is Vital For Analytics And AI   URL: https:/​/www.​forbes.com/​sites/​forbestechcouncil/​2019/​05/​22/​four-reasons-data-provenance-is-vital-for-analytics-and-ai/​#503e0cdb57d6 [accessed 2019-08-09]
    81. Prasser F, Kohlbacher O, Mansmann U, Bauer B, Kuhn K. Data Integration for Future Medicine (DIFUTURE). Methods Inf Med 2018;57(Suppl 1):e57-e65. [CrossRef]
    82. Price WN. Michigan Law Review. 2017. Regulating Black-Box Medicine   URL: https://michiganlawreview.org/regulating-black-box-medicine/ [accessed 2019-06-15]
    83. Price WN. Search eLibrary: SSRN. 2017. Medical Malpractice and Black-Box Medicine   URL: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2910417 [accessed 2019-10-31]
    84. The Topol Review.   URL: https://topol.hee.nhs.uk/ [accessed 2019-08-09]
    85. McEwen H. NEOMED Library Guides. Evidence-based Medicine I Course Guide: Introduction   URL: https://libraryguides.neomed.edu/EBM_I_Course [accessed 2019-09-10]
    86. Beam AL, Kohane IS. Translating artificial intelligence into clinical care. J Am Med Assoc 2016 Dec 13;316(22):2368-2369. [CrossRef] [Medline]
    87. Coravos A, Khozin S, Mandl KD. Erratum: Author Correction: Developing and adopting safe and effective digital biomarkers to improve patient outcomes. NPJ Digit Med 2019;2:40 [FREE Full text] [CrossRef] [Medline]
    88. Sverdlov O, van Dam J, Hannesdottir K, Thornton-Wells T. Digital therapeutics: an integral component of digital innovation in drug development. Clin Pharmacol Ther 2018 Jul;104(1):72-80. [CrossRef] [Medline]
    89. Alami H, Gagnon M, Fortin J. Digital health and the challenge of health systems transformation. MHealth 2017;3:31 [FREE Full text] [CrossRef] [Medline]
    90. Carter J, Bababekov YJ, Majmudar MD. Training for our digital future: a human-centered design approach to graduate medical education for aspiring clinician-innovators. NPJ Digit Med 2018;1:26 [FREE Full text] [CrossRef] [Medline]
    91. Healthcare Transformation Lab. Healthcare Transformation Lab’s Impact Report 2018   URL: http://healthcaretransformation.org/2019/03/05/htl-impact-report-2018/ [accessed 2019-08-09]
    92. ETS Home. GRE Mathematics Subject Test   URL: https://www.ets.org/gre/subject/about/content/mathematics [accessed 2019-09-01]
    93. Healthdata. About   URL: https://healthdata.gov/node/1 [accessed 2019-09-01]
    94. Society of General Internal Medicine. Public Datasets   URL: https://www.sgim.org/communities/research/dataset-compendium/public-datasets-description# [accessed 2019-09-01]
    95. UCSF Data Resources.   URL: https://data.ucsf.edu/ [accessed 2019-09-01]
    96. MIT OpenCourseWare. Artificial Intelligence | Electrical Engineering and Computer Science   URL: https:/​/ocw.​mit.edu/​courses/​electrical-engineering-and-computer-science/​6-034-artificial-intelligence-fall-2010/​ [accessed 2019-09-01]
    97. Wilk A. SAO/NASA ADS - Harvard University. 2019. Teaching AI, Ethics, Law and Policy   URL: https://arxiv.org/abs/1904.12470 [accessed 2019-10-31]
    98. Duke Law. Artificial Intelligence (AI)   URL: https://law.duke.edu/dclt/ai/ [accessed 2019-09-01]
    99. RPMS EHR Training Courses.   URL: https://www.ihs.gov/ehr/training/ [accessed 2019-09-01]
    100. Canadian Agency for Drugs and Technologies in Health. An Overview of Clinical Applications of Artificial Intelligence   URL: https://www.cadth.ca/dv/ieht/overview-clinical-applications-artificial-intelligence [accessed 2019-09-01]
    101. Federation Of American Scientists. JASON: Artificial Intelligence for Health Care   URL: https://fas.org/blogs/secrecy/2018/02/ai-health-care/ [accessed 2019-09-01]
    102. Center for Artificial Intelligence in Medicine & Imaging.   URL: https://aimi.stanford.edu/ [accessed 2019-09-01]
    103. Artificial Intelligence in Healthcare Accelerated Program.   URL: http://aihap.mgh.harvard.edu/program-info/ [accessed 2019-09-01]
    104. Udemy. The Data Science Course 2019: Complete Data Science Bootcamp   URL: https://www.udemy.com/the-data-science-course-complete-data-science-bootcamp/ [accessed 2019-09-01]
    105. edX. Data science courses on edX   URL: https://www.edx.org/course/subject/data-science [accessed 2019-09-01]
    106. Coursera. Data Science Specialization   URL: https://www.coursera.org/specializations/jhu-data-science [accessed 2019-09-01]
    107. Ai4 Artificial Intelligence Business Conferences. Ai4 Healthcare   URL: https://ai4.io/healthcare/ [accessed 2019-09-01]
    108. AI World Conference & Expo. AI in Healthcare   URL: https://aiworld.com/Healthcare-AI [accessed 2019-09-01]
    109. HIMSS conference. Machine Learning & AI for Healthcare Forum   URL: https://www.himssconference.org/education/specialty-programs/machine-learning-ai-healthcare [accessed 2019-09-01]
    110. MIT | Club of Northern California. AI in Healthcare @ JP Morgan Healthcare Conference   URL: https://www.mitcnc.org/events/ai-in-healthcare/ [accessed 2019-09-01]
    111. RSNA. RSNA Spotlight Course   URL: https://www.rsna.org/spotlight/ai-san-francisco [accessed 2019-09-01]
    112. AMIA | Informatics Professionals. AMIA 2019 Clinical Informatics Conference Topics and Keywords   URL: https://www.amia.org/cic2019/topics-keywords [accessed 2019-09-01]
    113. Association for the Advancement of Artificial Intelligence.   URL: http://aaai.org/ [accessed 2019-10-13]
    114. American Medical Association. AMA passes first policy recommendations on augmented intelligence   URL: https://www.ama-assn.org/ama-passes-first-policy-recommendations-augmented-intelligence [accessed 2018-10-15]
    115. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood) 2008;27(3):759-769. [CrossRef] [Medline]
    116. Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med 2014;12(6):573-576. [CrossRef] [Medline]


    Abbreviations

    AI: artificial intelligence
    ACGME: Accreditation Council for Graduating Medical Education
    AMA: American Medical Association
    CME: Continuing Medical Education
    EHR: electronic health record
    MCAT: Medical College Admission Test
    NLP: natural language processing
    USMLE: United States Medical Licensing Examinations


    Edited by G Eysenbach; submitted 09.09.19; peer-reviewed by C Khoury, A Palanica; comments to author 11.10.19; revised version received 16.10.19; accepted 20.10.19; published 03.12.19

    ©Ketan Paranjape, Michiel Schinkel, Rishi Nannan Panday, Josip Car, Prabath Nanayakkara. Originally published in JMIR Medical Education (http://mededu.jmir.org), 03.12.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.