For the construction of an AIFM-ed framework, we followed the analysis, design, development, implementation, and evaluation model for instructional design, using the first 3 activities to guide our work. We followed a two-step approach suggested by Redwood-Campbell et al [36] for framework development, wherein (1) a review of the literature was made focusing on curriculum frameworks and core competencies for AI education in medicine [34,35] and (2) a working group used qualitative or consensus methods for final development of the framework.

Our scoping review aimed to synthesize knowledge from the literature on curriculum frameworks and current educational programs that focus on the teaching and learning of AI for medical students, residents, and practicing physicians, and adhered to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. Details of this comprehensive study have been published elsewhere [34,35]. Our review specifically identified several AI educational curricula programs (eg, courses, workshops, webinars, and projects) and 2 curriculum frameworks for AI education, one outlining a broad framework for any level of education [37], while the other described a complete framework for ophthalmology residency education [38].

The outcome of our review was the identification of early concepts that could be applied to elements of the curriculum framework for family medicine and AI [34,35]. This initial curriculum framework was later used during the panel discussion as part of the co-development and redesigning of the framework. This discussion applied the curriculum framework structure described by Obadeji [39], which examines six common elements: (1) the need and the purpose of a curriculum or a program, (2) learning objectives and outcomes, (3) course content that will facilitate the accomplishment of the objectives or learning outcomes, (4) organization of the content, (5) implementation of curriculum, and (6) curriculum evaluation and refinement. This study examined all elements except the final element (curriculum evaluation and refinement). The initial framework was deemed successful by the expert team based on the following indicators: relevance to medical educators and curriculum designers, alignment to current family medicine competencies and educational goals, clarity of AI-specific content, and its potential for further validation. However, we acknowledge that further studies are needed.

The expert panel methodology follows the SRQR (Standards for Reporting Qualitative Research) checklist [40]. Expert panels help to attempt to reach consensus on controversial subjects [41,42] such as the risk of AI tools leading to reduced proficiency in independent critical thinking and clinical judgment among physicians. The use of qualitative consensus methods for curriculum development facilitates input from a wide range of stakeholders (eg, physicians and curriculum developers) in order to assess and validate expert knowledge [43]. The use of expert panel discussions to assist in creating curricula has become established in pedagogical research and development [44]. Examples within the field of medicine include discussions around social determinants of health for undergraduate medical education [45], telemedicine opportunities for postgraduate medical education [46], and geriatric oncology in continuing medical education [47].

Our panel size fell within the recommended average of 8 members or a median of 6 [48]. The definition of an expert in our case is flexible due to the limited knowledge and experience on this emerging topic; this is emphasized by Duncan et al [49], who state that, “[t]oo narrow a definition, however, can restrict the number of potential participants.” In our case, we chose experts according to the definitions of Fink et al [41], which state that they must be, “representative of their professional group, with either sufficient expertise not to be disputed or the power required to instigate the findings.” This was reinforced by Mead and Mosely [50], which state that, “experts can be defined in a number of ways, such as their position in a hierarchy [...] or as recommended by other participants in a study.” Therefore, from these definitions, we selected panelists based on their academic qualifications, their number and relevance of AI-related publications, professional experience within the development, implementation or research of AI, and finally, any participation in AI-specific projects or conferences.

For panel 1, we reached out to family medicine (clinical) educators from affiliated universities and professional organizations across Canada via email. Snowballing by this initial group generated the names of others known as family medicine educators. For panel 2, family medicine residents were invited from an initial group of residents who were knowledgeable and aware of AI, and that initial group helped to recruit relevant residents for this study.

Each participant voluntarily participated in the study by providing their explicit consent and agreement, which was confirmed through email correspondence. To uphold confidentiality, data were safeguarded through limited, secure data access, the disposal of audiotapes after transcription, and the anonymous analysis of transcripts.

This study involved a panel discussion with experts, which does not require formal ethics board approval under the Economic and Social Research Council Framework for Research Ethics guidelines [51]. According to these guidelines as well as Canada’s Tri-Council Policy Statement on Ethical Conduct for Research Involving Humans, research that presents minimal risk and does not involve sensitive information may be exempt from formal ethics review [52]. This study adhered to these recognized guidelines, ensuring that all participants were treated in accordance with principles of research integrity, voluntary participation, and informed consent.

Input from 2 different types of panelists was desired, and they were included as participants within 2 distinct expert panels. The first included family medicine educators practicing in Canada who were somewhat knowledgeable or have expertise in AI education. No limitations were placed on years of practice experience, years of knowledge or experience in AI, language proficiency, work setting, or the types of patients for whom they provided care. The second expert panel included participants who were at the time of the study family medicine residents at McGill University, and who were somewhat knowledgeable in AI. No limitations were placed on language proficiency, years of knowledge or experience in AI, work settings, or the types of patients they provided care for.

We conducted a recorded session of each expert panel via Zoom (version 5.16.10; Zoom Video Communications). The use of a web-based expert panel minimizes costs associated with travel; it also mitigates potential biases linked to panelists [53]. Each web-based expert panel discussion was approximately 2 hours long, followed the same format, used congruent discussion guides, and was facilitated by 2 members of the research team (RT and SAR). The discussions began with a brief presentation given by RT on the results of the first step of the project, that is, the comprehensive scoping review in the field [34,35]. Following the presentation, each of the five elements of the curriculum framework: (1) the need and the purpose of a curriculum or a program, (2) learning objectives and outcomes, (3) course content that will facilitate the accomplishment of the objectives or learning outcomes, (4) organization of the content, and (5) implementation of the curriculum were discussed sequentially and at length. When presenting each element, participants were invited to respond and discuss their opinions and thoughts related to each element, allowing for the co-development and redesigning of the framework together.

Expert panel discussion data were analyzed using content analysis strategies [54,55] as previously used in a study developed for a training model for nurses using a literature review and expert panel discussions, in which data were analyzed using a descriptive qualitative approach that includes content analysis [56]. In our work, the preparation phase included transcribing the data, immersing in the data, and obtaining a sense of whole through reading the transcript multiple times. In our study, once the recordings from the expert panel discussions were received, one of the authors (RT) listened to the entire recording and subsequently transcribed it verbatim. The next stage of data analysis was the organizing phase, in which open coding and the creating of categories were conducted along with the grouping of codes under higher-order headings. These were carried out by one of the authors (RT) and verified by the senior author (SAR).

As the analysis of data used an inductive approach, no prior coding systems were used, such that coded categories were derived directly from the data [55]. Sentences and phrases from the panelists were captured. In vivo coding was used to prioritize participants’ language and perspectives, while descriptive coding aided in categorizing key themes. Two independent coders reviewed the data (RT and SAR), with discrepancies resolved through discussion between coders and the research team. Saturation was achieved when no new themes emerged during the coding of the final transcript. The final step included the presentation of the final curriculum framework, which resulted from the incidence of codes and categories and its relation to the literature. Codes and categories derived were prioritized and highlighted with how frequently they appeared during the panel discussion as well as the overlap between both groups. These highlighted findings were then compared with existing literature to either support or challenge them. If these codes and categories were supported by the literature, they were subsequently integrated into the framework.

A total of 37 educator and resident experts were invited, 14 for the educator group and 23 for the resident group. Ultimately, 8 from the former and 6 from the latter group participated, for a total of 14 participants. Scheduling problems were the most common reasons for nonparticipation. The characteristics of those included in the expert panel discussion are displayed in Table 1.

In this study, we introduced a novel and evidence-based initial curriculum framework, that is, AIFM-ed developed for AI literacy education in family medicine postgraduate training. This systematically co-developed framework used a combination of validated methods including a comprehensive scoping review, resident and educator panel discussions, and the involvement of interdisciplinary experts in the field. During the development and cocreation of this framework, several key findings emerged. These include the crucial role of multiple resource partners and innovative practices when integrating AI educational content in family medicine education. For example, AI technology vendors specializing in health care, upcoming startups, and AI-focused organizations.

Furthermore, educators and residents stressed the importance of learning about the application of AI-based tools and simulating their use as a method of learning. Several innovative practices have already been implemented including case-based learning and flipped classroom models. Moreover, the adoption of AI-based tools can be diverse depending on its context (eg, teaching and learning and clinical practice) with several barriers and enablers. Additionally, the study identified several challenges in effectively integrating an AI curriculum framework into existing educational structures. These include the lack of AI definition standardization, the reduced urgency in practice due to the lack of time and resources, as well as the capacity to balance theoretical and practical curricular content.

During the development of the AIFM-ed curriculum framework, several resource partners were identified when discussing the implementation of AI education in family medicine. Interprofessional collaboration within multidisciplinary teams is essential in order for an AI curriculum to be effective [62]. Other researchers emphasize this sentiment when listing their recommendations of ensuring a responsible integration of AI technologies in medical education [63]. This multidisciplinary team and resource partners may include several stakeholders such as nurses, social workers, epidemiologists, AI experts, data engineers, software developers, and patients [64]. Other resource partners identified included AI technology vendors specializing in health care, upcoming startups, and AI-focused organizations. Residents brought up the concept that AI-based tools and AI in general will substantially change in the future (eg, improved tools, systems, and integrations) and thus stressed the importance of continuous partnerships with other professionals in order for relevant information and AI tools.

Educators emphasized that they were unaware of many AI-based tools for patient support and were thus apprehensive in advocating for AI-based tools. Therefore, family physicians and other primary care team members (eg, administrative staff and nurses) should share AI-based tools and resources, when available and known, with other family physician and family medicine communities. Additionally, residents have suggested that before advocating or suggesting AI-based tools, a list of recommended AI-based tools must be developed and released from a medical organization such as the CFPC. Currently, there is a scoping review and inventory that has identified and evaluated published studies that have tested or implemented AI in primary care settings [20,65]. This can be a starting point for such a list of recommended AI-based tools.

Both educators and residents emphasize that a curriculum should focus on how to use AI-based tools (application) along with how to decide when to use and evaluate them (critical appraisal). Residents are already doing this comparatively as seen through their discussions of using ChatGPT, an AI-based chatbot launched by OpenAI that can be used as a digital consultant (eg, simple inquiries about diagnoses and treatment plans). One resident stressed that although they use ChatGPT at times for inquiries related to patient care, they are cautious of the information, as they are aware that ChatGPT can make mistakes and always consult other resources. As ChatGPT rises in prominence, its impact on medical education has been evident through the resident panel discussion and the literature [66,67].

The incorporation of AI content in medical education has already begun with innovative practices, which include case-based learning and flipped classroom models. Case-based learning incorporates real-world AI use cases, where AI is used in clinical practice as examples for physicians [68]. Through this learning approach, students have a better understanding of the technical aspects of AI, as it allows physicians to compare their thought processes with other students and critically reflect or challenge their assumptions and biases of AI and clinical practice [68]. One study assessed the capabilities of ChatGPT within the framework of a preclerkship case-based active learning curriculum. Although the AI chatbot is not comprehensive enough to serve as a textbook, it was shown to answer questions, generate test questions, and appropriately respond to prompts in case-based learning scenarios [69]. According to a scoping review of teaching AI ethics in medical education, 5 publications reported in using case-based learning when understanding ethical challenges [70]. Resident panelists believe that simulation of these tools is beneficial, as it allows residents to enjoy the learning process and realize how these AI-based tools would operate in actual clinical settings. During these simulation sessions or case-based learning approaches, educator panelists highlighted reviewing the capabilities and basic functions of AI-based tools.

Another practical example of incorporating AI content through innovative practices is the flipped classroom model approach. Flipped classroom models can consist of web-based content supplemented by in-person classroom sessions [71], a key observation reinforced by residents of the panel discussion. One study designed and evaluated a novel AI course for medical students using a flipped classroom model, and they found that attending the course can increase self-perceived AI readiness in medical students [71]. In addition, educators have also commented on facilitating AI learning by integrating family medicine AI-based tools in quality improvement projects, which has been emphasized and recommended by other researchers [72].

Family physicians use AI, when implemented, primarily for diagnosis, detection, or surveillance purposes [20]. Although educators have flexibility in choosing from a wide range of AI tools, certain tools have proven to be particularly essential for effective integration. These include AI-enabled chatbots, clinical documentation support, and diagnostic decision support, which have shown to improve physicians’ efficiency and accuracy in their work [73-75]. However, there have been several barriers identified in previous reviews, which have made the adoption of AI-based tools difficult [76-78]. These issues include a lack of trust among educators, students, and clinicians; insufficient training and digital literacy; and resistance to change [77].

Additional challenges include data privacy and patient safety concerns, ethical and legal issues, interoperability issues, lack of funding, and inequities in access to AI tools—particularly between rural and urban settings [79]. In contrast, several strategies and enablers have been identified in order to better facilitate the adoption of AI and its continued use. These strategies include strategies fostering interdisciplinary collaboration between educators, clinicians, and AI developers; providing targeted training programs to build AI literacy; developing high-quality datasets for diverse use cases; and creating supportive regulatory frameworks [77]. Establishing national or local community networks to share resources and best practices, while leveraging trusted relationships within these networks, can also significantly enhance confidence in and adoption of AI-based tools. To identify relevant enablers and barriers to AI adoption of a certain audience, a comprehensive, stakeholder-centered approach is essential. For example, researchers in Canada conducted in-depth interviews with primary health care and digital health stakeholders and were able to ascertain their current barriers and potential facilitators of AI [80].

It is important to note that AI systems exist in diverse contexts and content with distinct implications, risks, and ethical and legal challenges depending on their application and domain. For example, in education, AI-enabled tools using large language models may offer personalized education, but biases may be propagated, inaccurate information may be generated, or students may overrely on AI, undermining their critical thinking skills [63,81]. In addition, there is potential for the exacerbation of inequities in accessing AI tools as well as the misuse of AI-generated content. In comparison, AI-enabled tools in clinical practice, such as decision-support systems, could carry risks of incorrect or biased recommendations that may directly impact patient outcomes [82,83], thus, raising ethical concerns about patient autonomy and safety as well as legal liability in cases of harm. Therefore, the differences of AI in each domain are important to understand in order to identify appropriate safeguards. Future research should conduct comparative analyses of AI’s risks, implications, and ethical and legal dimensions in educational versus clinical settings, examining factors such as accuracy, equity, accountability, and trust. These studies can inform best practices and policies to optimize AI’s potential while mitigating domain-specific risks.

During the development of this curriculum framework, there were several challenges in effectively integrating an AI curriculum framework into a family medicine residency training program. During the expert panel discussions, many experts emphasized the issue regarding the lack of standardization with the definition of AI. Although a definition of AI was chosen for the purpose of the panel, a specific and committed definition of AI within medical education has not been established [84-86]. Panelists argued that an AI definition must be properly explained to avoid confusion or misrepresentation. In relation to family medicine, a recent primer for AI in primary care was published, which provided the definition, “The field of AI is broad and rapidly expanding. The field is centred on how computers might be able to perform humanlike ‘intelligent tasks,’ such as summarizing large amounts of information or making inferences about a situation” [87]. The discussions regarding this framework highlight the necessity of a standardized AI definition for better development of teaching and learning content. This is especially true when specializing in different fields of medical education, including family medicine and primary care.

There is a need to introduce AI education within family medicine; however, the low urgency and priority to integrate this type of education at the moment were noted throughout the discussions. This can be due to the lack of AI-enabled tools for family physicians currently being developed, tested, and implemented in practice [88,89]. Furthermore, some residency programs lack the appropriate AI tools or are in lower-resource settings. As a result of the minimal exposure family physicians have with AI, their motivation to learn about the topic can also be reduced. This reduced priority of AI education competes with the CFPC’s 105 topics of family medicine curricula [4]. This is exacerbated by the fact that Canada is in a unique position, in which the length of residency training is only 2 years. In addition, the rapid advancement of AI introduces an extra layer of complexity. As new AI-based tools emerge and existing ones advance, educators and family physicians must frequently reassess and update their knowledge and skills. For example, the recent introduction of generative AI and generative AI tools such as ChatGPT has gained widespread popularity in medical and academic settings [90]. Thus, it is difficult to maintain a robust framework due to the inevitable rapid changes of AI in health care. Therefore, the eagerness to integrate this type of education within the curriculum should be met with caution to manage the expectations of both educators and learners.

A key observation made throughout the panel discussion was about the AI content and how much should a family physician know about AI. During the discussions, many of the participants voiced support on the application and appraisal of AI-enabled tools. This is especially challenging when residency is only 24 months, and there are no required AI educational programs presented in the Canadian undergraduate medical education system. Therefore, within the learning objectives, in regard to how much a family physician should know about AI remains undetermined. Further research must be conducted to investigate the level of AI education a family physician should be aware of. Overall, the aforementioned challenges must be addressed in order for this curriculum framework to be effectively implemented.

Following the analysis, design, development, implementation, and evaluation model process, researchers may move forward to the implementation and evaluation of the AIFM-ed framework. During the implementation step, an educational program such as a course or workshop can be developed with the main concepts originating from the curriculum framework. The training for family medicine is already packed; thus, the implementation of this framework will depend on several factors including the current use of AI-enabled tools in family medicine training, previous training in AI (eg, the undergraduate foundation of AI), and the capacity of experienced teachers. However, once implemented, certain success indicators will need to be evaluated to understand its impact as well as any areas for improvement. Future studies could explore indicators such as the perceived impact of the framework, degree of implementation, as well as knowledge and skill apprehensions. These indicators can be evaluated through the framework-derived educational training program according to the Kirkpatrick model [61].

This study had several strengths, including the formation of a national, multidisciplinary panel of family medicine educators. This diverse panel facilitated enriching discussions with varied expertise and insights, allowing for a comprehensive understanding of practical implications and current perspectives on AI education in family medicine postgraduate training. Additionally, by involving both educators and residents, the AIFM-ed curriculum framework ensures the representation from key stakeholders involved in the teaching and learning process of AI education. This co-design approach enhanced the relevance and applicability of the AIFM-ed curriculum framework. Regarding the overall development of this framework, a multi-method systematic approach was used, which includes a comprehensive systematic scoping review and multiple expert panel discussions. This approach allowed us to identify and build on existing AI curriculum topics and resources while also creating new ones. Furthermore, this structured and reproducible methodology ensures a robust foundation that can be used by other educators and researchers to develop training programs (eg, courses) following the established framework.

Despite the strengths, this study also had few limitations. First, the study was developed for programs in Canada, which limits its applicability to other countries due to the different medical education structures globally and their current relationships with AI. However, this could be a starting guide for other researchers to adapt it to their own context. Additionally, expert panel diversity was limiting, where the resident panel came from a single institution, which may further limit the generalizability of the framework. Furthermore, as the participants for the panel discussion were not randomized and were purposively recruited, the results may be subject to selection bias.

We co-developed an AIFM-ed framework for family medicine residency training that outlines its curricular purpose, learning objectives, AI curricular topics, delivery methods, and evaluation strategies to be used by medical institutions. The AIFM-ed curriculum framework ultimately aims to enhance the education of future family physicians, equipping them to effectively integrate AI-enabled tools into their practice and patient care. It is hoped that this framework will provide further advocacy, productivity, and gradual change within the area of curriculum development and AI medical education. Overall, medical institutions are encouraged to begin equipping future physicians with the knowledge, skills, and confidence to effectively use AI-enabled tools, as these technologies will continue to grow within the field of health care and family medicine.