Background: Many health professions faculty members lack training on fundamental lesbian, gay, bisexual, transgender, and queer (LGBTQ+) health topics. Faculty development is needed to address knowledge gaps, improve teaching, and prepare students to competently care for the growing LGBTQ+ population.
Objective: We conducted a program evaluation of the massive open online course Teaching LGBTQ+ Health: A Faculty Development Course for Health Professions Educators from the Stanford School of Medicine. Our goal was to understand participant demographics, impact, and ongoing maintenance needs to inform decisions about updating the course.
Methods: We evaluated the course for the period from March 27, 2021, to February 24, 2023, guided by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework. We assessed impact using participation numbers, evidence of learning, and likelihood of practice change. Data included participant demographics, performance on a pre- and postcourse quiz, open-text entries throughout the course, continuing medical education (CME) credits awarded, and CME course evaluations. We analyzed demographics using descriptive statistics and pre- and postcourse quiz scores using a paired 2-tailed t test. We conducted a qualitative thematic analysis of open-text responses to prompts within the course and CME evaluation questions.
Results: Results were reported using the 5 framework domains. Regarding Reach, 1782 learners participated in the course, and 1516 (85.07%) accessed it through a main course website. Of the different types of participants, most were physicians (423/1516, 27.9%) and from outside the sponsoring institution and target audience (1452/1516, 95.78%). Regarding Effectiveness, the median change in test scores for the 38.1% (679/1782) of participants who completed both the pre- and postcourse tests was 3 out of 10 points, or a 30% improvement (P<.001). Themes identified from CME evaluations included LGBTQ+ health as a distinct domain, inclusivity in practices, and teaching LGBTQ+ health strategies. A minority of participants (237/1782, 13.3%) earned CME credits. Regarding Adoption, themes identified among responses to prompts in the course included LGBTQ+ health concepts and instructional strategies. Most participants strongly agreed with numerous positive statements about the course content, presentation, and likelihood of practice change. Regarding Implementation, the course cost US $57,000 to build and was intramurally funded through grants and subsidies. The course faculty spent an estimated 600 hours on the project, and educational technologists spent another 712 hours. Regarding Maintenance, much of the course is evergreen, and ongoing oversight and quality assurance require minimal faculty time. New content will likely include modules on transgender health and gender-affirming care.
Conclusions: Teaching LGBTQ+ Health improved participants’ knowledge of fundamental queer health topics. Overall participation has been modest to date. Most participants indicated an intention to change clinical or teaching practices. Maintenance costs are minimal. The web-based course will continue to be offered, and new content will likely be added.
Lesbian, gay, bisexual, transgender, and queer (LGBTQ+) individuals have unique health care needs and face health disparities that are growing in scale . In a 2022 Gallup poll, 7.1% of the US population identified as something other than heterosexual, which is double the percentage of lesbian, gay, bisexual, and transgender (LGBT) respondents to the same poll in 2012 [ ]. Now, 1 in 5 Generation Z (born in 1997 to 2012) adults identify as LGBT, which is twice the number of Millennials (born in 1981 to 1996) and far outpaces the reported rates in older generations [ ]. If these trends continue, it is likely that >10% of the US population will identify as LGBT within the next several years [ ]. However, studies report a physician workforce that is underprepared to care for this growing cohort of Americans [ ].
Inadequate physician training in LGBTQ+ health is a remnant of the pathologization of queerness in the 1980s and 1990s during the AIDS crisis . Homophobia and moralistic dialogue in society during that time kept many LGBTQ+ patients from disclosing their sexual orientation to their providers, resulting in substantial unmet care needs [ ]. Medical education focused solely on HIV and AIDS at the expense of other LGBTQ+ health topics, such as gender-affirming treatments or medicolegal issues for unmarried couples [ ]. This left generations of physicians untrained in queer health, which meant that their students were then similarly untrained [ ]. This cyclical failure of medical education has had a ripple effect still felt in our medical schools today [ ]. For instance, a study of medical students at 170 US medical schools found that most assessed their training in queer health to be fair or worse [ ]. Another study that queried US medical school deans documented a median of 5 hours of LGBTQ+ health content in a 4-year curriculum, with one-third of schools offering no content during the clinical years [ ]. Similarly, a survey of US residency program directors in emergency medicine found that only 26% of programs teach LGBTQ+ health; on average, 45 minutes were dedicated to the topic in a 3-year residency program [ ]. A study of emergency physicians in Canada found that 97% of participants felt that 2-spirit LGBTQ+ patients deserve the same care as heterosexual patients, and 83% wanted more training [ ].
Medical schools must address this training gap. In 2014, the Association of American Medical Colleges published guidelines for teaching LGBTQ+ health in US medical schools and introduced 30 student competencies that would improve the health of LGBT patients . This resulted in new curricula in some schools [ ]. However, these training opportunities were generally designed by a small minority of faculty members or students who had expertise or advocacy experience in LGBTQ+ health [ - ]. The average medical school faculty member remains untrained in the basics of queer health, such as accurate vocabulary use (eg, terms related to sex, gender, and sexual orientation), social and behavioral determinants of LGBTQ+ health, medical prevention of HIV, transgender health care, and pelvic health in persons assigned female at birth [ ]. Faculty members who lack training in these fundamental domains are underprepared to teach their trainees about the care of LGBTQ+ patients in the clinical setting, perpetuating physician inexperience in these areas. Faculty development is needed for most clinician educators, who are likely underprepared to teach queer health in their daily practice [ ]. Descriptions of curricula for faculty training in LGBTQ+ health remain a gap in the medical literature, and this likely reflects the absence of such training in most schools. Only 1 US medical school has published the details of a faculty development program in sexual and gender minority health to date [ ].
In 2021, the Stanford School of Medicine released a free, open access, web-based, continuing medical education (CME) course called Teaching LGBTQ+ Health: A Faculty Development Course for Health Professions Educators  ( ). It is an introductory-level course aimed at clinician educators seeking to improve their knowledge of LGBTQ+ health and the care of LGBTQ+ patients, with a focus on ways to incorporate the course content in their clinical teaching of trainees ( ). It meets the definition of a massive open online course (MOOC) in that the course is available to anyone who wants to take it without charge or limits on participation. In this study, we conducted a program evaluation of the course to understand its impact and inform upcoming revisions and additions to the curriculum.
Using a constructivist paradigm, we conducted a program evaluation of the web-based course Teaching LGBTQ+ Health using the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) conceptual framework as our guide [, ]. The purpose of our program evaluation was to understand which types of learners engaged in the course (through participant self-identification as physicians, nurses, other health professionals, or trainees), measure its impact on each group through testing and evaluations, identify any course maintenance concerns, and inform the addition of new course material in upcoming revisions of the curriculum. We defined impact as high user engagement, evidence of learning, and likelihood of practice changes, consistent with elements of the model by Kirkpatrick [ , ]; our specific measures were relevant to web-based learning. We selected the RE-AIM framework as it emphasizes key domains that matched the goals of our evaluation. RE-AIM guides users to evaluate and sustain educational programs such as our course by considering contextual factors to improve public health relevance and population health impact [ ]. RE-AIM has been used successfully for the evaluation of MOOCs and other web-based learning courses similar to ours [ , ].
Study Setting and Population
We conducted this study in 2023 at Stanford School of Medicine (Stanford University, Stanford, California, United States) with data provided by the Stanford Medicine Educational Technology department (Stanford Medicine EdTech), which maintains the course on the internet, and the Stanford Center for Continuing Medical Education (Stanford Medicine CME). Stanford Medicine EdTech instructional designers developed and implemented the course on the internet and were the source of most of the study data. Stanford Medicine CME provided course evaluations from participants who claimed CME credits after course completion. Stanford Medicine CME accredited our course to offer Accreditation Council for Continuing Medical Education, American Nurses Credentialing Center, American Academy of Physician Associates, and American Medical Association continuing education credits. We included data on all participants who completed any section of the course regardless of whether they completed the entire course. We excluded participants who registered for but never started the course.
Our investigator team authored, designed, and built Teaching LGBTQ+ Health over a 2-year period from 2019 to 2021. We chose to offer the faculty development course on the web to easily disseminate the content as broadly as possible, and we used closed captioning for all audio elements to increase accessibility. However, we acknowledge that there remain some accessibility challenges inherent to web-based learning platforms. We wrote the course objectives and content based on a needs assessment of novice learners (); therefore, the course is limited to introductory material and does not cover advanced topics such as medical and surgical affirming care for transgender patients. We drafted a storyboard that Stanford Medicine EdTech used to build the course platform, illustrate animated characters, develop video content, and create web-based learning activities. We divided the course into four sections: (1) Orientation, (2) Fundamentals of Teaching LGBTQ+ Health (subsections: Introduction, Pretest, LGBTQ+ Health Vocabulary, Social and Behavioral Determinants of LGBTQ+ Health, and Teaching Strategies; ), (3) Teaching LGBTQ+ Health Cases (including Carla, case of a bisexual woman with a new cancer diagnosis [case 1]; Jesse, case of medical HIV prevention for a serodiscordant couple [case 2]; and Teddy, case of a nonbinary patient seeking affirming pelvic health care [case 3]), and (4) Conclusions, Resources, and CME Credit Instructions ( ). Instructional methods included animated videos, interactive clinical cases, written content, and quizzes. The course was beta tested by an extensive number of Stanford Medicine EdTech staff members and a group of volunteer physicians and medical students outside Stanford; these reviews resulted in the correction of typographical, hyperlink, and caption errors. A second group of experts in LGBTQ+ health also reviewed the course content for accuracy, and no content changes were recommended.
We launched Teaching LGBTQ+ Health on the web on March 27, 2021. It is a free, interactive, and self-paced MOOC and requires approximately 90 minutes to complete. Continuing education credits are offered without charge to those who complete the course. Teaching LGBTQ+ Health is hosted on the Stanford Medicine Med Education website and supported by Stanford Medicine EdTech. The Med Education learning management system (LMS) is built by Stanford on WordPress (WordPress Foundation) using the LearnDash LMS plug-in (Liquid Web Brand). The interactive components of the course were built using H5P (H5P Group, Flow Coworking), the CM Glossary Tooltip WordPress plug-in (CreativeMinds), and Gravity Forms (Rocketgenius, Inc). Elements of the course can be downloaded by users for free and embedded in other sites via HTML. In February 2023, the course became available on Coursera (Coursera, Inc), a global web-based learning platform that hosts college or university courses, certificates, degree programs, and other MOOCs . We did not edit the content of the course for Coursera; both the Coursera- and Stanford-hosted versions of the course are identical.
Data Collection and the RE-AIM Framework
We collected data over several weeks from February 2023 to March 2023. We used quantitative and qualitative data analyses to evaluate the course across the 5 RE-AIM domains ().
Reach refers to participation and demographics. We determined reach based on (1) participation numbers (which included the number of people who registered for the course, participated in any portion of the course, completed the course, or received continuing education credits), (2) demographic data (which included occupation type, area of clinical practice, years in clinical practice, and role or title), and (3) engagement of the target audience (which was defined as Stanford School of Medicine faculty members).
Effectiveness is the impact of the course and its effects, both positive and negative. We judged this domain using (1) the results of a 10-question precourse/postcourse quiz (we designed the quiz to align with the learning objectives of each subsection of the course using best practices for item writing () [ ], beta tested the quiz among the study authors and later a panel of volunteers who completed the course, and iterated it based on feedback; a standard-setting exercise determined a minimum passing standard of 8 out of 10 correct questions, which were required for successful completion of the course and necessary to obtain continuing education credit [ ]; one of the questions purposely did not have a correct answer and was scored as 1 point, so the minimum test score was 1, not zero), (2) a review of CME course evaluations (these included Likert-style responses on a 5-level scale from strongly agree to strongly disagree for statements about the utility of the course content; the effects on the professional growth of the participants; relevance to clinical practice; whether the course had an engaging and interactive format; quality of the content; delivery and effectiveness; value of the topic; overall course rating; and improvement in knowledge, skills, and attitudes; this also included open-text responses to questions about intention to change practice and knowledge and skills learned), (3) sentiment analysis of CME course evaluations and responses to prompts during the 3 clinical cases, and (4) thematic analysis of CME course evaluations.
Adoption refers to the ways in which participants can be involved in the intervention. We assessed adoption by performing a (1) thematic analysis of open-text responses to prompts throughout the course (these were questions about how the participant would teach the course content to trainees) and (2) sentiment analysis of clinical case prompts. We also (3) estimated the representativeness of settings and instructors involved in the course.
Implementation describes the development and execution of the course. We analyzed this domain through (1) review of costs and grant funding, (2) key project milestones, (3) strategies used for dissemination of course content to other health care organizations, and (4) presentations to potential learner groups internal and external to Stanford.
Maintenance refers to the sustainability of the course. For this domain, we evaluated the (1) course platform, (2) continuous quality assurance methods, and (3) plans for the addition of new course content. We also described (4) evidence of institutionalization of the course and related policies in various settings.
We analyzed participation rates, quantitative course feedback, and learner demographics using descriptive statistics.
As the pretest and posttest were identical, we performed a paired 2-tailed t test to compare learners’ scores (percentage of correct test items) before and after the course. Some participants took the pretest or posttest multiple times, and some of those participants took the posttest repeatedly until they achieved a perfect score. Therefore, we averaged individual participants’ multiple test scores to obtain a single pretest and a single posttest score for statistical analysis. Only participants who completed both the pretest and posttest were included in the analysis. All statistical analyses were performed using the tidyverse suite of data analysis tools implemented in the statistical computing language R (version 4.3.0; R Foundation for Statistical Computing) [, ].
We performed a qualitative thematic analysis of open-text responses and learner feedback following the 6 steps outlined by Braun and Clarke . First, our investigator team met to familiarize ourselves with the type and amount of qualitative data collected. Next, 2 investigators (MAG and SZ) independently coded a portion of the data. They then met to discuss the codes, define them, rename them, and resolve disagreements. This generated the initial codebook. MAG and SZ then coded all the data using the codebook and recorded any new codes that were identified. They reviewed these together once more and made adjustments. Then, the full team met to review the codes and examples from the data to construct potential themes. After discussion, a consensus was reached regarding the final themes, which were named and defined.
In addition, open-text responses and learner feedback were subjected to sentiment analysis using a custom natural language processing pipeline. Specifically, text responses were tokenized into individual words and cross-referenced with the National Research Council Word-Emotion Association Lexicon for emotion and polarity annotation [, ]. After annotation, the proportion of words associated with each emotion or polarity was calculated for each participant and plotted. Only text responses with ≥5 words were included in the analysis; all others were considered too few for analysis and omitted.
Finally, a program evaluation should not solely report data (ie, what) but also offer explanations for the data (ie, why). Wherever appropriate, we offer our interpretation of the patterns in the data, potential causes, or implications.
We acknowledge that our personal experiences may have biased our analyses. MAG is a senior faculty member, an expert in medical education, and an emergency physician. TK is a MD, PhD student who has extensive experience in LGBTQ+ advocacy and affinity groups as well as deep content knowledge of queer health topics. SZ is an internal medicine resident who has expertise in LGBTQ+ health education and curriculum design. DB is an instructional designer who created the interactive features of the course, oversees its continued maintenance, directs marketing, liaises with continuing education credit providers, and adapts the course to other LMSs such as Coursera. We initially met to discuss our goals for this program evaluation and acknowledged our biases ahead of data analysis. We coded the data based on what was said, not what was inferred. We discussed our biases during coding and theme identification.
We report the results of our data analyses using the RE-AIM framework.
As of February 24, 2023, a total of 2577 people had registered for the course, of whom 1782 (69.15%) participants engaged with some of the course content. The 30.85% (795/2577) attrition likely reflects some individuals who registered for the course before continuing education credits were made available (many months after the course launch), decided to wait, and never returned. Only 38.05% (679/1782) of the participants completed the course as defined by the achievement of a minimum passing score on the postcourse exam. The other 61.95% (1103/1782) of participants completed some or all of the course modules but chose not to take the postcourse test, perhaps because they were not eligible for or interested in continuing education credits. Only 13.3% (237/1782) of the participants claimed CME credits; however, CME credits are awarded only to physicians, and this low percentage reflects the many nonphysicians who completed the course as well.
There were 2 ways to access and register for the web-based course. One method was from the course landing page, which yielded 1544 registrants. Of these 1544 registrants, 1516 (98.19%) provided their occupation type—physicians (n=423, 27.9%), students (n=327, 21.57%), and health educators (n=121, 7.98%) represented most participants (). In addition, 1323 registrants provided their area of clinical practice, within which the largest categories were students (n=327, 24.72%), others (n=190, 14.36%), and family medicine and community health (n=139, 10.51%). Participants who self-identified within the “Other” category did not choose an area of clinical practice from the options, perhaps because they were nonstudents and non–health care providers; we did not obtain additional demographic information from this cohort and cannot characterize them further. Of the 423 physicians who reported their area of clinical practice, the largest specialties were emergency medicine and trauma (n=99, 23.4%), family medicine and community health (n=86, 20.3%), and internal medicine (n=60, 14.2%). Finally, 1411 registrants provided their number of years in clinical practice, which revealed that most course registrants were either still in training (n=578, 40.96%) or within <5 years of having finished training (n=254, 18%; ).
The other method available for course registration was through the Stanford Medicine EdTech Med Education LMS, which yielded 1333 registrants and different demographic questions. Most of these participants were physicians (473/1333, 35.48%) or students (364/1333, 27.31%). Of these participants, 46.96% (626/1333) identified as “in training” when asked about their years in clinical practice; this cohort likely included resident physicians, fellows, and other “nonstudent” trainees. Another 21.16% (282/1333) of participants identified as being within 5 years of completion of their training.
|Profession||Participants, n (%)|
|Health educator||121 (8.73)|
|Non–health care provider||47 (3.39)|
|Social worker||40 (2.89)|
|Nurse practitioner||34 (2.45)|
|Health care administrator||30 (2.16)|
|Physician assistant||25 (1.8)|
|Other hospital staff||22 (1.59)|
|Years of practice||Participants, n (%)|
|In training||578 (40.96)|
Engagement of the Target Audience
This faculty development course was designed for educators across the health professions, although Stanford faculty members were the target learners. A minority of participants (64/1516, 4.22%) had a Stanford University email address. Gmail addresses were the most commonly used, and it is unknown how many Stanford affiliates used their personal email addresses rather than their Stanford email addresses. Therefore, the number of Stanford participants was likely higher. Of the 2610 physician faculty members of the Stanford School of Medicine, no fewer than 18 (0.69%) completed the course. Similarly, this number resulted from a review of email addresses and represents a minimum. We assumed that more Stanford affiliates would complete the course simply because it was created at our institution. Reasons for low engagement potentially included the lack of incentives, perception that the course material was not relevant, saturation of professional development opportunities offered at our institution, or poor marketing. We did not set a target participation rate, although we expected more learners from Stanford to participate.
Precourse and Postcourse Quiz
A total of 65.43% (1166/1782) of the participants completed the precourse quiz, and 35.63% (635/1782) attempted the postcourse quiz. We suspect that this attrition is related to the desire to obtain continuing education credits as the posttest was a requirement and not all users pursued credits. Alternatively, this attrition may represent users who simply did not complete the course for any reason, such as the lack of interest or time.
None of the participants passed the pretest on their first attempt, and 67% (431/643) of the participants met the minimum passing standard on the posttest on their first attempt. The pretest scores are notable as we designed Teaching LGBTQ+ Health as an introductory-level course; all participants failed the pretest regardless of background and years of training and practice. The median number of postcourse quiz attempts was 1. The median change in test scores was 3 out of 10 points, or a 30% improvement (t678=20.44; P<.001;). Most participants across all major subgroups improved their scores similarly; for instance, students and practicing physicians had nearly the same median change in scores (2.5 for students and 3.0 for physicians). This analysis indicates that the course effectively improved participant knowledge related to our course objectives across all learner subgroups as the course objectives mapped directly to the 9 scored questions on the test.
CME Course Evaluations
We analyzed CME course evaluations from the 13.3% (237/1782) of physician participants who completed the course and claimed CME credit. Most participants strongly agreed with positive statements about the course design and effectiveness (and ). Overall, the course evaluations were outstanding for each of the survey items, as illustrated in the figures.
The strength of the course evaluations was further supported by quantitative sentiment analysis of open-text responses on both the CME evaluation and in response to the course’s case presentations (). Overall, positive sentiments were significantly more prominent than negative sentiments, with statistically significant differences in the CME evaluation (Mann-Whitney U=215; n1=n2=152; P<.001), the “Carla” case presentation (Mann-Whitney U=1225; P<.001), the “Teddy” case presentation (Mann-Whitney U=129; n1=n2=118; P<.001), and the “Jesse” case presentation (Mann-Whitney U=212; n1=n2=130; P<.001) [ ].
Thematic Analysis of CME Course Evaluations
We identified 3 themes from the open-text responses to questions on the CME evaluation (). The questions asked about anticipated changes in practice and new knowledge, skills, or attitudes acquired from the course. These themes included (1) “LGBTQ+ Health as a Distinct Domain” (the acknowledgment by participants that LGBTQ+ health is a unique body of knowledge and skills), (2) “Inclusivity in Practices” (the use of communication techniques [clinical and teaching] and clinic design to ensure LGBTQ+ patients and students feel welcome and respected), and (3) “Teaching LGBTQ+ Health Strategies” (the variety of instructional techniques that can be used to teach this material). These data demonstrate the effectiveness of the course in changing provider perspectives, teaching content, and refining skills.
|Theme||Operational definition||Representative quotes|
|LGBTQ+a health as a distinct domain||LGBTQ+ health is a distinct body of knowledge and related skills that requires intentional training and provider competence.|
|Inclusivity in practices||Routine clinical, nonclinical, and teaching practices must be inclusive of sexual and gender minority groups. Examples include the correct use of vocabulary, the design of inclusive clinical environments, and the practice of affirming care in clinical encounters.|
|Teaching LGBTQ+ health strategies||LGBTQ+ health content can be included in most routine teaching activities throughout health profession schools. Modalities include the addition of new LGBTQ+ health content to courses, role modeling of the 5 P’s of sexual health and correct vocabulary use, and new item writing for tests.|
aLGBTQ+: lesbian, gay, bisexual, transgender, and queer.
Thematic Analysis of Open-Text Responses
We analyzed the responses to prompts in the course that queried how participants would teach the course content to their trainees (). Themes identified were (1) “LGBTQ+ Health Concepts” (participants reported key concepts of queer health that they learned in the course and that would inform their teaching practices) and (2) “Instructional Techniques” (participants identified 2 instructional methods that must be used to teach LGBTQ+ health). These themes broadly reflected changes in participants’ knowledge, skills, and attitudes that resulted from the course. We identified subthemes that represented “LGBTQ+ Health Concepts” that participants planned to teach their students using specific “Instructional Techniques.” The subthemes are listed in with definitions and representative quotes. These data indicated ways in which participants planned to incorporate what they learned into their clinical and teaching practices, an important measure of adoption.
|Theme and subtheme||Operational definition||Representative quotes|
|LGBTQ+a health concepts: unique health needs of queer and gender-diverse patients|
|Social and behavioral determinants of health||Environmental, situational, and behavioral characteristics affect health care access and a wide range of health outcomes. With respect to this course and the care of queer patients, examples include substance use, minority stress, familial rejection, access to health care, sexual practices, and victimization.|
|Medicolegal issues||Unique legal challenges and health disparities experienced by sexual and gender minority groups, whether individually or as couples. Examples include advance directives, surrogate decision maker designations, and visitor policies.|
|Chosen families||Chosen families consist of nonbiological individuals who have deep bonds of support and mutual love. As reflected in our Teaching LGBTQ+ Health cases, chosen families result from engagement in supportive communities or rejection by nuclear families.|
|Sexual health||LGBTQ+ health is more than just HIV and AIDS. It includes a global discussion of sexual practices and behaviors that affect health, among other topics. An example from our course includes the use of the CDCb 5 P’s of sexual behavior history taking.|
|Affirming care||Clinical practices that respect sexual and gender minority groups. Examples include the use of nonjudgmental and inclusive interviewing techniques, correct vocabulary, inclusive clinical environments, and trauma-informed care.|
|Instructional techniques: the maturation of daily, routine teaching activities to include LGBTQ+ health|
|Role modeling||A teaching modality that is historically important in medicine and timeless and requires faculty competence. Key is the demonstration of “how to say” and “how to do” simultaneously to students, generally at the bedside.|
|Student practice||Students need distinct opportunities to practice what they learn in safe spaces, with feedback from trained faculty, and before clinical encounters.|
aLGBTQ+: lesbian, gay, bisexual, transgender, and queer.
bCDC: Centers for Disease Control and Prevention.
Sentiment Analysis of Clinical Case Prompts
Further evidence that participants internalized these core themes during the course can be seen in the sentiment analysis of their free-text responses to each of the 3 cases (). In particular, the “Carla” case presentation (a bisexual woman with a new cancer diagnosis) emphasized many of the more challenging themes of the course, including familial rejection, trauma, and mortality associated with poor access to care ( ). Accordingly, sentiment analysis revealed a statistically significant increase in fear- and sadness-associated words—such as “rejection,” “abuse,” and “trauma”—in the free-text responses to “Carla” compared with the responses to “Jesse” (Mann-Whitney U=14,438; nCarla=163; nJesse=130; P<.001) and “Teddy” (Mann-Whitney U=12,320; nCarla=163; nTeddy=118; P<.001). In contrast, no difference was observed between “Jesse” and “Teddy” (Mann-Whitney U=8431; nTeddy=118; nJesse=130; P<.001). Similarly, the “Carla” and “Teddy” (a nonbinary patient seeking affirming pelvic care) case presentations incorporated themes involving discrimination and unfair treatment in the clinical environment, whereas the “Jesse” case (a gay male patient considering medical HIV prevention strategies) centered on a patient with largely positive experiences with his primary care provider ( ). This difference in context between the cases was reflected in participants’ free-text responses to the “Jesse” case, which contained significantly fewer anger-associated words—such as “discrimination” and “bias”—than both the “Carla” (Mann-Whitney U=12,766; nCarla=163; nJesse=130; P<.001) and “Teddy” (Mann-Whitney U=8688; nTeddy=118; nJesse=130; P<.001) cases. Together, these data suggest that the participants learned the key features of each case presentation.
Representativeness of Settings and Instructors
We did not ask participants about their sexual orientation or gender. Demographic data suggest that the course appealed to both student and faculty audiences despite being marketed as a faculty development program. Future iterations of the course will be designed to appeal more broadly to nonphysician audiences. We partnered with the Medical Student Pride Alliance to recruit volunteers to provide character voices in course videos, beta test the course before launch, and promote the course on the web . Our guiding principle during course development reflects representativeness: “nothing about us, without us” [ ]. The course is well regarded among the LGBT training opportunities at Stanford Medicine.
Costs and Grant Funding
The cost of developing the course was divided into direct charges to support the efforts of Stanford Medicine EdTech designers and programmers and uncompensated efforts by course faculty, volunteers, and administrative staff. The direct charges totaled approximately US $57,000. This was funded by an education innovations research grant from the Stanford Medicine Teaching and Mentoring Academy (US $19,701), a subsidy from Stanford Medicine EdTech (US $15,000), and the Precision Education and Assessment Research Lab in the Stanford Department of Emergency Medicine (US $22,299). The Department of Emergency Medicine provided administrative support. Course faculty (1 medical school faculty member and 2 medical students) collectively spent >600 hours preparing the course. Additional volunteers included voice actors (5 total, 20 total hours worked), expert review of the course (2 work hours), and beta testing by physician volunteers (6 work hours). On the basis of these experiences, the projected direct cost of adding a new 10-minute animated case module to the existing course platform is US $17,000 in 2023.
Key Project Milestones
Over 2 years, we have achieved the following key milestones: grant submission and funding, securing Stanford Medicine EdTech collaboration, needs assessment, delineation of learning goals and course objectives, content and script finalization, storyboarding, character animation, custom visual development, audiovisual editing, beta testing, launch communications and webinars, marketing, continuing education accreditation, distribution to Coursera, and program evaluation. A similar cycle of key milestones can be expected for any additional course content to be developed.
Strategies for Dissemination of the Course
We used social media, CME listserves, and cross-marketing with another Stanford web-based course to publicize our course (). We contacted LGBT health organizations in major US cities and large cities in English-speaking countries, notified LGBT news organizations in the United States, and did direct outreach to medical professional societies. We used a snowball technique in which we asked participants to share the course with their colleagues and someone outside their institution, and we made the course searchable on the internet. Now, the course has also been made available on Coursera, which has substantially increased participation in the several weeks between Coursera launch and the preparation of this manuscript (several hundred new participants in <2 months; not analyzed in this study).
Presentations to Potential Learners
We presented the course to live audiences via Zoom (Zoom Video Communications, Inc) for educational and marketing purposes. The course is animated and interactive and, therefore, lends itself well to live demonstrations of functionality, content, and user experience. It is more visually appealing than many other web-based CME courses, which we hoped would dispel biases about web-based learning. Some of the initial audiences included the National LGBTQ Health Awareness Week; a women’s organization within the US Navy; the Stanford Ethics, Society, and Technology Hub Unconference; a case study presentation on how to build robust web-based courses for the 2021 Stanford Medicine CME Live Conference; a webinar for educators from historically Black colleges and universities in the United States; numerous medical school grand rounds lectures; and internal Stanford Medicine department presentations.
The course continues to be hosted on the Stanford Med Education LMS at a cost of US $1000 per year, subsidized by the Stanford Medicine EdTech department. We have not had to make any adjustments to the platform since the course launch 2 years ago. Regular maintenance and troubleshooting support are supplied as needed by the Stanford Medicine EdTech department.
Continuous Quality Improvement
We closely monitor course feedback to ensure that the LMS is functioning properly and identify any content that needs to be edited. No signals have resulted in a change to the course yet, although a medication recommendation will be modified this year. Course evaluations have remained very positive throughout the 2 years that the course has been on the web, and we believe that the course content remains up-to-date and relevant.
Plans for Additional Content
The course was launched in March 2021, and if successful, the goal was to add new content by 2024. We conducted this program evaluation to determine whether that plan should continue. We have met with numerous stakeholders within the Stanford LGBTQ+ enterprise as a needs assessment for new content within our institution. There is a demand for additional course modules regarding the care of transgender patients, especially adolescents.
A large number of trainees completed this faculty development course. We know of 1 US medical school that requires preclinical students to complete the course, likely explaining this observation. We also noted a very large number of registrants with the same email address from another US medical school, which suggests that the course was likely required for this cohort of participants as well. We are unaware of other mandated audiences or policies related to this course. The Stanford offices that funded the course continue to market it regularly. We anticipate that this study will facilitate future institutionalization and incentivization for completion at our medical school.
We conducted a rigorous program evaluation of the Teaching LGBTQ+ Health course that provided an understanding of its impact to date and informed our decisions about the course moving forward. We assessed impact using many measures of course engagement, evidence of learning, and likelihood of practice change [, ]. Although we found a low participation rate by the target population, we were pleased with the degree of course engagement outside our institution and across disciplines, with excellent participant feedback. Participation beyond the Stanford School of Medicine spoke to our a priori decisions to make the course free, open access, and available for continuing education credits for physicians and nurses. The analysis of our pre- and postcourse quiz and CME course evaluations provided evidence of effective learning. Our thematic analysis identified meaningful ways in which participants intended to change their clinical or teaching practices based on the course content; we hope that such actions ultimately translate into improved care for LGBTQ+ patients. Sentiment analysis confirmed that most participants achieved the learning goals of the interactive clinical cases. Therefore, our summary appraisal is that the course has been impactful, recognizing that action is needed to increase reach. With proper marketing and incentivization of faculty participants, we believe that the course can be successfully implemented to scale at many different health profession schools.
We believe that Teaching LGBTQ+ Health is a unique learning resource for health profession educators that fills an important training gap . It was purposely designed as a faculty development course that would simultaneously provide an introduction to LGBTQ+ health content and methods of teaching that content to trainees. Faculty development programs aimed at improving the teaching of queer health content are rarely described and are primarily found in the nursing literature [ , ]. However, Harvard Medical School recently published a comprehensive sexual and gender minority health curriculum for medical students that included an impressive faculty development plan; notably, they used web-based learning modules somewhat similar to our course [ ]. Most other published LGBTQ+ health curricula or curriculum mapping exercises have been used in undergraduate or graduate medical education programs but not for CME or faculty initiatives [ , , , ]. We believe that faculty training—not just student training—is critical for the normalization of LGBTQ+ health content in the routine teaching activities of our schools. The Stanford and Harvard web-based courses can provide free faculty development on queer health to other health professionals, most of whom practice at medical centers that do not offer such faculty training.
We separately analyzed 2 collections of qualitative data, one sourced from CME course evaluations and the other from open-text responses to prompts throughout the course. These data reflected learned content (Effectiveness domain in RE-AIM) and intentions to change (Adoption domain), and the themes that we identified aligned well with these concepts. Interestingly, the themes discerned from each data set were quite similar, representing 3 broad findings. First, participants recognized LGBTQ+ health as a distinct body of knowledge, a notion that is well established in the literature but may have been new to novice learners in our course [- ]. Of note, only practicing physicians and not trainees complete CME evaluations, so it can be surmised that faculty participants were those who were the most struck by the scope of LGBTQ+ health content. Second, of the instructional methods reviewed in the course, role modeling stood out as particularly important; this is consistent with the historical use of role modeling as a classic bedside teaching technique [ , ]. Faculty would be wise to incorporate role modeling when teaching new trainees, especially with respect to initially challenging communication skills such as obtaining a comprehensive sexual history or counseling patients about end-of-life decisions. Finally, participants intended to be more affirming and inclusive in their future practices, which represented an important change in attitudes and skills that resulted from the course. Affirming care practices are critically important for the treatment of sexual and gender minority groups’ health concerns and have been shown to improve patient care [ , ].
MOOCs are often evaluated using measures of learning, learner engagement, and learners’ experiences interacting with the platform . We examined these measures in a variety of ways using the RE-AIM framework to guide our program evaluation. The RE-AIM domains included these important measures and many others and aligned well with our stated goals for the exercise. Similar to other studies of MOOCs, the use of both quantitative and qualitative data in a RE-AIM evaluation resulted in a robust set of inputs and outputs to examine [ ]. These data are voluminous, which therefore requires this lengthy summary report. We found that RE-AIM was a valuable method to gather the broad data we needed for informed decisions regarding our course. We believe it to be a practical and effective framework that can be useful when conducting a program evaluation of a curriculum of any size.
Program evaluation outcomes may be instrumental (used to make improvements or changes), conceptual (used to evolve an understanding of a program but without changing it), or symbolic (used when an evaluation is required for justification of a change or for reporting purposes) . We conducted this evaluation for instrumental purposes, specifically to answer whether the course should continue to be offered and should be expanded. We confirmed that the course is inexpensive to maintain on the internet; therefore, we will continue to offer it. However, it was very costly to develop. New modules designed to match the current course esthetic will again require significant funding. However, as the course LMS site has already been built, it is much easier and less expensive to develop new content. We will apply for new grant funding for this purpose. Key design and implementation milestones for new content development will mirror those of the initial course, as described previously in the Key Project Milestones subsection of the Results section. Other health profession educators interested in developing similarly interactive and animated web-based courses should be aware of the costs involved.
There were several important study limitations. Registration and participation numbers may be misinterpreted as the primary measures of a successful course (ie, an assumption that more participation means more impact). Although reach is very important for the program evaluation of an MOOC, it does not assess course quality and, therefore, would limit our understanding of impact if considered alone. The additional domains of RE-AIM offered a richer understanding of impact in this study. However, we acknowledge that course participation may be overemphasized by our stakeholders and readers. Only a subset of the participants completed the pre- and postcourse quiz or provided course feedback; we must assume that participants who did not fully engage in the course were less affected by it. The length of the quiz—only 10 questions—may not have been discriminating enough to fully appreciate the degree of participant learning, although our data were statistically significant (t678=20.44; P<.001) and implied its effectiveness. In addition, we do not have longitudinal data from participants about changes in their practice habits; we only have intention-to-change data. Finally, the evaluation of MOOCs is subject to biases that result from potentially large and diverse groups of learners; these biases are somewhat mitigated by the use of pre- and posttests, as in our study . However, we did not control for other confounding variables related to these biases.
In summary, our evaluation of Teaching LGBTQ+ Health suggests that it was an expensive and time-consuming course to create, was impactful, met its learning objectives for those who completed the course, missed its target audience but had broad appeal, and requires very little ongoing maintenance. On the basis of this evaluation, the course will continue to be offered by Stanford Medicine EdTech and Coursera, and we plan to include additional content if appropriate funding is identified. Our goal is to use the web-based platform as a flagship for a suite of LGBTQ+ health curricula; this program evaluation was viewed as foundational to such an initiative.
Teaching LGBTQ+ Health improved participants’ knowledge of fundamental queer health topics. Overall participation has been modest to date. Most participants indicated an intention to change their clinical or teaching practices. Maintenance costs are minimal, and the course will continue to be offered on the web for free. New content is likely to be added.
We give special thanks to Erin Bennett, Mollie Marr, Sabina Spigner, Austen Ott, Baffour Kyerematen, Shanna Polley, William Bottini, Andrew Baek, Katherine Cao, Huy Tran, Lauren Watley, and Jessica Whittemore for their contributions to the course content and production.
The codebook for the demographic and quantitative analysis is located on the following GitHub repository . The qualitative data sets analyzed in this study are available from the corresponding author upon reasonable request.
Conflicts of Interest
Teaching LGBTQ+ Health course logo. LGBTQ+: lesbian, gay, bisexual, transgender, and queer.PNG File , 132 KB
Teaching LGBTQ+ Health course trailer. LGBTQ+: lesbian, gay, bisexual, transgender, and queer.MP4 File (MP4 Video), 140791 KB
Learning objectives.DOCX File , 18 KB
Didactic clip: sexually transmitted infections.MOV File , 65445 KB
Curriculum summary.DOCX File , 20 KB
Data sources for the Reach, Effectiveness, Adoption, Implementation, and Maintenance evaluation of the Teaching LGBTQ+ Health course. LGBTQ+: lesbian, gay, bisexual, transgender, and queer.PNG File , 114 KB
Pre- and postcourse test questions and answers.PDF File (Adobe PDF File), 119 KB
Carla case clip.MP4 File (MP4 Video), 62425 KB
Didactic clip: pre-exposure prophylaxis.MOV File , 30150 KB
Teaching LGBTQ+ Health press release. LGBTQ+: lesbian, gay, bisexual, transgender, and queer.PDF File (Adobe PDF File), 1089 KB
- Yeung H, Luk KM, Chen SC, Ginsberg BA, Katz KA. Dermatologic care for lesbian, gay, bisexual, and transgender persons: terminology, demographics, health disparities, and approaches to care. J Am Acad Dermatol 2019 Mar;80(3):581-589 [https://europepmc.org/abstract/MED/30744874] [CrossRef] [Medline]
- Jones J. LGBT Identification in U.S. Ticks Up to 7.1%. Gallup. 2022 Feb 17. URL: https://news.gallup.com/poll/389792/lgbt-identification-ticks-up.aspx [accessed 2023-02-27]
- Ruedas NG, Wall T, Wainwright C. Combating LGBTQ+ health disparities by instituting a family medicine curriculum. Int J Psychiatry Med 2021 Sep;56(5):364-373 [CrossRef] [Medline]
- Florêncio J. AIDS: homophobic and moralistic images of 1980s still haunt our view of HIV – that must change. The Conversation. 2018 Nov 27. URL: https://theconversation.com/aids-homophobic-and-moralistic-images-of-1980s-still-haunt-our-view-of-hiv-that-must-change-106580 [accessed 2023-02-27]
- Gisondi MA, Bigham B. LGBTQ + health: a failure of medical education. CJEM 2021 Sep;23(5):577-578 [CrossRef] [Medline]
- Johnson N. LGBTQ health education: where are we now? Acad Med 2017 Apr;92(4):432 [CrossRef] [Medline]
- White W, Brenman S, Paradis E, Goldsmith ES, Lunn MR, Obedin-Maliver J, et al. Lesbian, gay, bisexual, and transgender patient care: medical students' preparedness and comfort. Teach Learn Med 2015;27(3):254-263 [CrossRef] [Medline]
- Obedin-Maliver J, Goldsmith ES, Stewart L, White W, Tran E, Brenman S, et al. Lesbian, gay, bisexual, and transgender-related content in undergraduate medical education. JAMA 2011 Sep 07;306(9):971-977 [CrossRef] [Medline]
- Moll J, Krieger P, Moreno-Walton L, Lee B, Slaven E, James T, et al. The prevalence of lesbian, gay, bisexual, and transgender health education and training in emergency medicine residency programs: what do we know? Acad Emerg Med 2014 May;21(5):608-611 [http://hdl.handle.net/2027.42/106813] [CrossRef] [Medline]
- Lien K, Vujcic B, Ng V. Attitudes, behaviour, and comfort of Canadian emergency medicine residents and physicians in caring for 2SLGBTQI+ patients. CJEM 2021 Sep;23(5):617-625 [CrossRef] [Medline]
- Implementing curricular and institutional climate changes to improve health care for individuals who are LGBT, gender nonconforming, or born with DSD: a resource for medical educators. Association of American Medical Colleges. 2014. URL: https://tinyurl.com/raetsxvd [accessed 2023-02-27]
- Pregnall AM, Churchwell AL, Ehrenfeld JM. A call for LGBTQ content in graduate medical education program requirements. Acad Med 2021 Jun 01;96(6):828-835 [CrossRef] [Medline]
- Agapoff 4th JR. The LGBTQ psychiatrist educator. Clin Teach 2021 Oct;18(5):472-473 [CrossRef] [Medline]
- Braun HM, Ramirez D, Zahner GJ, Gillis-Buck EM, Sheriff H, Ferrone M. The LGBTQI health forum: an innovative interprofessional initiative to support curriculum reform. Med Educ Online 2017;22(1):1306419 [https://europepmc.org/abstract/MED/28399716] [CrossRef] [Medline]
- Cohen RD. Medical students push for more LGBT health training to address disparities. National Public Radio. 2019 Jan 20. URL: https://tinyurl.com/bddbcsvn [accessed 2023-03-31]
- Gentile D, Boselli D, MacNeill E. Clinician's experience and self-perceived knowledge and attitudes toward LGBTQ + health topics. Teach Learn Med 2021 Jun;33(3):292-303 [CrossRef] [Medline]
- Keuroghlian AS, Charlton BM, Katz-Wise SL, Williams K, Jarvie EJ, Phillips R, et al. Harvard medical school's sexual and gender minority health equity initiative: curricular and climate innovations in undergraduate medical education. Acad Med 2022 Dec 01;97(12):1786-1793 [https://europepmc.org/abstract/MED/35947484] [CrossRef] [Medline]
- Gisondi M, Keyes T, Zucker S, Bumgardner D. Teaching LGBTQ+ health: a faculty development course for health professions educators. Stanford Medicine. 2021. URL: https://mededucation.stanford.edu/courses/teaching-lgbtq-health/ [accessed 2023-02-27]
- Varpio L, MacLeod A. Philosophy of science series: harnessing the multidisciplinary edge effect by exploring paradigms, ontologies, epistemologies, axiologies, and methodologies. Acad Med 2020 May;95(5):686-689 [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327 [CrossRef] [Medline]
- Kirkpatrick D. Great ideas revisited. Techniques for evaluating training programs. Revisiting Kirkpatrick’s four-level model. Train Dev 1996 Jan;50(1):54-59 [https://eric.ed.gov/?id=EJ515660]
- Measure online training effectiveness: a hands-on guide to metrics and models. F Learning Studio. 2021. URL: https://flearningstudio.com/evaluate-learning-outcomes-online-courses/ [accessed 2023-06-03]
- WELCOME to re-aim and PRISM: implementation in context. RE-AIM. URL: https://re-aim.org/ [accessed 2023-01-05]
- Smith-Lickess SK, Woodhead T, Burhouse A, Vasilakis C. Study design and protocol for a comprehensive evaluation of a UK massive open online course (MOOC) on quality improvement in healthcare. BMJ Open 2019 Dec 23;9(12):e031973 [https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=31874877] [CrossRef] [Medline]
- Yilmaz Y, Sarikaya O, Senol Y, Baykan Z, Karaca O, Demiral Yilmaz N, et al. RE-AIMing COVID-19 online learning for medical students: a massive open online course evaluation. BMC Med Educ 2021 May 27;21(1):303 [https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-021-02751-3] [CrossRef] [Medline]
- Gisondi M. Teaching LGBTQ+ Health. Coursera. URL: https://www.coursera.org/learn/teaching-lgbtq-health [accessed 2023-03-17]
- Rudolph MJ, Daugherty KK, Ray ME, Shuford VP, Lebovitz L, DiVall MV. Best practices related to examination item construction and post-hoc review. Am J Pharm Educ 2019 Sep;83(7):7204 [https://europepmc.org/abstract/MED/31619832] [CrossRef] [Medline]
- Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med 2006;18(1):50-57 [CrossRef] [Medline]
- R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. 2021. URL: https://www.R-project.org/ [accessed 2023-07-04]
- Wickham H, Averick M, Bryan J, Chang W, McGowan LD, François R, et al. Welcome to the tidyverse. J Open Source Softw 2019 Nov;4(43):1686 [https://joss.theoj.org/papers/10.21105/joss.01686] [CrossRef]
- Braun V, Clarke V. Thematic analysis. In: Cooper H, Camic PM, Long DL, Panter AT, Rindskopf D, Sher KJ, editors. APA Handbook of Research Methods in Psychology, Research Designs. Volume 2. Washington, DC, USA: American Psychological Association; 2012:57-71
- Silge J, Robinson D. tidytext: text mining and analysis using tidy data principles in R. J Open Source Softw 2016 Jul;1(3):37 [https://joss.theoj.org/papers/10.21105/joss.00037] [CrossRef]
- Mohammad SM, Turney PD. Crowdsourcing a word–emotion association lexicon. Comput Intell 2013;29(3):436-465 [https://onlinelibrary.wiley.com/doi/10.1111/j.1467-8640.2012.00460.x] [CrossRef]
- Medical student pride alliance homepage. Medical Student Pride Alliance. 2023. URL: https://www.medpride.org/ [accessed 2023-05-03]
- Charlton JI. Nothing About Us Without Us: Disability Oppression and Empowerment. Oakland, CA, USA: University of California Press; 2000.
- Sherman AD, Cimino AN, Clark KD, Smith K, Klepper M, Bower KM. LGBTQ+ health education for nurses: an innovative approach to improving nursing curricula. Nurse Educ Today 2021 Feb;97:104698 [https://linkinghub.elsevier.com/retrieve/pii/S0260-6917(20)31548-3] [CrossRef] [Medline]
- Ji P, Haehnel AA, Muñoz DN, Sodolka J. The effectiveness of using new instructors to teach an LGBT ally development course. J Prev Interv Community 2013;41(4):267-278 [CrossRef] [Medline]
- Raygani S, Mangosing D, Clark KD, Luong S, Flentje A, Sarah G. Integrating LGBTQ+ health into medical education. Clin Teach 2022 Apr;19(2):166-171 [https://escholarship.org/uc/item/22z2244h] [CrossRef] [Medline]
- Roth LT, Friedman S, Gordon R, Catallozzi M. Rainbows and "ready for residency": integrating LGBTQ health into medical education. MedEdPORTAL 2020 Nov 04;16:11013 [https://europepmc.org/abstract/MED/33204837] [CrossRef] [Medline]
- Keuroghlian AS, Ard KL, Makadon HJ. Advancing health equity for lesbian, gay, bisexual and transgender (LGBT) people through sexual health education and LGBT-affirming health care environments. Sex Health 2017 Feb;14(1):119-122 [CrossRef] [Medline]
- Hsieh N, Shuster SM. Health and health care of sexual and gender minorities. J Health Soc Behav 2021 Sep;62(3):318-333 [CrossRef] [Medline]
- McNamara MC, Ng H. Best practices in LGBT care: a guide for primary care physicians. Cleve Clin J Med 2016 Jul;83(7):531-541 [https://www.ccjm.org/cgi/pmidlookup?view=long&pmid=27399866] [CrossRef] [Medline]
- Mohammadi E, Shahsavari H, Mirzazadeh A, Sohrabpour AA, Mortaz Hejri S. Improving role modeling in clinical teachers: a narrative literature review. J Adv Med Educ Prof 2020 Jan;8(1):1-9 [https://europepmc.org/abstract/MED/32039267] [CrossRef] [Medline]
- Obadia SJ. Role modeling in the first 2 years of medical school. J Am Osteopath Assoc 2015 Aug;115(8):510-512 [https://www.degruyter.com/document/doi/10.7556/jaoa.2015.105] [CrossRef] [Medline]
- Green R, Eckstrand KL, Faeder M, Tilstra S, Ufomata E. Affirming care for transgender patients. Med Clin North Am 2023 Mar;107(2):371-384 [CrossRef] [Medline]
- de Vries E, Kathard H, Müller A. Debate: why should gender-affirming health care be included in health science curricula? BMC Med Educ 2020 Feb 14;20(1):51 [https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-020-1963-6] [CrossRef] [Medline]
- Alturkistani A, Lam C, Foley K, Stenfors T, Blum ER, Van Velthoven MH, et al. Massive open online course evaluation methods: systematic review. J Med Internet Res 2020 Apr 27;22(4):e13851 [https://www.jmir.org/2020/4/e13851/] [CrossRef] [Medline]
- Moreau KA, Eady K. Program evaluation use in graduate medical education. J Grad Med Educ 2023 Feb;15(1):15-18 [https://europepmc.org/abstract/MED/36817537] [CrossRef] [Medline]
- Case sentiment analysis. GitHub. URL: https://tinyurl.com/rd7anp5d [accessed 2023-07-06]
|CME: continuing medical education|
|LGBT: lesbian, gay, bisexual, and transgender|
|LGBTQ+: lesbian, gay, bisexual, transgender, and queer|
|LMS: learning management system|
|MOOC: massive open online course|
|RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance|
|Stanford Medicine CME: Stanford Center for Continuing Medical Education|
|Stanford Medicine EdTech: Stanford Medicine Educational Technology department|
Edited by G Eysenbach, T de Azevedo Cardoso; submitted 31.03.23; peer-reviewed by R Gordon, DS Mulkalwar, D Mangosing, V Podder; comments to author 28.04.23; revised version received 07.05.23; accepted 12.06.23; published 21.07.23Copyright
©Michael Albert Gisondi, Timothy Keyes, Shana Zucker, Deila Bumgardner. Originally published in JMIR Medical Education (https://mededu.jmir.org), 21.07.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on https://mededu.jmir.org/, as well as this copyright and license information must be included.