A comprehensive list of accredited integrated vascular surgery, thoracic surgery, and interventional radiology residencies was obtained from the Accreditation Council for Graduate Medical Education (ACGME) database. Programs participating in the 2020 National Resident Matching Program (NRMP) were eligible for study inclusion. Following identification of all programs with websites, programs were accessed and evaluated by two independent reviewers (one medical student and one resident) for availability of recruitment and educational content items. The websites were viewed independently by each reviewer. The program search and review was performed in November 2019.

Are there key differences in the three specialty program websites for integrated residencies that could potentially impair recruitment efforts in the virtual environment?

Accessibility of websites was determined by surveying the ACGME database for the total number of programs listed and the presence or absence of website links. Links, if they were provided, were characterized as either functional or nonfunctional. Functional links led to a website. Nonfunctional links led to an error page. Functional links were then evaluated as being either direct (landing directly on the program webpage) or indirect (landing on a different page such as the departmental website, requiring further action by the reviewer to access the specific program webpage if possible).

Websites for integrated vascular surgery, thoracic surgery, and interventional radiology residency programs were analyzed for availability of information used to inform and recruit prospective applicants. Content items on recruitment and education (listed in Textbox 1) were selected based on ACGME program requirements as well as previously published literature reviewing the online content of residency and fellowship programs [5,14,19,20]. Content on the training program websites was counted as present if it was present on the main training program webpage or it was accessible via a direct link provided on the main training program webpage.

Websites were evaluated for content relevant to program recruitment and education. Program recruitment information included faculty listings, faculty and departmental research interests, alumni career placements, and information on current residents. Recruitment information regarding the application and interview process as well as general resident quality of life metrics were also evaluated (see Textbox 1). Program education content addressed operative and didactic training. It also covered resident research opportunities. Overall, 41 program recruitment and 16 program education content items were evaluated.

Each website was accessed and evaluated by two reviewers (one medical student and one resident) for availability of content items as well as quality of websites (determined as a function of four dimensions: content, design, organization, and user friendliness). Each reviewer was trained by examining an optimal website, an average website, and a below average website with the senior author. Disputed assessments were resolved by consensus following discussion with the senior author. Reviewers were not blinded.

Overall, there was considerable interrater reliability with 81% agreement (κ=0.74).

Intergroup analysis of continuous variables was performed using ANOVA. Categorical variables were compared using chi-square analysis. Statistical significance was defined as P<.05. Percent agreement and kappa statistics were calculated for interrater reliability. Statistical analysis was performed using statistical software R version 4.0.0 (R Foundation for Statistical Computing).

All data reviewed was open to the public, and there was no contact with fellowship staff; thus, no institutional review board review, ethics approval, or informed consent was necessary.

Of the programs included in this analysis, 87% (53/61) of the vascular surgery, 89% (24/27) of the thoracic surgery, and 95% (81/85) of the interventional radiology programs provided a link to their program webpage on the ACGME webpage (P=.18). Of those programs that provided links, the majority of the links were functional with no difference between the specialties (P=.24). However, few links landed directly on the program webpage. Less than one-third of the programs with functional links provided links that landed directly on the program webpage (P=.52). Overall, 56 of 61 (92%) vascular surgery, 27 of 27 (100%) thoracic surgery, and 74 of 85 (87%) interventional radiology programs had a dedicated webpage (Table 1).

Content was assessed in two domains: recruitment and education. Of the 57 recruitment and educational content items included in this analysis, vascular surgery program webpages contained a median of 26 (IQR 20-32) content items, thoracic surgery program webpages contained a median of 27 (IQR 21-32) content items, and interventional radiology program webpages contained a median of 23 (IQR 18-27) content items. Of the 41 recruitment content items included in this analysis, vascular surgery program webpages contained a median of 19.5 (IQR 15-24) content items, thoracic surgery program webpages contained a median of 20 (IQR 16-24) content items, and interventional radiology program webpages contained a median of 18 (IQR 15-21) content items. Of the 16 education content items included in this analysis, vascular surgery program webpages contained a median of 7 (IQR 4-9) content items, thoracic surgery program webpages contained a median of 6 (IQR 4-7) content items, and interventional radiology program webpages contained a median of 4 (IQR 3-7) content items (Table 2).

For program recruitment, almost all programs provided information on program description, faculty listing, faculty education, administrator or coordinator contact information, facility description, descriptive faculty profiles, and the number of incoming positions. The majority of programs provided information on application requirements, a functional link to the Electronic Residency Application Service (ERAS), benefits, salary, current residents, city information, resident education history, domestic considerations, vacation policy, and interview dates. Less than one-half of programs provided information on program director contact information, faculty publications, well-being strategies, faculty contact information, program director message, alumni listing, alumni career placement, educational fund, parking, nonnational visa information, meal allowance, call requirement, alumni career placement, and debt management. Fewer than one-quarter of the programs provided information on sample contracts, resident profiles, interview details, work hours, and alumni education history. Almost no programs provided information on their selection process, resident contact information, program director message, board examination performance, and alumni contact information (Table 2).

For program education, almost all programs provided information on rotation schedule (84%; 47/56). The majority of programs provided information on didactic instruction, research requirements, journal club, and vascular lab training (Registered Physician in Vascular Interpretation [RPVI]). Less than one-half of programs provided information on departmental research interests, operative experience, meetings attended, elective rotations, conference schedule, curriculum, and simulation training. Fewer than one-quarter of the programs provided information on evaluation criteria and faculty mentorship. Almost no programs provided information on national organizational links and cardiovascular product company links (Table 2).

For program recruitment, almost all programs provided information on program description, faculty listing, faculty education, administrator or coordinator contact information, facility description, and application requirements. The majority of programs provided information on descriptive faculty profiles, a functional link to ERAS, benefits, the number of incoming positions available, salary, current residents, faculty publications, well-being strategy, city information, resident education history, vacation policy, call requirements, and domestic considerations. Less than one-half of programs provided information on program director contact information, interview date, faculty contact information, sample contracts, resident profiles, alumni listings, debt management, educational fund, parking, nonnational visa information, meal allowance, and work hours. Fewer than one-quarter of the programs provided information on program director message, alumni career placement, interview details, and resident contact information. Almost no programs provided information on selection process, alumni education history, program chair message, board examination performance, and alumni contact information (Table 2).

For program education, the majority of programs provided information on rotation schedule, departmental research interests, didactic instruction, and research requirements. Less than one-half of programs provided information on journal club, meetings attended, elective rotation, curriculum, national organization links, and simulation training. Fewer than one-quarter of the programs provided information on operative experiences, conference schedule, faculty mentorship, and evaluation criteria (Table 2). Almost no programs provided information on vascular lab training (RPVI), which should be expected as it is not a core component of thoracic surgery.

For program recruitment, almost all programs provided information on program description, faculty listing, faculty education, administrator or coordinator contact information, and descriptive faculty profiles. The majority of programs provided information on application requirements, a functional link to ERAS, facility description, vacation policy, benefits, the number of incoming positions available, salary, current residents, program director contact information, and faculty publications. Less than one-half of programs provided information on well-being strategies, city information, educational fund, parking, nonnational visa information, meal allowance, call requirement, resident education history, domestic considerations, interview dates, faculty contact information, and sample contracts. -quarter of the programs provided information on a program director message, resident profiles, interview details, alumni listing, alumni career placement, selection process, and debt management. Almost no programs provided information on resident contact information, work hours, alumni education history, program chair message, board examination performance, and alumni contact information (Table 2).

For program education, the majority of programs provided information on rotation schedule, departmental research interests (49/74, 66%), didactic instruction, and research requirements. Less than one-half of programs provided information on operative experiences, journal club, meetings attended, and elective rotations. Fewer than one-quarter of the programs provided information on conference schedule, curriculum, faculty mentorship, vascular lab training (RPVI), and national organization links. Almost no programs provided information on evaluation criteria, simulation training, and cardiovascular product company links (Table 2).

Vascular surgery webpages provided the most information on rotation schedule, journal club, and vascular lab (as compared to thoracic surgery and interventional radiology webpages (P=.02, P=.008, and P<.001, respectively). Vascular surgery webpages provided less information on departmental research interests as compared to thoracic surgery and interventional radiology webpages (P=.008; Table 2).

Thoracic surgery webpages provided the most information on call requirement, national organization link, and work hours as compared to vascular surgery and interventional radiology webpages (P=.02, P=.04, and P<.001, respectively). Thoracic surgery webpages provided less information on descriptive faculty profile as compared to vascular surgery and interventional radiology webpages (P=.02; Table 2).

Interventional radiology webpages provided the most information on operative experience and program director contact information as compared to vascular and thoracic surgery webpages (P=.008 and P=.03, respectively). Interventional radiology webpages provided less information on facility description, debt management (P=.007), research requirements (P<.001), curriculum (P=.04), and simulation training (P<.001) as compared to vascular surgery and thoracic surgery webpages (Table 2).

On an overall assessment, integrated vascular surgery, thoracic surgery, and interventional radiology websites were found to be comparable. The average vascular surgery website score was 2.66 (SD 0.95), the average thoracic surgery website score was 2.18 (SD 0.92), and the average interventional radiology website score was 2.25 (SD 0.88). The vascular surgery websites had the highest scores in content, design, organization, and user-friendliness. The thoracic surgery websites had the lowest scores in content, organization, and user-friendliness, while the interventional radiology websites had the lowest score in design. Additional details regarding website quality, broken down by category, are visible in Table 3.

As resident recruitment moves to a virtual platform, the internet is an increasingly important resource for residency applicants as they research programs. Thoracic surgery program webpages had the most information, followed by vascular surgery program webpages, with interventional radiology program webpages reporting the least content. This trend in availability of content items mirrors the percent of positions filled by each specialty, with 100% of PGY-1 thoracic surgery positions filled, 97% of vascular surgery PGY-1 positions filled, and 97% of PGY-1 interventional radiology positions filled (with 94% of PGY-2 interventional radiology positions filled), as reported by the NRMP 2020 Main Residency Match Results and Data report [28].

Other factors, beyond program websites, that have been identified to influence applicant interest in a program include geography, advice from a mentor, advice from a peer, and other online information. The integrated vascular track was first accredited by the ACGME in 2006 [27], the first integrated thoracic surgery program accepted residents in 2007 [29], and the first integrated interventional radiology programs participated in the NRMP in 2016 [30]. The majority of these integrated programs have been established for less than 10 years. This increase in the number of integrated programs, though necessary to meet the high demand for integrated residency positions, means that many programs do not have an established national presence. Applicants cannot receive the same quality of advice from mentors and peers on newer programs, as compared to programs that have been established for longer periods of time. Furthermore, many programs are geographically clustered, specifically in the northeast and along the west coast (see Figure 1). These factors combine to place additional weight on program websites, perhaps serving as the initial source of information for potential applicants and allowing for comparison.

Of ACGME-accredited programs, 56 of 61 (92%) vascular surgery programs, 27 of 27 (100%) thoracic surgery programs, and 74 of 85 (87%) interventional radiology programs had functional websites. Thoracic surgery program webpages had the most information (content item median 27, IQR 21-32), then vascular surgery program webpages (content item median 26, IQR 20-32), with interventional radiology program webpages reporting the least content (content item median 23, IQR 18-27). The greater amount of content on vascular surgery and thoracic surgery program webpages could be expected, given the young age of many interventional radiology programs. Previous studies have acknowledged integrated interventional radiology program webpages to be a work in progress [15]. This study confirms that finding in relation to longer-established vascular surgery and thoracic surgery program webpages.

Two content items that have been identified to be highly influential to the applicant program decision are operative experience and faculty mentorship [31-33]. This analysis found those items to be reported at 32% (18/56) and 11% (6/56) for vascular surgery, 19% (5/27) and 11% (3/27) for thoracic surgery, and 50% (37/74) and 15% (11/74) for interventional radiology programs (P=.008 and P=.75), respectively. Additional notable deficits for vascular surgery websites were resident profiles, sample contracts, and departmental research interests. Thoracic surgery websites lacked program director contact information and message as well as information on operative experience. Interventional radiology websites had deficits in work hours, debt management, and curriculum. All specialty websites had deficits in evaluation criteria and faculty mentorship. In addition to addressing the deficits in program recruitment and education content items, the deficits in lifestyle management cannot be disregarded; medical students increasingly report controllable lifestyle as a major factor in specialty choice [34,35].

The deficits identified by this analysis are comparable to deficits identified for other specialties. Other studies have found considerable deficits in newsletter, resident listings and photographs, faculty contact information, and away elective rotation information for dermatology websites [3]; resident call schedule, alumni career placement, and salary for orthopedic surgery websites [7]; academic conference schedule, call schedule, operative case listing, graduate fellowship information, and board exam performance for plastic surgery websites [19]; evaluation criteria, call schedule, operative exposure, national meetings attended, debt management, alumni contact, and work hours for neurosurgery websites [20]; call schedule, away elective rotation information, resident profiles, and faculty research for general surgery websites [21]; and call schedule, active/past research projects, area information, message from the program director or chair, selection criteria, salary, and surgical statistics for otolaryngology websites [22].

Overall, we recommend that programs address the deficits in specific content items identified by this analysis. Given the increasingly important role of online information in the residency application process and the anticipated transition to a virtual application process for the 2021 cycle, it would behoove programs to increase their online presence. In addition to the content items included in this analysis, it might be fitting for programs to include more personal information (ie, more detailed resident and attending profiles) to give applicants a better idea of the personality of different programs, replacing the role previously served by in-person away rotations and interviews.

This study had several limitations. First, this data is representative of what information was available online at the time of data collection. It is possible that websites could have been edited or new program websites could have been published since that time. Additionally, though an extensive list of content items were evaluated by reviewers regarding program recruitment and education, it is possible that other unmentioned content items could hold bearing on an applicant’s decision. Finally, reviewers were not blinded to what program they were evaluating. Thus, any inherent bias reviewers might have had for particular programs was not controlled for. The nature of this study did not lend itself to evaluating the association between website content, to what specialty and to what programs applicants apply, and ultimate applicant program placement. Future studies could seek to characterize this trajectory.

This study has uncovered key differences in the availability of online content for residency programs recruiting from the same pool of applicants. Thoracic surgery program webpages have the most information, then vascular surgery program webpages, with interventional radiology program webpages reporting the least content. Recruitment is highly competitive between vascular surgery, thoracic surgery, and interventional radiology with the ratio of applicants to positions being highest for thoracic surgery, then interventional radiology, and lastly vascular surgery, as reported by ERAS. To attract valuable candidates, programs should aim to increase the availability of online content for potential applicants.