Critical Comparison of the Quality and Content of Integrated Vascular Surgery, Thoracic Surgery and Interventional Radiology Residency Training Program Websites

Objective: With the move to virtual interviewing, residency websites are an important recruitment resource, introducing applicants to programs across the country and allowing for comparison. Recruitment is highly competitive from a common potential pool between vascular surgery, thoracic surgery and interventional radiology with the ratio of applicants to positions being highest in interventional radiology, followed by thoracic surgery and lastly vascular surgery, as reported by the National Resident Matching Program. The aim of this study is to evaluate the accessibility and availability of online content for those integrated residency programs. Methods: A list of accredited vascular surgery, thoracic surgery, and interventional radiology residencies was obtained from the ACGME. Program websites were evaluated by trained independent reviewers (n=2) for content items pertaining to program recruitment and education (scored absent or present). Statistical analysis was performed in R software. Results: Of ACGME accredited programs, 56 of 61 (92%) vascular surgery, 27 of 27 (100%) thoracic surgery, and 74 of 85 (87%) interventional radiology programs had functional websites (P=0.122). Vascular surgery websites contained a median of 26 content items (IQR: 20, 32), thoracic surgery websites contained a median of 27 content items (IQR: 21, 32), and interventional radiology websites contained a median of 23 content items (IQR: 18, 27). Two content items considered highly influential to applicant program decision are procedural experience and faculty mentorship, were reported at 32% and 11% for vascular surgery, 19% and 11% for thoracic surgery, and 50% and 15% for interventional radiology (P=0.008 and P=0.751, respectively). Key deficits were work hours, debt management and curriculum for interventional radiology; resident profiles, sample contracts, and research interests in vascular surgery; operative experiences, program director contact and message for thoracic surgery. Interventional radiology deficits were work hours and thoracic surgery deficits were procedural experience. Both IR and CT websites lacked information in evaluation criteria and faculty mentorship. Conclusions: This study has uncovered key differences in availability of online content for residencies recruiting from the same pool of applicants. Thoracic surgery has the most information, followed by vascular surgery, with interventional radiology reporting the least content. In the era of virtual interviewing from the same potential pool of applicants, programs should review and revise their web presence with the aim to increase the availability of online content in order to attract valuable candidates.


Introduction
The role of vascular surgeons in the medical environment has changed considerably with the increasing use of endovascular approaches for treatment of vascular lesions [1]. By 2026, it is predicted that 75% to 95% of overall vascular lesions (aneurysms, stenosis, occlusive disease, traumatic vascular lesions, etc.) will be treated endovascularly [1,2]. Vascular surgery, as always, will continue to compete in recruitment with cardiac surgery for procedural domain but, with the increasing use of endovascular approaches, it faces additional recruitment competition from interventional radiology [2]. Due to the overlap in patient populations, professional interests, skills, and treatments performed by vascular surgeons, thoracic surgeons, and interventional radiologists, these specialties appeal to a common potential applicant pool and recruitment is highly competitive among these training programs.
Candidates for residency programs increasingly use the Internet to research potential programs for application [3][4][5]. Online information has been analyzed for a range of residency and fellowship programs, including orthopedic surgery, plastic and reconstructive surgery, emergency medicine, cardiothoracic surgery, neurosurgery, otolaryngology, trauma surgery, surgical critical care, acute care surgery, microsurgery, interventional radiology, and vascular surgery [3,[5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21]. Studies have individually analyzed the availability of online content for integrated vascular surgery, [21] thoracic surgery, [8] and interventional radiology [11,19] training program websites but, to our knowledge, no study has compared the accessibility and availability of online content across these training programs. Given the importance of online resources in recruiting prospective applicants and the current mandates to move to virtual interviewing, we sought to assess the current state of integrated vascular surgery, thoracic surgery, and interventional radiology training program websites. The purpose of this study is to evaluate the presence, accessibility, and comprehensiveness of integrated vascular surgery, thoracic surgery, and interventional radiology training program websites.

Methods
Study design: A comprehensive list of accredited integrated vascular surgery, thoracic surgery, and interventional radiology residencies was obtained from the Accreditation Council for Graduate Medical Education (ACGME) database. Programs participating in the 2020 National Resident Matching Program were eligible for study inclusion. Following identification of all programs with websites, programs were accessed and evaluated by two independent reviewers (one medical student and one resident) for availability of recruitment and educational content items. The websites were viewed independently by each reviewer. Program search and review was performed in November 2019.

Research Question:
Are there key differences in the three specialty program websites for integrated residencies that could potentially impair recruitment efforts in the virtual environment? Accessibility of websites: Accessibility of websites was determined by surveying Accreditation Council for Graduate Medical Education (ACGME) database for the total number of programs listed and the presence or absence of website links. Links, if they were provided, were characterized as either functional or nonfunctional. Functional links led to a website. Nonfunctional links led to an error page. Functional links were then evaluated as being either 'direct' (landing directly on the program webpage) or 'indirect' (landing on a different page, such as the departmental website, requiring further action by the reviewer to access the specific program webpage if possible).

Availability of content:
Websites for integrated vascular surgery, thoracic surgery, and interventional radiology residency programs were analyzed for availability of information used to inform and recruit prospective applicants. Content items on recruitment and education listed in (Table 1) were selected based on ACGME program requirements as well as previously published literature reviewing online content of residency and fellowship programs [5,10,14,15] Content on the training program websites was counted as present if (1) it was present on the main training program webpage or (2) it was accessible via a direct link provided on the main training program webpage.
Program recruitment and education: Websites were evaluated for content relevant to program recruitment and education. Program recruitment information included faculty listings, faculty and departmental research interests, alumni career placements and information on current residents. Recruitment information regarding the application and interview process as well as general resident quality of life metrics were also evaluated see (Table 1). Program education content addressed operative and didactic training. It also covered resident research opportunities. Overall, 41 program recruitment and 16 program education content items were evaluated.
Rater training and consistency: Each website was accessed and evaluated by two reviewers (one medical student and one resident) for availability of content items as well as quality of websites (determined as a function of four dimensions: content, design, organization, and user friendliness, see Figure 1 for evaluation criteria). Each reviewer was trained by examining an optimal website, an average website, and a below average website with the senior author. Disputed assessments were resolved by consensus following discussion with the senior author. Reviewers were not blinded. Overall, there was considerable inter-rater reliability with 81% agreement (kappa = 0.74).

Data analysis:
Intergroup analysis of continuous variables was performed using ANOVA. Categorical variables were compared using Chi-squared analysis. Statistical significance was defined at p<0.05. Percent agreement and kappa statistics were calculated for inter-rater reliability. Statistical analysis was performed using statistical software R version 4.0.0 (2020-04-24, R Core Team, Vienna, Austria).  Table 2). Availability of content: Content was assessed in two domains: recruitment and education. Of the 57 recruitment and education content items included in this analysis, vascular surgery program webpages contained a median of 26 content items (IQR: 20, 32), thoracic surgery program webpages contained a median of 27 content items (IQR: 21, 32), and interventional radiology program webpages contained a median of 23 content items (IQR: 18,27). Of the 41 recruitment content items included in this analysis, vascular surgery program webpages contained a median of 19.5 content items (IQR: 15, 24), thoracic surgery program webpages contained a median of 20 content items (IQR: 16,24), and interventional radiology program webpages contained a median of 18 content items (IQR: 15,21). Of the 16 education content items included in this analysis, vascular surgery program webpages contained a median of 7 content items (IQR: 4, 9), thoracic surgery program webpages contained a median of 6 content items (IQR: 4, 7), and interventional radiology program webpages contained a median of 4 content items (IQR: 3, 7). A detailed quantification of availability of content items can be seen in Table 3.
For program education, almost all programs provided information on rotation schedule (84%). The majority of programs provided information on didactic instruction (70%), research requirements (73%), journal club (61%), and vascular lab training (RPVI; 64%). Less than one-half of programs provided information on departmental research interests (39%), operative experience (32%), meetings attended (46%), elective rotations (45%), conference schedule (38%), curriculum (38%), and simulation training (34%). Fewer than one-quarter of programs provided information on evaluation criteria (16%) and faculty mentorship   Table 2 Accessibility of integrated vascular surgery, integrated thoracic surgery, and integrated interventional radiology training program websites from the ACGME webpage.   Table 3 Availability of content on U.S. integrated vascular surgery, integrated thoracic surgery, and integrated interventional radiology training program websites.

Journal of Vascular and Endovascular Therapy
(11%). Almost no programs provided information on national organization links (7%) and cardiovascular product company links (4%).

Quality of websites:
On an overall assessment, integrated vascular surgery, thoracic surgery, and interventional radiology websites were found to be comparable. The average and standard deviation vascular surgery website score was 2.66 ± 0.95, the average and standard deviation thoracic surgery website score was 2.18 ± 0.92, and the average and standard deviation interventional radiology website score was 2.25 ± 0.88. The vascular surgery websites had the highest scores in content, design, organization, and user-friendliness. The thoracic surgery websites had the lowest scores in content, organization, and user-friendliness while the interventional radiology websites had the lowest score in design. Additional details regarding website quality, broken down by category, are visible in Table 4.

Discussion
As resident recruitment moves to a virtual platform, the internet is an increasingly important resource for residency applicants as they research programs. Thoracic surgery program webpages had the most information, then vascular surgery program webpages, with interventional radiology program webpages reporting the least content. This trend in availability of content items mirrors the percent of positions filled by each specialty, with 100% of PGY-1 thoracic surgery positions filled, 97% of vascular surgery PGY-1 positions filled, and 97% of PGY-1 interventional radiology positions filled (with 94% of PGY-2 interventional radiology positions filled), as reported by the NRMP 2020 Main Residency Match Results and Data report [22].
Other factors, beyond program websites, that have been identified to influence applicant interest in a program, include geography, advice from a mentor, advice from a peer, and other online information. The integrated vascular track was first accredited by the ACGME in 2006, [21] the first integrated thoracic surgery program accepted residents in 2007, [23] and the first integrated interventional radiology programs participated in the National Resident Matching Program (NRMP) in 2016 [24]. The majority of these integrated programs have been established for less than ten years. This increase in the number of integrated programs, This article is available in: https://vascular-endovascular-therapy.imedpub.com/ Journal of Vascular and Endovascular Therapy though necessary to meet the high demand for integrated residency positions, means that many programs do not have an established national presence. Applicants cannot receive the same quality of advice from mentors and peers on newer programs, as compared to programs that have been established for longer periods of time. Furthermore, many programs are geographically clustered, specifically in the Northeast and along the West coast (see Supplemental Figure 1). These factors combine to place additional weight on program websites, perhaps serving as the initial source of information for potential applicants and allowing for comparison.
Of ACGME accredited programs, 56 of 61 (92%) vascular surgery programs, 27 of 27 (100%) thoracic surgery programs, and 74 of 85 (87%) interventional radiology programs had functional websites. Thoracic surgery program webpages had the most information (content item median: 27, IQR: 21, 32), then vascular surgery program webpages (content item median: 26, IQR: 20, 32), with interventional radiology program webpages reporting the least content (content item median: 23, IQR: 18, 27). The greater amount of content on vascular surgery and thoracic surgery program webpages could be expected, given the young age of many interventional radiology programs. Previous studies have acknowledged integrated interventional radiology program webpages to be a work in progress [11]. This study confirms that finding in relation to longer-established vascular surgery and thoracic surgery program webpages.
Two content items that have been identified to be highly influential to applicant program decision are operative experience and faculty mentorship [25][26][27]. This analysis found those items to be reported at 32% and 11% for vascular surgery, 19% and 11% for thoracic surgery, and 50% and 15% for interventional radiology programs (P=0.008 and P=0.751, respectively). Additional notable deficits for vascular surgery websites were resident profiles, sample contracts, and departmental research interests. Thoracic surgery websites lacked program director contact information and message as well as information on operative experience. Interventional radiology websites had deficits in work hours, debt management, and curriculum. All specialty websites had deficits in evaluation criteria and faculty mentorship. In addition to addressing the deficits in program recruitment and education content items, the deficits in lifestyle management cannot be disregarded; medical students increasingly report controllable lifestyle as a major factor in specialty choice [28-30].
The deficits identified by this analysis are comparable to deficits identified for other specialties. Other studies have found considerable deficits in newsletter, resident listings and photographs, faculty contact information, and away elective rotation information for dermatology websites, [3] resident call schedule, alumni career placement, and salary for orthopedic surgery websites, [6] academic conference schedule, call schedule, operative case listing, graduate fellowship information, and board exam performance for plastic surgery websites, [14] evaluation criteria, call schedule, operative exposure, national meetings attended, debt management, alumni contact, and work hours for neurosurgery websites, [15] call schedule, away elective rotation information, resident profiles, and faculty research for general surgery websites, [16] and call schedule, active/past research projects, area information, message from the program director or chair, selection criteria, salary, and surgical statistics for otolaryngology websites [17].
Overall, we recommend that programs address the deficits in specific content items identified by this analysis. Given the increasingly important role of online information in the residency application process and the anticipated transition to a virtual application process for the 2021 cycle, it would behoove programs to increase their online presence. In addition to the content items included in this analysis, it might be fitting for programs to include more personal information (i.e. more detailed resident and attending profiles) to give applicants a better idea of the personality of different programs, replacing the role previously served by in-person away rotations and interviews.
This study had several limitations. First, this data is representative of what information was available online at the time of data collection. It is possible that websites could have been edited or new program websites could have been published since that time. Additionally, though an extensive list of content items were evaluated by reviewers regarding program recruitment and education, it is possible that other, unmentionedcontent items could hold bearing on applicant decision. Finally, reviewers were not blinded to what program they were evaluating. Thus, any inherent bias reviewers might have had for particular programs was not controlled for. The nature of this study did not lend itself to evaluating the association between website content, to what specialty and to what programs applicants apply, and ultimate applicant program placement. Future studies could seek to characterize this trajectory.

Conclusion
This study has uncovered key differences in availability of online content for residency programs recruiting from the same pool of applicants. Thoracic surgery program webpages have the most information, then vascular surgery program webpages, with interventional radiology program webpages reporting the least content. Recruitment is highly competitive between vascular surgery, thoracic surgery, and interventional radiology with the ratio of applicants to positions being highest for thoracic surgery, then interventional radiology and lastly vascular surgery, as reported by ERAS. To attract valuable candidates, programs should aim to increase the availability of online content for potential applicants.

Funding
This research received no specific grant support from any funding agency in the public, commercial, or not-for-profit sectors.