To compile the clinicians’ profiles and their evaluations of the CEI online CME/CNE courses, we queried the CEI Student Portal and obtained data of: (1) all clinicians who successfully completed at least one online CME/CNE course from November 2014 through October 2015; and (2) their evaluations of all completed online CME/CNE courses during this period. The CEI Student Portal is a centralized system to manage clinician students, online courses, and program evaluations. A clinician user can sign up for the student portal at any time. As part of the registration, clinicians need to provide their personal (contact information, demographics, and education levels) and professional (clinical discipline, employment setting, practice years, patient caseload, and clinical services) background information. When taking an online CME/CNE course, a clinician must complete watching the multimedia materials and pass a short exam to test his/her knowledge related to the topics covered by the course. After completion of each online CME/CNE course, the clinician student must provide an evaluation with self-reported data on: (1) usefulness/relevance of the information presented in the course; (2) ease of comprehension; (3) knowledge of the trainer; (4) appropriateness of the online training format; (5) knowledge level on the course topic before and after the training; (6) intention to utilize the learned knowledge; and (7) plan to change practice after the training. Here, for items (1), (2), (3), and (6), we used a five-point Likert-scale measure (strongly agree, agree, neutral, disagree, and strongly disagree), and the responses were further grouped into positive (strongly agree and agree) and nonpositive (neutral, disagree, and strongly disagree) responses for data analyses. For item (5), we used five discrete levels to indicate a clinician’s knowledge on a specific training topic (novice, not very knowledgeable, knowledgeable, very knowledgeable, and expert) before and after the training, and then calculated the difference (at least one level increase vs no increase) for data analyses. In addition to these structured questions for course evaluation, clinicians can provide free-text comments on their experiences of the CEI online training program. Once the evaluation is completed, the CME/CNE credits associated with an online course are immediately awarded. At any time, the clinician student can review and print the CME/CNE certificate. Partial screenshots of the CEI Student Portal to collect course evaluation and student profile data are shown in Figures 1 and 2.

For data analyses, we categorized the online CME/CNE courses based on a custom-developed domain ontology of HIV-HCV-STD clinical topics [9]. We then profiled all clinicians’ backgrounds and computed their evaluations of the online CME/CNE courses with the seven measures discussed above. As the last step, we reviewed the themes from the free-text comments to capture additional feedback for the CEI online training program. This study was approved by the University of Rochester Research Subjects Review Board and Arizona State University Institutional Review Board.

We recorded a total of 4363 completions of 88 online CME/CNE courses by 1976 unique clinicians during the study period. Among these clinicians, 1333 (67.46%) were female, 1103 (55.82%) were white, 492 (24.90%) were African American, 128 (6.48%) were Asian, and 436 (22.06%) were Hispanic. In terms of the clinicians’ education level, 308 (15.59%) held a doctoral degree and 486 (24.60%) held a master’s degree. The details of the clinicians’ demographics and education levels are shown in Table 1.

The top categories of the recorded primary professional discipline were physician (n=257, 13.01%), nurse (n=229, 11.59%), case/care manager (n=219, 11.08%), nurse practitioner (n=148, 7.49%), and social worker (n=148, 7.49%). Regarding the practice years in the primary professional discipline, 124 (6.28%) indicated more than 30 years of experience, 224 (11.34%) between 21 and 30 years, 402 (20.34%) between 11 and 20 years, and 1182 (59.82%) 10 years or less. For clinicians’ employment setting, the top categories recorded were community health center (n=379, 19.18%), hospital/hospital clinic (n=350, 17.71%), state/local health department/clinic (n=190, 9.62%), private/group practice (n=169, 8.55%), managed care organization (127, 6.43%), and professional organization (n=108, 5.47%). Among these, 1257 (63.61%) were in urban settings, 439 (22.22%) were in the suburbs, and 280 (14.17%) were in rural areas. Details of the clinicians’ primary professional disciplines, years in practice, and employment settings are provided in Table 2.

For the caseload, 171 (8.65%) clinicians reported seeing more than 100 HIV patients per month, 82 (4.15%) reported seeing more than 100 HCV patients per month, and 108 (5.47%) reported seeing more than 100 STD patients per month. The caseload category of 1-10 patients per month recorded the largest number of clinicians (n=515, 26.06% for HIV; n=466, 23.58% for HCV; and n=454, 22.98% for STD). The top clinical services provided for HIV patients were risk reduction intervention (n=617, 31.22%), case management (n=456, 23.08%), and HIV screening/testing (n=447, 22.62%). The top services for HCV patients were risk reduction intervention (n=538, 27.23%), HCV screening/testing (n=381, 19.28%), and HCV treatment (n=179, 9.06%). The most popular services for STD patients were risk reduction intervention (n=623, 31.53%), STD screening/testing (n=359, 18.17%), and STD treatment (n=289, 14.63%). The details of the clinicians’ caseloads and clinical services are provided in Table 3.

Regarding the number of online CME/CNE courses taken by the health care providers, 1454 (73.58%) clinicians completed only one course, 182 (9.21%) completed two, and the remaining 340 (17.21%) clinicians completed three or more. Excluding a CEI staff who worked as a system tester, the largest recorded number of online CME/CNE courses completed by a single clinician was 47, and the average number of courses taken was 2.17.

As a measurement of course popularity, we recorded a list of courses with the most numbers of completions. The top three on the list were: (1) HIV/AIDS Confidentiality Law Overview, with 1379 completions; (2) Street Drugs & HIV, with 195 completions; and (3) STD-HIV Inter-Relationship, with 158 completions. A complete list of the top 20 most popular online CME/CNE courses is shown in Table 4.

In terms of course evaluation, the overall responses from the clinicians were very positive. With the 4363 completed courses, 3974 responses (91.08%) indicated that the information presented was useful and relevant, 3887 (89.09%) indicated that the course was easy to comprehend, 4014 (92.00%) indicated that the trainer was knowledgeable, and 3680 (84.35%) indicated that the online training format was appropriate. With regard to the impact of the online training, 2117 of the 4363 responses (48.52%) indicated at least one level increase in knowledge, 3720 (85.26%) indicated intention to use the learned knowledge, and 959 (21.98%) indicated a plan to change clinical practice. Excluding those who did not provide direct patient services at the time, 39.43% (959/2432) of those who responded planned to change their clinical practice after the training. A summary of the course evaluation data is shown in Table 5.

A total of 288 comments from the clinicians were collected during the study period. Review and analysis of these comments excluded 156 invalid or meaningless responses. For the remaining 132 comments, we further categorized them into five themes: information/content, format, trainer, technical issues, and general positive feedback, with 4 belonging to two separate themes. Among these, 39 indicated that the clinicians learned a lot from the presentation and found it very informative. For example, a comment under this theme was “the information is useful for updating agency policy,” indicating that the clinician planned to utilize the learned information in their facility. The 53 general positive feedbacks such as “I like it” and “excellent program” only expressed clinicians’ overall positive experiences but without details. Additional positive comments were recorded on knowledgeable trainer (n=3) and appropriate online format (n=1). We recorded 25 comments about technical issues encountered by the clinicians, for example, “video was cut off at 40 minutes’ mid-sentence.” Other negative comments included challenging materials or other information issues (n=4), problem with the trainer (n=5), and request for improvement of the format (n=6).

In this study, for the first time, we reported a detailed profile of health care professionals participating in a statewide online HIV-HCV-STD knowledge dissemination program. Although there were limited reference data, we had a few interesting findings. First, compared to the data of general health care workforce in New York State [37], the proportion of the CEI program participants working in nonhospital community settings was much higher (82% vs 58%). Comparison to the data of active registered nurse in New York State [38] showed similar results (82% vs 29%). These data demonstrated that the CEI online program successfully reached out to the primary care communities, as defined by its missions. Second, 14% of the CEI program participants were from rural areas, similar to the active registered nurse data in New York State (14%) [38]. Further research is required to assess the detailed geographical distribution of the participants and the potential changes over time. Third, the demographics of the CEI program participants indicated a slightly more racially/ethnically diversified workforce (56% white, 25% black, 6% Asian, and 22% Hispanic) as compared to the data of active registered nurses in New York State (61% white, 18% black, 12% Asian, and 6% Hispanic) [38]. This perhaps reflected the higher incidence and prevalence of HIV, HCV, and STD infections in minority groups and therefore the need for more racial/ethnic diversity in health care provider teams. Future research is required to explore race and ethnicity as co-variates for clinician’s course evaluation.

To relieve the potential concern on privacy, we did not collect CEI participant’s date of birth. Instead, we used years of practice as an alternative measure, which we believed could roughly estimate a health care provider’s age in most cases. The data showed that a majority (60%) of the CEI audience had 10 years or less of experience in practice. This result was consistent with previous finding that an online program tended to attract younger clinicians [14]. In addition, 18% of clinicians in the program had more than 20 years of working experience, indicating that this group of health care providers was actively engaged in the CEI online program.

Similar to a previous pilot study [39], we recorded data of clinicians from a variety of professional disciplines participating in the CEI online program. As increasingly recognized and advocated by the research community [40], the interprofessional approach for knowledge dissemination defined another contribution of our study. Related to that, we identified a comprehensive set of clinical services provided by the professionals participating in the CEI online program. Future research is required to assess potential interactions among professional disciplines, clinical services, training topics, and course evaluations.

With regard to caseload, our data indicated a wide range of distribution, from 0 to 100+ HIV, HCV, and STD patients per month. The largest category of the participating clinicians had 1-10 patients per month, a typical scenario in the primary care setting. It is interesting to note that 4%-9% of the clinicians had 100+ patients per month, indicating that the CEI online program also attracted HIV, HCV, and STD specialists, as suggested by anecdotal evidence. In addition, 39%-54% of the clinicians had no patients at the time—they likely were those: (1) in the HIV, HCV, and STD clinical care and public health teams but with no direct interactions with patients; or (2) new practitioners preparing to engage in HIV, HCV, and STD patient care.

According to the Accreditation Council for Continuing Medical Education 2015 Annual Report, there were close to 37,000 enduring internet material activities that provided more than 74,000 hours of instruction to 4.8 million physicians and over 6.8 million other learners [41]. Nevertheless, just posting materials online for reading is not enough. Studies showed that multimedia materials and interactivity would increase the satisfaction of health care professionals for their participations in online learning [42]. Other important factors used by clinicians in the selection of an online course included the quality of content, flexibility of access, ease of use, and convenience of obtaining continuing professional development credits [42]. We believe that effectively addressing these issues has led to the initial success of the CEI online program.

The data from a recent national survey called for a central repository for listing educational opportunities and tracking continuing education credits [43]. Nonetheless, another study found that online materials from open public sources such as YouTube had overall low quality [44]. A large repository of centrally managed, high-quality online courses with a student portal to track the progress of learning and CME/CNE credits is likely an important contributing factor for the success of the CEI online program in engaging a core group of clinicians. Our data showed that this core group of clinicians kept coming back to the CEI online program to take new courses.

This study has a few limitations. First, we focused only on a specific category of online resource, multimedia CME/CNE course, while many other types of resources such as case simulation, InfoButton, and clinical decision support could also be leveraged for dissemination of clinical evidence [20,43,45]. As advocated by others, multi-faceted interventions will likely be more effective for knowledge dissemination [20]. The CEI online program has already developed a few other types of resources and communication channels [9-11]. Future research is required to assess the effectiveness to use multiple categories of online resources for knowledge dissemination. Second, this research was based on a cross-sectional descriptive study design without comparative groups (except for the measure of clinicians’ knowledge level, which involved a before-after comparison). Future research is required to compare the effectiveness of the interventions delivered through the online resources with specific types of controls in order to demonstrate that the benefits of online courses outweigh the disadvantages [42-43]. Third, we relied on self-reported data [46] to measure clinicians’ satisfaction on content, format, knowledge change, and potential improvement in clinical practice. Given that CEI is a statewide program, this is perhaps the best we could do. Future research should include direct measures on health care processes and patient outcomes [47]. Finally, since the CEI online program was newly developed at the time, there were still technical issues (as shown in the clinicians’ comments) during the study period. The CEI program staff were actively working to address those issues found by the clinician students. Ideally, a system should be frozen during evaluation [48]. However, considering a large program such as CEI that serves thousands of clinicians, it is unrealistic to freeze the system and ignore the identified issues for a long study period. We therefore decided to evaluate the system along the normal process of system maintenance, which reflected the real status of its performance.

In conclusion, we have successfully characterized the profiles of clinicians participating in the New York State CEI online CME/CNE program for dissemination of the latest HIV, HCV, and STD clinical evidence. We have identified a diverse group of health care providers in terms of the demographics, education levels, professional disciplines, practice years, employment settings, caseloads, and clinical services. The participating clinicians’ evaluation of the CEI online CME/CNE courses was very positive with regard to the content, format, and knowledge increase. A significant portion of the participating clinicians planned to adopt the learned knowledge and skills in their practice. These initial set of evaluation data have demonstrated the effectiveness of using online resources for dissemination of HIV, HCV, and STD clinical evidence to primary care clinicians. Future research is required: (1) to assess the effectiveness of knowledge dissemination with multiple types of online resources and using direct measures on health care processes and patient outcomes; (2) to examine the comparative effectiveness with control groups; and (3) to analyze the co-variates through development of a predictive model for effective online training.