This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.
Massive open online courses (MOOCs) have been criticized for focusing on presentation of short video clip lectures and asking theoretical multiple-choice questions. A potential way of vitalizing these educational activities in the health sciences is to introduce virtual patients. Experiences from such extensions in MOOCs have not previously been reported in the literature.
This study analyzes technical challenges and solutions for offering virtual patients in health-related MOOCs and describes patterns of virtual patient use in one such course. Our aims are to reduce the technical uncertainty related to these extensions, point to aspects that could be optimized for a better learner experience, and raise prospective research questions by describing indicators of virtual patient use on a massive scale.
The Behavioral Medicine MOOC was offered by Karolinska Institutet, a medical university, on the EdX platform in the autumn of 2014. Course content was enhanced by two virtual patient scenarios presented in the OpenLabyrinth system and hosted on the VPH-Share cloud infrastructure. We analyzed web server and session logs and a participant satisfaction survey. Navigation pathways were summarized using a visual analytics tool developed for the purpose of this study.
The number of course enrollments reached 19,236. At the official closing date, 2317 participants (12.1% of total enrollment) had declared completing the first virtual patient assignment and 1640 (8.5%) participants confirmed completion of the second virtual patient assignment. Peak activity involved 359 user sessions per day. The OpenLabyrinth system, deployed on four virtual servers, coped well with the workload. Participant survey respondents (n=479) regarded the activity as a helpful exercise in the course (83.1%). Technical challenges reported involved poor or restricted access to videos in certain areas of the world and occasional problems with lost sessions. The visual analyses of user pathways display the parts of virtual patient scenarios that elicited less interest and may have been perceived as nonchallenging options. Analyzing the user navigation pathways allowed us to detect indications of both surface and deep approaches to the content material among the MOOC participants.
This study reported on first inclusion of virtual patients in a MOOC. It adds to the body of knowledge by demonstrating how a biomedical cloud provider service can ensure technical capacity and flexible design of a virtual patient platform on a massive scale. The study also presents a new way of analyzing the use of branched virtual patients by visualization of user navigation pathways. Suggestions are offered on improvements to the design of virtual patients in MOOCs.
The rise of interest in massive open online courses (MOOC) is remarkable. What started as an experiment in connectivist learning theory turned, in just a few years, into a phenomenon involving several million participants and the most prestigious universities [
Health sciences, traditionally slower in adaptation of new trends in education, are inspired by these changes as well [
The currently predominant type of massive open online courses, called xMOOCs, has been criticized for building on bite-sized video lecture clips, textbooks, and multiple-choice questions—regarded as a rather outdated form of learning [
Virtual patients have shown a positive effect on learning [
We began with a prior theoretical analysis of the idea of embedding virtual patients in MOOCs, from an educational point of view [
The primary objective of this paper is to present an in-depth analysis of the technical preparations required to include virtual patients in a MOOC. This will inform the parameters for future preparatory tests and suggest solutions for dealing with the intensive usage of information technology infrastructure resources while offering a server-side educational component for a massive audience. The second objective is to identify ways of highlighting the different navigation pathways of virtual patient interactive use. We hope that this study will reduce the technical uncertainty related to such extensions, indicate aspects that could be optimized for a better learner experience, and also raise prospective research questions by describing indicators of virtual patient use on a massive scale.
In particular, we are interested in answering the following two research questions: (1) What are the information technology challenges and technical solutions for offering virtual patients in a MOOC? (2) How can user navigation pathways be presented for virtual patients integrated within a MOOC?
This report is a case study carried out at Karolinska Institutet (KI), a Swedish medical university, in the second half of 2014. KI was the first Scandinavian university to join the edX consortium launched by Massachusetts Institute of Technology and Harvard to create and disseminate MOOCs [
Being part of the edX consortium required using the edX MOOC platform to host the course [
The Behavioral Medicine course was designed for a 5-week run. In order to increase active learning opportunities it was decided that weeks 2 and 3 would be illustrated by virtual patients in the form of interactive patient scenarios. The week 2 scenario dealt with treatment of stress-related symptoms; week 3 with treatment of sleep problems. These scenarios were presented as separate cases but both were connected by the story of John Nilsson, a high school teacher suffering from stress and sleeping problems (
The virtual patient scenarios consisted of 80 and 61 screen cards or nodes, respectively, containing text description, decision elements, free-text assignments, multiple-choice questions, and videos (
The Behavioral Medicine MOOC started on September 9, 2014, and lasted 5 weeks, until October 14. The week 2 virtual patient was available for the first time on September 16 and the week 3 virtual patient became available on September 23. All services were active for two more weeks (until October 28) for a tapering period.
Virtual patient presented in the edX Behavioral Medicine course.
The OpenLabyrinth system is successfully used at several medical universities worldwide. However, to the best of our knowledge, it has not been used as part of a large course yet. As a recent systematic review has shown, an average MOOC involves 43,000 enrollments, and there have been reports of courses with over 200,000 registered users [
Cloud computing is a way of providing computing resources as services that can be created on demand, usually in the form of virtual servers (machines). Clouds have been applied by industry to dynamically scale Web applications to respond to varying peaks in workloads. They have also been successfully used in scientific and healthcare applications, where they allow for obtaining computing resources quickly to compute intensive simulations or data analysis tasks [
To evaluate the performance and scalability of OpenLabyrinth, we used Gatling (v1.5.5) [
The stress tests showed a very good response time in all cases (around 100 ms), low RAM (max 460MB) and processor use (max 1.3CPU) but around one percent (0.66%) of requests for 300 concurrent users led to a 404-page error. This was traced to a database deadlock problem. The error rate for 50 users was significantly lower (0.02%). Since it was not feasible for us to find the source of the deadlocks in the given timeframe, we decided to mitigate the risk of this error by reducing the number of concurrent users using cloud technologies. The idea was to have more numerous but less powerful virtual servers to share the user requests in a balanced way and thus decrease the likelihood of database problems.
For the implementation we approached the VPH-Share project, specialized in offering cloud services for biomedical applications [
As an emergency backup solution in case of unexpected technical problems, we had prepared an alternative virtual server template consisting merely of the Web server with a preinstalled HTML version of the virtual patients, not requiring the database system. This version was designed by writing a script exporting OpenLabyrinth cases to a set of static HTML web pages. Obviously, this version had fewer computational requirements; however, it also had limitations in terms of restricted functionality, as tracing user sessions or recording students’ answers was not possible. In the end it turned out to be unnecessary to use this template during the MOOC, but it provided security for the project team.
VPH Share cloud platform with virtual patient system templates prepared for the Behavioral Medicine MOOC and a running instance of one of the templates on a virtual server.
Cloud architecture for the Behavioral Medicine MOOC.
Web server and database logs as well as user sessions were recorded in the OpenLabyrinth system. The data were archived and aggregated from the four virtual servers after the tapering period had ended. There was no need to add new parallel servers. The Web server logs were parsed using a simple Java language script and postprocessed in Excel (Microsoft). Spearman correlation coefficients were calculated in Statistica v10 (2010) (Statsoft), using a significance level of 5%.
User pathways were analyzed by an on-purpose project-designed analytic tool developed in Java by one of the authors of this article (AAK). The developed tool visualizes the results by displaying them as numerical and grayscale values in the navigation graph. The structure of the graph is read from VUE files used previously in designing the virtual patients. Next, the XML content from VUE files is copied and modified based on the user session statistics from OpenLabyrinth and once again opened in VUE to display the result. Four different types of visual analyses were generated: (1) number of visitors in each screen card of the virtual patient, (2) pathway exit points: percentage of visitors who ended the session in the given node, (3) time in seconds spent in each screen card on average, (4) percentage of visits selecting individual decision options and branching ability of decision nodes.
In order to evaluate participant satisfaction with the course in general as well as the virtual patient experience and to confirm the quality of technical capacity offered in the MOOC, the participants were surveyed for their opinions on using virtual patients via an anonymous questionnaire to which invitations were distributed immediately after the course. The Likert-scale questions were analyzed using Excel for descriptive statistics. Free-text comments detailing the technical issues encountered by course participants were analyzed qualitatively for recurring themes.
Data on user enrollment and declared completion were acquired from the edX platform statistics (edX insights). In this study, we did not trace the link between sessions and user demographics or learning outcomes and treated user data entirely anonymously. This type of research does not require explicit permission of an ethical review board according to Swedish law (Act 2003:460).
The number of enrollments reached 19,236 but just 4586 (23.84%) logged in during the first week of the course. On the official closing date of the course (October 14), 2317 (12.05% of the total enrollment rate) and 1640 (8.53%) participants had declared their respective completion of the week 2 and week 3 virtual patients. This number kept growing after the course was officially closed in the tapering period. The honor code certificate for the whole course was earned by 740 participants, or 3.85% of the total number of original enrollments. The most frequent participant countries of residence were: United States (27.61%), India (8.97%), and United Kingdom (4.84%). See
In general, the most active hour of the day was 18:00-18:59 CET with 181 server requests on average. The least active time was 07:00-07:59 CET with 84 server requests on average (
We also analyzed the number of errors reported in the Web server logs. Out of the 131,303 server requests, only 35 could be traced as having been caused by database problems (0.03%).
Virtual patient server requests per day in the Behavioral Medicine MOOC.
Average virtual patient server requests per hour in the Behavioral Medicine MOOC.
The analysis of the number of visitors to the virtual patient scenarios shows a relatively high number of initiated sessions: 3467 in week 2 and 2201 in week 3 (
The frequency distribution of the number of visited screen cards in one session had a local peak around the length of the critical path (number of screen cards in the shortest pathway connecting the start and end nodes). The frequency distribution for week 2 had an additional peak around 51 nodes, corresponding to the group of students who selected a branch with 16 nodes for additional explanations. The frequency distribution of user pathways longer than the length of the critical path drops steadily, reaching the zero level at 79% (week 2) and 90% (week 3) of the total number of nodes in the scenarios.
Number of visitors in screen cards of week 2 virtual patient.
Average time spent on screen card types.
|
Week 2 | Week 3 | ||
Screen card type | Count | Time [s] | Count | Time [s] |
Decision node | 19 | 28.6 | 11 | 27.2 |
Free text question | 6 | 120.3 | 5 | 131.6 |
Multiple choice question | 1 | 66.4 | 2 | 23.1 |
Review | 0 | - | 4 | 124.3 |
Text | 35 | 17.5 | 25 | 20.4 |
Video | 19 | 98.8 | 14 | 73.4 |
In the week 2 scenario, the shortest possible path from the start to the end node required 37 minutes, but the average session in which the case was completed (containing one of the end nodes) lasted 48 minutes. One hundred twenty-one (n=121) sessions that completed the virtual patient were longer than one hour. For the week 3 scenario, the shortest path consisted of nodes taking 27 minutes on average, where the mean case completion session was 39 minutes long, with 56 sessions longer than one hour.
Percentage of visitors who left the virtual patient after visiting the particular screen card.
Visualization of learner choices in decision nodes.
We received 479 responses to the participant survey (2.49% of all enrolled and 20.67% of those who declared they had completed at least one virtual patient).
Eighty-one percent (389 out of 479; 81.2%) of surveyed participants did not encounter any technical problems while using virtual patients. Nevertheless, it was surprising that a relatively high percentage (77 out of 479; 16.1%) reported experiencing technical issues. The dominating themes in the free text description of technical issues were (1) problems with videos, (2) Internet connection problems, (3) problems with broken sessions, and (4) unrelated problems. Each of these themes contained around 15 comments. The video problems related to the inability to access YouTube videos in some countries or the inability to download them offline. Sometimes the videos did not seem to start properly, got stuck, or played only the audio track—these problems could be explained by slow Internet connections or compatibility issues with Web browsers on some platforms. The problems explicitly classified as Internet issues referred in general to the inaccessibility of broadband connections in regions of the world such as India or rural Canada. The issues classified as broken sessions were most likely related to the virtual patient platform and manifested by nonsaved user answers, page display errors, unexpected session reset, or (in two reported cases) surprising redirection between nodes. Some of the problems could be explained by session timeout or browser issues but they could just as likely be due to previously identified software problems in the platform. Luckily, these difficulties were not reported often and did not influence the general positive picture of the learning experience. Lastly, the technical complaints also contained issues not directly related to the technical side of virtual patient scenarios, like language issues, difficulties in posting comments on the discussion board, or complaints about videos that were not part of the virtual patient scenarios.
This case study presents experiences collected while organizing what is probably the first MOOC including virtual patients. The scope of the report is on virtual patient usage expressed in terms of required technical capacity and user navigation pathways to inform preparations of further, similar courses. The general impression after introducing virtual patient scenarios into the Behavioral Medicine MOOC was positive. The great majority of participants regarded virtual patients as a helpful exercise in the course, and the declared completion rate of the virtual patients activity (12.05% and 8.53% out of 19,236, weeks 1 and 2, respectively) was relatively high in relation to the total course completion rate (3.85%). We received many free text comments in the discussion forums praising this learning resource (Berman et al, manuscript in preparation). The course organizers will retain this activity for continued runs of this MOOC and have begun using the virtual patients in their traditional campus-based teaching activities.
From the organizers’ perspective, the intimidating number of participants at start-up was not as challenging from the technical perspective as expected. One contributing factor is the high dropout rate typical for MOOCs, and the more evenly distributed frequency of requests than expected due to the truly global reach of the learning activity. In the end, the technical infrastructure was loaded, in terms of concurrent users, 20 times less intensively than anticipated as the worst-case scenario. The server capacities of one virtual machine would be most likely have been sufficient to host the course even when considering the database problems noted in the course live run.
Considering the virtual patient statistics obtained, the necessity of using cloud infrastructure might be questioned. However, several benefits to this solution were confirmed in this study. A platform-as-service saves costs, as it was not necessary to purchase any new hardware for the MOOC to guarantee a dedicated server for the virtual patient platform. The possibility of scaling the solution—horizontally (number of CPU and RAM) and vertically (number of instances)—provided us with security in case of higher than expected popularity of the course. The openness of MOOCs suggests the real need of taking such precautions. The solution of preparing different types of virtual server templates was very helpful. This provided flexibility for the course organizers in terms of switching between different services outside the MOOC platform, depending on how the course evolves in real time. It also saved time while reusing the services for other occasions (eg, for on-campus classes with small groups of students). The idea of creating micro-instances of virtual servers to circumvent problems with concurrent access to legacy code is new and has not been discussed in the literature before [
The OpenLabyrinth system proved its usefulness in this study as a virtual patient player for a MOOC. The technical problems reported by some students did not seem to overshadow the general positive picture of the activity. The problems we traced as directly related to the OpenLabyrinth environment were not numerous. This would be interpreted differently in the case of commercial courses or high-stake examinations organized on a massive scale. The most often reported problems with low Internet connection for videos or banned YouTube services are not related to the virtual patient software. The course participants were informed by the organizers in the general course rules about restrictions in use of the YouTube service in some countries and offered manual downloads of the video clips outside OpenLabyrinth; however, this solution was not offered explicitly in OpenLabyrinth. This seemed to negatively influence users and suggests other options should be considered for hosting videos. We recommend uploading all videos as separate files to the MOOC platform to be downloaded on demand by participants from countries with blocked access to services such as YouTube. An alternative solution would be to host a video streaming service on a generic cloud infrastructure, but this would considerably raise the technical requirements. We were not able to spot any irregularities which would indicate users of specific web browsers had particular problems using virtual patients in OpenLabyrinth. It is to be acknowledged that we did not optimize the display for mobile devices (tablets, smartphones), a factor which could be an issue for some of the participants. We recommend considering this group in particular in upcoming MOOCs introducing virtual patients.
The course was slightly less frequently visited and completed than an average MOOC. This might be explained by the specificity of the topic and the general trend of larger supply and shrinking interest in MOOCs [
One worrisome aspect is the discrepancy between the self-reported completion rate of the exercise and the actual session logs. It was not necessary to complete the virtual patient exercise in order to tick off the assignment; users merely had to spend 30 minutes in the scenario and comment on this on the discussion board. The declared completion rate of the week 2 assignment was 2317, but the number of sessions containing the second node of virtual patient was 1473. This raises doubts regarding how seriously participant declarations are to be taken. The observation of digital dishonesty is not new [
One innovative aspect of this study is the use of visual analytics methods to report user activities in virtual patients. Visual analytics is an emerging trend employing human cognitive abilities to recognise visual patterns in analytical tasks [
The observation of the number of visited nodes in user pathways and the moderately strong positive correlation of the percentage of viewed videos to the distance from the start node might suggest the existence of two groups of MOOC participants using virtual patients: learners with either surface or deep approaches to the content material. The existence of such groups is predicted by educational theories [
This study has its limitations. There were just two virtual patient scenarios introduced to the course using OpenLabyrinth. The branching potential of virtual patients has not been used to its full extent because of limited resources for developing alternative versions of videos, fixed recommendations of professional behavior which should not encourage too much experimentation, and the novelty of this form of teaching to the subject matter experts of the course. It was also assumed from the beginning that this study would not link session data to user details. The study evaluates the idea of introducing virtual patients in a MOOC at the first level (participation) of Kirkpatrick’s hierarchy [
Future research should focus on observing how participants choose navigation pathways depending on participant-related factors. This could give insight into how experts, expert students, and lay people approach problem solving in a MOOC. It would be interesting to look for correlations between such indicators as branching level and student satisfaction from learning. Technical solutions used in this paper, such as building different variants of virtual patient system templates or using visual analytics methods to improve virtual patient quality are innovative and not yet optimized nor standardized. These could form integral elements of a cloud-based platform dedicated to organizing health-related MOOCs. The introduction of virtual patients to massive audiences, thanks to the large number of participants, opens up new, previously inaccessible venues for experimentation with this learning design with potential for future research on the topic.
This study reported on the probable first introduction of virtual patients to a MOOC. It positively verified the feasibility of using OpenLabyrinth, an open-source, freely available virtual patient system for a large audience setting. The system can now be added with greater confidence to future health care MOOCs. This report delivered concrete technical parameters (like the number of users per hour) to inform preparatory stress tests carried out prior to extension of health care MOOCs by adding nonstandard server-based interactive components. It demonstrated to the medical community how a cloud infrastructure (using the example of VPH-Share, but generalizable to comparable commercial solutions) can be employed in teaching activities on a large scale to deal with problems with errors in legacy code preventing high numbers of concurrent users, limitations in availability of hardware resources, or the need to prepare and store different configurations of the software tool. The paper further recommends adding to the existing branched virtual patient systems additional components graphically visualizing user pathways by demonstrating the spatial relationship between statistics pertaining to particular nodes. This is an added value to the existing tabular forms enabling the presentation of session statistics in learners’ areas of interest. Nodes rarely visited or visited for a shorter time than expected based on content, as well as branches seldom taken, can be analyzed in a context of preceding and following nodes and in a general overview of the whole case structure. This can be done to increase the attractiveness of the case or detect problems in understanding the content.
The innovation was warmly welcomed by the majority of course participants responding to the survey. A few challenges remained regarding accessibility and slow transfer of YouTube videos and occasional unexpected technical behavior on the part of the virtual patient system. We hope that this article will contribute to the expansion of future health-related MOOCs with interactive elements such as virtual patients.
Geographic breakdown of participation.
Detailed results of the user survey.
Karolinska Institutet
massive open online course
Virtual Physiological Human
We would like to thank all members of the KIBEHMEDx project group in preparing the MOOC (Bios section [
Author AHB wrote the virtual patient scenarios following consultation with guidance from AAK and NS for structuring the scenarios. TB, PN, and MM were part of the team that developed the Atmosphere cloud platform. TB, PN, MM, and AAK designed the architecture and deployment of the virtual patients in the cloud environment. TB, AAK, and NS configured, tested, and monitored the server templates instances. AAK developed the visual analytics tool prototype. CMG and NZ coordinated KI’s participation in the edX initiative. AHB was director of the KIBEHMEDx MOOC. AAK, AHB, NZ, and NS planned the analysis and analyzed and interpreted the results. AAK wrote the text of the manuscript with contributions from all authors. All authors have read and approved the final manuscript.
None declared