This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.
The use of mobile technology in e-learning (M-TEL) can add new levels of experience and significantly increase the attractiveness of e-learning in medical education. Whether an innovative interactive e-learning multimedia (IM) module or a conventional PowerPoint show (PPS) module using M-TEL to teach emergent otorhinolaryngology–head and neck surgery (ORL-HNS) disorders is feasible and efficient in undergraduate medical students is unknown.
The aim of this study was to compare the impact of a novel IM module with a conventional PPS module using M-TEL for emergent ORL-HNS disorders with regard to learning outcomes, satisfaction, and learning experience.
This pilot study was conducted at an academic teaching hospital and included 24 undergraduate medical students who were novices in ORL-HNS. The cognitive style was determined using the Group Embedded Figures Test. The participants were randomly allocated (1:1) to one of the two groups matched by age, sex, and cognitive style: the IM group and the PPS group. During the 100-min learning period, the participants were unblinded to use the IM or PPS courseware on a 7-inch tablet. Pretests and posttests using multiple-choice questions to evaluate knowledge and multimedia situational tests to evaluate competence were administered. Participants evaluated their satisfaction and learning experience by the AttrakDiff2 questionnaire, and provided feedback about the modules.
Overall, the participants had significant gains in knowledge (median of percentage change 71, 95% CI 1-100,
Using M-TEL for undergraduate medical education on emergent ORL-HNS disorders, an IM module seems to be useful for gaining knowledge, but competency may need to occur elsewhere. While the small sample size reduces the statistical power of our results, its design seems to be appropriate to determine the effects of M-TEL using a larger group.
ClinicalTrials.gov NCT02971735; https://clinicaltrials.gov/ct2/show/NCT02971735 (Archived by WebCite at http://www.webcitation.org/6waoOpCEV)
Generalism is one of the most important aspects of the novel 6-year program of undergraduate medical education (UME) that was implemented in Taiwan in 2013. The goal of UME is to provide graduates with core knowledge and skills at the highest level of competency and then to become general physicians [
Since increasing the number of hours dedicated to ORL-HNS in the classroom and hospital is not practical, novel UME requires enabling self-directed learning and augmenting learning outside the classroom [
In this study, we have reported the results of a pilot study of the feasibility and qualitative evaluation of a novel interactive multimedia (IM) module versus a conventional PowerPoint show (PPS) module of e-learning using the same mobile device to teach emergent ORL-HNS disorders.
A convenience sample of 24 consecutive student volunteers were prospectively recruited according to accessibility and individuals willing to participate in the pilot study at an academic teaching hospital (Department of ORL-HNS, Linkou Chang Gung Memorial Hospital, Taoyuan, Taiwan) from November 23, 2016 to January 14, 2017. All of them had at least a basic level of computer literacy, and they were also introduced to the practical aspects of using tablets and applications. Blinding to the purpose of the prestudy during recruitment was maintained to minimize preparation bias. This study was approved by the institutional review board of Chang Gung Medical Foundation (No.: 105-5290C). Written informed consent was obtained from all participants. The study proposal was registered at ClinicalTrials.gov (Identifier: NCT02971735). The study flowchart following the Consolidated Standards of Reporting Trials (CONSORT) 2010 guidelines (
Emergent ORL-HNS disorders are sensitive and acute and require many consultations for the patients to receive appropriate point-of-care service and follow-up [
The Consolidated Standards of Reporting Trials flow diagram of this pilot study. GEFT: Group Embedded Figures Test; IM: interactive multimedia; ORL-HNS: otorhinolaryngology–head and neck surgery; PPS: PowerPoint show.
To design effective instructional material, we analyzed our tasks and topics and needs of 10 undergraduate students after traditional ORL-HNS lectures [
Subsequently, the content was translated into an e-learning app including a novel gamified IM module and a conventional visual-auditory text-image PPS module. We created an 80-min storyboard for each module, and both modules had the same design of user interface (
In the PPS module, we used video lectures to present the visual-auditory text-image context (multimedia learning) that was intended to reduce the cognitive load [
The content for the novel IM module was derived from and corresponded to the textbook-based learning material of the conventional PPS module. In the IM module, we used a game-based learning method to implement the instruction, in which the learners operated a character to run, jump, and interact with other characters (in a parkour style) to obtain learning materials (7 text-image slides per disorder) in the four domains of the 10 disorders (
Two investigators from the study team reviewed the instructional content of each module using the Software Evaluation Checklist [
The inclusion criteria were as follows: (1) age >20 years and (2) undergraduate medical students (defined as 3 or 4 years of medical school training [clerkship]). The exclusion criteria were (1) previous ORL-HNS training and (2) declining to participate.
User interface of the start screen contained four instructional domains, an adventure story and a review center.
A screenshot of the review center allowing the learners to review the acquired instruction materials anytime.
Screenshots of the PowerPoint Show module. Learners in this group needed to watch visual-auditory text-image videos including linearly arranged instructional slides (top, middle, bottom).
Screenshots of the interactive multimedia module. Learners in this group operated a character to run, jump, and interact with other characters (top) to obtain instructional materials (middle) and complete small game-based quizzes (bottom).
A general design of the multimedia situational tests.
Scenarios | Questions | Specifications of |
S1: Elicit history of acute otorhinolaryngology–head and neck surgery illness with an example picture. | Q1: Which is the most impossible diagnosis from four disorders? | Ability of remembering |
S2: Additional symptoms and signs. | Q2: Which the less likely diagnosis from three disorders? | Ability of applying |
S3: Seek critical physical findings. | Q3: Which is the most preferable diagnostic tool for further physical examination? | Ability of analyzing |
S4: Interpret key physical findings of a video. | Q4: Which is the more possible diagnosis from two disorders? | Ability of analyzing |
S5: Prescribe treatments according to the key features. | Q5: Which is the most effective solution? | Ability of evaluating |
There were six different face-to-face assessments in this study.
The 25-item Group Embedded Figures Test (GEFT) was given to the students after enrollment to assess their cognitive style [
In this study, the participants were required to complete the same MCQ pretest and another different posttest immediately after the M-TEL. A 15-min 10-question standard MCQs were used to evaluate the students’ knowledge (range, 0-100) with regard to
The participants were required to complete the same MST pretest and another different posttest immediately after the M-TEL. The MST was a variation of the key feature test for assessments of clinical reasoning ability involving knowledge and intellectual skills (range, 0-100) [
We used the global satisfaction score (GSS; range, 0-100) to measure
We used the AttrakDiff2 questionnaire to compare
We used anonymous feedback to assess
The students were unblinded after randomization. The students in the PPS group used an app on a 7-inch tablet to watch video lectures in 10 linear-designed sessions and review the instructional materials in an ordinary office environment for 100 min. Meanwhile, the IM group played a parkour-like game to find and read the instructional materials, completed small game-based quizzes, and reviewed the instructional materials.
The percentage change in MCQ score (ie. “knowledge gain”) after the M-TEL was the primary outcome measure. The percentage changes in MST (ie, “competence gain”), GSS, and AttrakDiff2 questionnaire scores were the secondary outcomes.
There were 6 students who helped to establish and evaluate the M-TEL system for emergent ORL-HNS disorders (percentage change in MCQ: mean=31, standard deviation [SD]=16, effect size=1.94; percentage change in MST: mean=45, SD=52, effect size=0.87). In this pilot feasibility study, we needed to confirm that the students could gain knowledge and competence significantly. We estimated the sample sizes by a priori calculation (one-sample Wilcoxon signed-rank test, two-tailed, normal parent distribution, alpha=.05, power=0.95) and found that we needed at least 7 subjects for knowledge gain and at least 21 subjects for competence gain. Due to a fixed block size of 6, we determined that the sample size of the pilot study was 24.
Due to the relatively small sample size in the pilot study, we analyzed all variables using a nonparametric approach. Descriptive statistics were expressed as median and 95% CI. Percentage (%) changes ([after value-before value]/[before value] × 100) in the MCQ and MST were calculated. Differences between groups were analyzed using the Wilcoxon signed-rank test or the Mann-Whitney
Twenty-four volunteers (15 males, 63%, and 9 females, 37%; median age 23 years, range 22-25 years; 21 FI, 87% and 3 FD, 13%) were recruited in the pilot study.
Overall, all participants showed a significant improvement in MCQ score (ie, “knowledge gain”; median of percentage change 71, 95% CI 14-100;
The M-TEL positively impacted the GSS (median of difference 2.5, 95% CI 1.0-4.0;
Although the percentage change in MCQ was not significantly different between the two groups (median of difference −24, 95% CI −75 to 36;
The qualitative feedback from the PPS group emphasized that they found the PPS module “easy to use and follow,” “clear layout,” “enhanced knowledge,” “suitable small sessions,” and “simulated lectures.” However, they also reported that the module was “tedious,” “hypnogenetic,” and “difficult to play back.” The IM group reported that the IM module was “fun learning (attractive),” contained “enjoyable small game-based quizzes,” and was an “amazing learning experience.” However, they also considered it “difficult to use and follow,” that it contained “nonlinear instructional materials” and “some tough games.”
Demographic data, cognitive style, learning outcomes, and satisfaction.
Variables | Overall, N=24 | Interactive multimedia group, N=12 | PowerPoint show group, N=12 | Effect size, median of difference (95% CI) or odds ratio (95% CI)a | ||
Age in years, median (95% CI) | 23 (22-23) | 23 (22-23) | 23 (22-24) | 0 (−1 to 0) | .32 | |
Male sex, n (%) | 15 (63) | 7 (58) | 8 (67) | −0.09 (−0.49 to 0.32) | >.99 | |
Group Embedded Figures Test score, median (95% CI) | 18 (17-18) | 18 (15-18) | 17 (17-18) | 0 (−1 to 1) | .80 | |
Field-dependence, n (%) | 3 (13) | 2 (17) | 1 (8) | −0.13 (−0.54 to 10.28) | >.99 | |
Multiple-choice question_before, median (95% CI) | 40 (40-50)c | 40 (40-50)c | 40 (30-60)c | 5 (−10 to 10) | .52 | |
Multiple-choice question_after, median (95% CI) | 70 (60-80)c | 70 (50-80)c | 70 (60-80)c | 0 (−10 to 10) | .71 | |
Percentage change in multiple-choice question, median (95% CI) | 71 (14-100)d | 63 (0-100) | 84 (0-125) | −24 (−75 to 36) | .55 | |
Multimedia situational test_before, median (95% CI) | 80 (60-80)c | 80 (60-100) | 70 (40-80)c | 20 (0-20) | .13 | |
Multimedia situational test_after, median (95% CI) | 80 (80-100)c | 80 (60-80) | 90 (80-100)c | −20 (−20 to 0) | .02 | |
Percentage change in multimedia situational test, median (95% CI) | 25 (0-33)d | 0 (−20 to 33) | 29 (25-75) | −41 (−67 to −20) | .008 | |
Global satisfaction score, median (95% CI) | 8 (6-9)d | 8 (7-9)d | 6 (3-8) | 2 (0-4) | .01 | |
Pragmatic quality, median (95% CI) | 1.7 (0-2.0)d | 1.8 (1.4-2.4)d | 0 (−1.0 to 2.0) | 1.7 (0.1-2.7) | .03 | |
Hedonic stimulation, median (95% CI) | 1.1 (0.3-1.9)d | 1.7 (0.9-2.3)d | −0.2 (−1.7 to 1.6) | 1.9 (0.3-3.1) | .01 | |
Hedonic identification, median (95% CI) | 1.7 (1.1-2.0)d | 2.0 (1.4-2.0)d | 1.1 (−0.6 to 2.3) | 0.8 (−0.3 to 2.3) | .18 | |
Attractiveness, median (95% CI) | 1.4 (0.9-2.1)d | 1.7 (0.9-2.1)d | 1.2 (0.4-2.1)d | 0.2 (−0.5 to 1.0) | .59 |
aEffect sizes were calculated with the use of Hodges-Lehmann method for Mann-Whiney
bMann-Whiney
c
d
Gains of knowledge and competence. There was no significant difference in multiple choice (MCQ) test scores between the interactive multimedia (IM) and PowerPoint show (PPS) groups (left). The multimedia situational test (MST) score of the IM group was significantly lower than that of the PPS group (right). Data are expressed as median (95% CI). "a" indicates significance.
Satisfaction and learning experience. Global satisfaction score (GSS) of the IM group was significantly higher than that of the PowerPoint show (PPS) group (left). Using the AttrakDiff2 questionnaire, pragmatic quality (PQ) and hedonic stimulation (HQ-S) in the IM group were significantly higher than those of the PPS group. There were no significant differences in hedonic identification (HQ-I) and attractiveness (ATT) between the two groups (right). Data are expressed as median (95% CI). "a" indicates significance.
To the best of our knowledge, this is the first study to investigate the benefits of M-TEL to improve knowledge and competence of emergent ORL-HNS disorders. Our findings indicate that using well-designed M-TEL instructional materials can help undergraduate medical students to reinforce their existing knowledge (intermediate effect) and competence (small effect) of such a sensitive and important subject and provide an enjoyable learning experience (small-to-intermediate effect). In addition, our findings suggest that an IM module has the potential to provide an instructional approach to enhance knowledge as effectively as a PPS module. Although the PPS module was superior to the IM module with regard to competence gained (small effect), the students preferred the IM module to the PPS module because of it being more efficient and enjoyable to use (small-to-intermediate effect). However, qualitative feedback recommended that both modules needed to have better quality of design and function. Since the development of the IM module was more time-consuming (3 months vs 1 months) and more expensive (US $12,500 vs US $2500) than that of the PPS module, the IM module needs to be further improved with regard to competence gain in the future. For example, we can modify the IM model according to the teaching strategies and principles of instructional design and pedagogy used in virtual patient cases to support the development of clinical reasoning skills [
Some caveats of our study merit comment. First, we included a convenience sample which may have led to exclusion bias. A more even distribution of the cognitive styles will provide more accurate data. However, FD volunteers are not frequently encountered in our undergraduate medical students (less than 10%). Moreover, it is very difficult to perform probability sampling at a regular medical school. Second, we did not investigate social interaction, self-motivation, and self-regulation (important elements of e-learning) in detail [
The role of M-TEL, especially as it pertains to the undergraduate medical student, is evolving. It is superior to classical learning in that it provides opportunities to learn outside of the classroom via the Internet and computer software [
Most previous studies have compared interactive e-learning with text-based learning or classroom lectures. Fundamental differences among these learning methods such as a learner-focused design [
Unlike traditional classroom lectures, learners can start and stop M-TEL (or text-based learning) at any time or place of their choosing [
The use of different learning strategies is one of the most important prerequisites of academic success among undergraduate medical students and can lead to a positive attitude toward learning [
CONSORT-EHEALTH checklist (V 1.6.1).
attractiveness
Consolidated Standards of Reporting Trials
field-dependent
field-independent
Group Embedded Figure Test
global satisfaction score
hedonic identification
hedonic stimulation
interactive multimedia
multiple-choice questionnaire
multimedia situational test
mobile technology in e-learning
otorhinolaryngology–head and neck surgery
PowerPoint show
pragmatic quality
undergraduate medical education
This work was financially supported by grants from the Ministry of Science and Technology, Taiwan, ROC (104-2511-S-182-010 & 105-2511-S-182A-006) and a grant from the Chang Gung Medical Foundation, Taiwan, ROC (CMRPG3C1732). The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication. We are indebted to our trained study staff, especially Chung-Fang Hsiao, Hsiu-Yen Yu, Cheng-Hao Yang, Hao-Xiang Zhang, and Ming-Xuan Chen.
LAL, SLW, YPC, HYL, and CKC participated in the conception and design of this work. LAL, YPC, MST, LJH, CJK, CHF, WCC, and HYL collected data. LAL, SLW, YPC, MST, HYL, and CKC analyzed and interpreted data. LAL, SLW, YPC, LJH, CJK, CHF, WCC, and HYL carried out the development of the project software. All authors participated in the writing of the manuscript and take public responsibility for it. LAL, SLW, YPC, CGH, HYL, and CKC revised it critically for important intellectual content. All authors reviewed the final version of the manuscript and approve it for publication. All authors attested to the validity and legitimacy of the data in the manuscript and agree to be named as author of the manuscript.
None declared.