Background: The inclusion of new technologies in education has motivated the development of studies on mental workload. These technologies are now being used in the teaching and learning process. The analysis enables identification of factors intervening in this workload as well as planning of overload prevention for educational activities using these technologies.
Objective: To analyze the mental workload of an educational intervention with the Mobile Virtual Learning Object for the Assessment of Acute Pain in adults and newborns, according to the NASA Task Load Index criteria.
Methods: A methodological study with data collected from 5 nurses and 75 students, from November of 2013 to February of 2014.
Results: The highest students’ and specialists’ means were in the dimensions of “Mental demand” (57.20 ± 22.27; 51 ± 29.45) and “Performance” (58.47 ± 24.19; 73 ± 28.85). The specialists’ mental workload index was higher (50.20 ± 7.28) when compared with students’ (47.87 ± 16.85) on a scale from 0 to 100 (P=.557).
Conclusions: The instrument allowed for the assessment of mental workload after an online educational intervention with a mobile learning virtual object. An excessive overload was not identified among participants. Assessing mental workload from the use of educational technologies at the end of a task is a key to their applicability, with the aim of providing a more effective, stimulating, and long-lasting experience of the learning process.
Following the evolution of information and communications technologies (ICTs), the concept of the virtual learning object (VLO) has been emphasized in the educational scenario. It is defined as a unit that creates an educational context, is reusable in different learning contexts, and consists of an active and constructive teaching-learning strategy [- ].
The interest in research on the organization of mental workload has gradually increased given the popularization of information and communications technologies (ICTs) in the workplace. The use of technology has also increased in teaching and learning, resulting in a growing interest for understanding the flow of information, as well as the mental workload generated by the use of these technologies .
Considering the current technological advances, some issues related to the teaching and learning process need to be reviewed; additionally, there is a need to establish metrics for the effective assessment of important parameters that can affect students’ performance, such as mental workload.
Mental workload can be defined as the interaction between internal and external factors and the individual, resulting in a subjectively described experience . This is the effect that a particular demand has on the individual in terms of mental and physical effort, which can be related to the amount of processed information, as well as the effort for the performance of that particular task [ ].
It should be noted that the assessment of mental workload is subjective, namely, it is assessed from the perspective of the person performing the task, not the task itself. The assessment depends on individual characteristics that can increase or decrease this individual perception. These characteristics include the individual’s knowledge for performing the task, experience with the activity, age, education, humor, etc. [- ].
The assessment of mental workload assumes that an individual’s burnout level is directly associated with his/her ability to perform a certain activity [- ]. In this sense, the NASA Task Load Index (NASA TLX) has been more widely used, which is freely available to researchers (see ).
Although the interest in measuring mental workload as a means of understanding human-machine interfaces has been around for some years, only for the last 20 years has there been interest in the study of this effect on Web-based learning in humans [- ].
By identifying the rates of the dimensions that construct the mental workload, it can be classified as either “overloaded” or “underloaded.” Overload corresponds to a saturation of resource consumption, whereas underload results from the lack of stimulus to perform the task .
The calculation of mental workload through the NASA TLX combines the assessment rates of different dimensions and their weights, weighted according to their subjective importance for a given task . In addition, studying the factors intervening in mental workload from a given task enables the planning of overload prevention during activities performed by these individuals [ ], and can assist with the planning and developing of educational technologies.
The aim of this study was to analyze the mental workload of an educational intervention using the Mobile Virtual Learning Object for the Assessment of Acute Pain (m-OVADor) in adults and newborns, according to the NASA TLX criteria.
A methodological study  approved by the Research Ethics Committee at the Universidade Federal de Santa Catarina (UFSC) in 2012 (certificate 2456), with online data collection from November 1/2013 to February 15/2014.
A total of 170 students in the second to the eighth semesters of the undergraduate nursing course were eligible to participate in this study, of which 120 agreed to do so. The nonprobabilistic and intentional final sample, after applying the exclusion criteria, included 75 students and 5 specialist nurses, who completed all planned steps. All participants were identified by alphanumeric codes, students S1 to S75 and nurse specialists NS1 to NS5.
The inclusion criteria for students were (1) being regularly enrolled in the second to the eighth semester of the undergraduate nursing course at UFSC during the study period and (2) being available for online participation in the study during an extracurricular period. The inclusion criteria for nurse specialists were (1) minimum of a master’s degree; (2) a nurse with minimum experience of 2 years in the critical care area, neonatology/pediatrics, and/or medical-surgical clinic; (3) experience with the use or development of ICTs in the teaching-learning process in nursing; (4) having a mobile device with Internet access (cell phone, smartphone, personal digital assistant, or tablet). The exclusion criterion was missing any of the study steps.
All participants received the instructions to participate in the study via email and Facebook. The m-OVADor was accessed from their own mobile devices (see).
The technology developed for the study had 3 different simulated clinical scenarios (surgical clinic, adult intensive care, and neonatology; see), which enabled the study of different variables involved in the assessment of acute pain ( ).
Identification of the measures required for the analysis of mental workload of students and specialists (rate, weight, magnitude, and mental workload index) was performed by application of the NASA TLX instrument , according to the following protocol ( ).
The NASA TLX used in the study is a version that was translated and adapted by the authors, available for free use on the NASA website (see). Access to the electronic instrument for data collection was made available to participants through the Google Drive tool.
The application of the NASA TLX provides a subjective assessment of mental workload, namely, an assessment from the perspective of the person performing the task (self-assessment). This tool was chosen because of its ease of application, objectivity, and low cost of implementation .
The assessment of the participants’ mental workload was performed in 3 stages: assessment of weights for each dimension, assessment of rates for each dimension, and calculation of mental workload index (overload).
Initially, for the assessment of the weights for each dimension, 15 pairs of combinations between 6 assessment dimensions were presented to the participants (effort and physical demand, mental demand and effort, temporal demand and frustration, frustration and effort, performance and frustration, temporal demand and mental demand, temporal demand and effort, physical demand and temporal demand, effort and performance, frustration and mental demand, physical demand and frustration, performance and mental demand, mental demand and physical demand, performance and temporal demand, and performance and physical demand). At this stage, only 1 option should be selected in each pair, which should be the most prominent one during the proposed activity .
The frequencies assigned to each dimension were summed up and the results were expressed on a scale of 0-5. Then the rates assigned to each dimension were identified from the participants’ indication on a scale of 0-100 with 20 five-point equal intervals .
The weight of each dimension was multiplied by its respective rate, and the magnitude of mental workload was obtained. The magnitude values of the 6 dimensions were summed up, their results divided by 15, and the mental workload (overload) index was obtained .
The data were exported to electronic spreadsheets using Excel for Mac 2011 for data classification and analysis of the mental workload (overload) index. The dimensions comprising this mental workload were analyzed by descriptive (mean, minimum, maximum, standard deviation, and variance) and inferential (Student t test, analysis of variance [ANOVA] and Bonferroni) statistics. A significance level of P<.05 for a 95% confidence interval was used in this study.
Students’ and Specialist Nurses’ Characteristics
The sample (n=75) comprised young students between 19 and 32 years of age (22.65 ± 3.12 years). Most were women (92%, 69/75) and exclusively dedicated to studying (79%, 59/75); only 8% (n=6) were working as nursing technicians.
Regarding the semester of the students (n=75) at the time of data collection, the following distribution was observed: 1% (n=1) in the second semester; 27% (n=20) in the third semester; 17% (n=13) in the fourth semester; 9% (n=7) in the fifth semester; 9% (n=7) in the sixth semester; 12% (n=9) in seventh semester, and 24% (n=18) in the eighth semester.
As for the use of the Internet in their personal and academic environments (multiple-choice questions), 131 answers were obtained, distributed as follows: 46% (60/131) accessed the Internet on their laptops; 34% (45/131) on mobile phones; 13% (17/131) on desktops, and 7% (9/131) on tablets.
The results draw attention to the issue of connectivity from small-sized mobile devices, thereby confirming the global trend of popularization of these technologies among young people .
Regarding the sociodemographic characteristics of specialist nurses (n=5), 100% (n=5) were female, 80% (n=4) had a master’s degree, and 20% (n=1) had a doctorate. As for the area of experience, the participants were distributed in the following areas: education and/or providing care for pain (40%, 2/5), nursing informatics (40%, 2/5), and medical surgical (20%, 1/5). All used mobile technologies and the Internet for personal and academic reasons every day.
Mental Workload Assessment
The application of NASA TLX enabled identification of the rates, weights, and magnitudes assigned to 6 assessment dimensions, which were used to calculate the overload.and show the results obtained from the assessment performed by students and specialists, as well as the mean overload for each group, respectively.
aMean overload of the students is 47.87.
aMean overload of the specialists is 50.20.
Students and specialists obtained the highest means in these respective dimensions: “Performance” (58.47 ± 24.19, 73 ± 28.85) and “Mental Demand” (57.10 ± 22.27, 51 ± 29.45). It should be noted that the dimension “Physical Demand” (29.27 ± 27.91, 10 ± 8.66) was identified as the one that least contributed to mental workload during the educational intervention. In general, the distribution of rates assigned by the participants for each dimension had the same behavior ().
The repeated measures one-way ANOVA was used to identify statistically significant differences in the mean values of the 6 dimensions of mental workload for the group of students ().
|Assessment sources||Sum of squares||Degrees of freedom||Mean squares||Critical factor||P value|
|Factor 1 (6 dimensions)||61.053||5||13.570||29.45||<.001|
To identify the significant variables according to the ANOVA test, the Bonferroni test was applied, with no statistically significant difference in the mean values of the following pairs: “Mental Demand” and “Performance,” “Physical Demand” and “Temporal Demand,” “Physical demand” and “Frustration,” “Temporal Demand” and “Effort,” and also “Temporal Demand” and “Frustration.” For the other possible comparisons, significant differences between means were identified ().
|Dimensions||Mean||95% Confidence interval|
aEqual lowercase letters do not statistically differ according to the Bonferroni test.
The ANOVA test could not be performed for comparison of means of assessment of the instrument dimensions due to the small number of specialist nurses. With regard to the mental workload index (overload), the specialists’ mean was higher (50.20 ± 7.28) when compared with the students’ mean (47.87 ± 16.85). Among the students, 2 outliers were identified (S47 and S67), which had high overload means (90.33 and 100), moving away from the means of most students ().
The increasing inclusion of VLOs and mobile devices for teaching and learning in nursing has also been raising concerns about the assessment of mental workload that these resources can cause in their users. The assessment of these factors becomes essential to develop new directions for technological productions using these resources.
The analysis of the perceived mental workload from a given task is complex and personal, because it includes both specific characteristics of the task and the effort required for its development, which also has a direct relationship with the individual’s personal factors such as motivation, skills .
From the application of the NASA TLX instrument after an educational intervention using a VLO, we present the discussion of our results, starting with the sample characterization, followed by an analysis of individual assessment rates of the 6 dimensions of the instrument and the overload of students and specialists.
During the analysis of the students’ sociodemographic characteristics, it was found that although UFSC provides free Wi-Fi networks for students, students report that they most frequently access the Internet at their own homes 63% (70/111). This finding can be explained by the fact that the students are more focused on actual classes at the university, using other spaces to complement their online learning.
The fact that 7% (8/111) of the students access the Internet regularly during commuting also drew attention, confirming the global trend of mobility of individuals made possible by the use of mobile devices. Currently, technologies are very popular, especially for providing easy access to information, regardless of time or space .
Regarding rates, the 6 dimensions were similarly assessed by both students and specialists, with the dimensions “Performance” and “Mental Demand” standing out, as these had the highest assessment means.
The dimension “Performance” aimed at measuring how satisfied participants felt with their own level of performance for the assessment of pain using the m-OVADor. It had the highest rate of assessment, both among students (58.47 ± 24.19) and specialists (73 ± 28.85).
The assessment indicates a positive contribution of technology to learning acute pain assessment using mobile devices, suggesting it as a viable process to be used in the teaching-learning process in nursing, comparable to results of previous studies [- ].
By contrast, “Mental Demand” had the second highest rate, both among students (57.20 ± 22.27) and specialists (51 ± 29.45), indicating that the educational intervention through m-OVADor demanded a greater mental workload, especially regarding the amount of mental activity for execution (think, decide, remember, look, and search).
This may be associated with the fact that the simulation of pain assessment in clinical scenarios requires constant involvement and judgment by the individuals, and may also be due to accessing content from mobile devices.
By contrast, “Physical Demand,” defined as the amount of physical activity that the online educational intervention demands from the individual, was indicated by students (29.27 ± 27.91) and specialists (10 ± 8.66) as the dimension that contributed least to the mental workload, just as reported in a previous nursing study .
For some authors , this result is justified by the low physical demand required for the handling of small mobile devices, which are very popular nowadays. The result is justified by both the familiarity of these individuals with access to the Internet, and the use of their own mobile devices in their academic and personal daily life, which would facilitate interaction with them. In addition, there is the project presentation, which sought to simplify the presentation layout of m-OVADor, thereby facilitating navigation through clinical scenarios.
The presence of 2 outliers (S47 and S67), which were distanced from the mean of most participants, with high levels of overload from the online educational intervention, can be explained by the individual perception of these individuals at the time of data collection, occurring at the end of the school year, when all students had examinations and term papers.
It must be considered that the assessment of mental workload is considered a complex and personal function, which also involves the specific characteristics of the task, the effort for its performance, and the direct relationship between motivation, emotional state, skills, among others .
Still, workload can be influenced by the nature of the task, especially when considering that the educational intervention with m-OVADor included access to 3 different clinical scenarios, in which there were a diversity of resources (animation, short texts with definitions and/or descriptions, assessment exercises, and also the clinical judgment to determine nursing diagnoses and interventions), required for the assessment of each presented context.
One should also consider that mental workload generated from VLOs can relate to factors such as the complexity of the individual, resources used for the development of the activity, and factors unrelated to content, such as the multimedia resources used [- ].
In this sense, one should also reflect on increasing access to mobile devices with Internet access, and the potential of these technologies when applied in the teaching-learning process in nursing.
Arguably these technologies have become popular, especially for the flexibility that allows users to quickly access information at any place or time, to expand their potential to transform individuals’ attitudes and behaviors [, ].
Specifically in the context of this study, an assessment of mental workload has indicated that the use of these technologies is feasible in higher education in nursing, and can influence the way students build their knowledge, establishing an innovative process for teaching/learning.
The NASA TLX instrument demonstrated applicability for the assessment of mental workload from the use of m-OVADor. The results enabled analysis of the sources related to students’ and specialists’ mental workload from the use of a VLO accessed from small mobile devices. The analysis revealed that the dimension “Mental demand” obtained a greater weight both from the students’ and specialists’ perceptions, which is expected from technologies for online learning, due to the level of attention that they require. Although the study achieved the proposed aim, in no way did it intend to exhaust the possible approaches to the theme. The development of other studies to explore the theme from other perspectives is recommended, to deepen our understanding of the effects of mental workload on learning outcomes and even when planning new strategies for mobile learning in nursing education as well as for validation of the instrument in Brazil.
We thank the International Association for the Study of Pain for the grant awarded for technological production; CNPQ for providing the internship at the School of Biomedical Informatics at University of Texas Health Science Center through the SWE grant, Science Without Borders Program, and CAPES for the doctoral grant awarded during the research development.
Conflicts of Interest
Multimedia Appendix 1
Nasa Task Load Index Form.PNG File, 70KB
Multimedia Appendix 2
m-OVADor cover page.PNG File, 279KB
Multimedia Appendix 3
Clinical scenarios selection by m-OVADor.PNG File, 285KB
- Garcia PS. A Internet Como Nova Mídia na Educação. 2010 Aug 19. URL: http://www.educadores.diaadia.pr.gov.br/arquivos/File/2010/artigos_teses/EAD/NOVAMIDIA.PDF [accessed 2015-08-20] [WebCite Cache]
- Schibeci R, Lake D, Phillips R, Lowe K, Cummings R, Miller E. Evaluating the use of learning objects in Australian and New Zealand schools. Comput Educ 2008 Jan;50(1):271-283. [CrossRef]
- García-Barriocanal E, Sicilia M, Lytras M. Evaluating pedagogical classification frameworks for learning objects: A case study. Comput Hum Behav 2007 Nov;23(6):2641-2655. [CrossRef]
- Borgman C, Abelson H, Dirks L, Johson R, Koedinger KR, Linn MC, et al. Fostering Learning in the Networked World: The Cyberlearning Opportunity and Challenge. 2006 Jun 24. URL: http://www.nsf.gov/pubs/2008/nsf08204/nsf08204.pdf [accessed 2015-08-20] [WebCite Cache]
- Hart SG, Battiste V, Lester PT. POPCORN: A Supervisory Control Simulation for Workload and Performance Research. 1984 Aug 19. URL: http://ntrs.nasa.gov/search.jsp?R=19850006206 [accessed 2015-08-19] [WebCite Cache]
- Diniz R, Guimarães LBM. Avaliação da carga mental de trabalho mental. In: Guimarães LBM, editor. Ergonomia Cognitiva. Porto Alegre, Rio Grande do Sul, Brazil: FEENG; 2004.
- Corrêa FP. Carga Mental e Ergonomia. 2003. URL: https://repositorio.ufsc.br/bitstream/handle/123456789/86036/191674.pdf?sequence=1 [accessed 2015-08-20] [WebCite Cache]
- Stanton N, Hedge A, Brookhuis K, Salas E, Hendrick H. Mental workload. In: Handbook of Human Factors and Ergonomics Methods. Boca Raton, FL: CRC Press; 2005.
- Perry C, Sheik-Nainar M, Segall N, Ma R, Kaber D. Effects of physical workload on cognitive task performance and situation awareness. Theor Issues Ergon Sci 2008 Mar;9(2):95-113. [CrossRef]
- Cardoso MS, Gontijo LA. Avaliação da carga mental de trabalho e do desempenho de medidas de mensuração: NASA TLX e SWAT. Gest Prod 2012 Dec;19(4):873-884 [FREE Full text] [CrossRef]
- Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Adv Psychol 1988;52:139-183 [FREE Full text] [CrossRef]
- Hart SG. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proc Hum Fact Ergon Soc Annu Meet 2006 Oct 01;50(9):904-908. [CrossRef]
- Ayres P, Gog TV. State of the art research into cognitive load theory. Comput Hum Behav 2009 Mar;25(2):253-257. [CrossRef]
- Mayer RE, Moreno R. A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. J Educ Psychol 1998;90(2):312-320. [CrossRef]
- Sweller J, Van MJ, Paas F. Cognitive architecture and instructional design. Educ Psychol Rev 1998 Sep;10(3):251-296. [CrossRef]
- Wiebe EN, Roberts E, Behrend TS. An examination of two mental workload measurement approaches to understanding multimedia learning. Comput Hum Behav 1998;10(3):474-481. [CrossRef]
- Mayer RE. Elements of a science of e-learning. J Educ Comput Res 2003;29(3):297-313 [FREE Full text]
- Sasso GTM. Mobile learning object in CPR—Advanced cardiac life support: An application of the persuasive technology in nursing. In: Congresso Nacional de Ambientes Hipermídia para Aprendizagem. 2009 Presented at: 4º Congresso Nacional de Ambientes Hipermídia para Aprendizagem (4º Conahpa); Nov 05, 2009; Florianópolis, Brazil URL: http://wright.ava.ufsc.br/~alice/conahpa/anais/2009/
- Barra DCC. Computadorized Nursing Process and Patient Safety in Intensive Care from the ICNP Version 1. 2012. URL: https://repositorio.ufsc.br/handle/123456789/100938 [accessed 2015-09-07] [WebCite Cache]
- Amalberti R. La Conduite de systèmes à risques. In: Le Travail Humain. Paris, France: Presses Universitaires de France; 2001.
- Cao A, Chintamani KK, Pandya AK, Ellis RD. NASA TLX: Software for assessing subjective mental workload. Behav Res Methods 2009;41(1):113-117. [CrossRef] [Medline]
- Abdellah FG, Levine E. Better patient care through nursing research. Int J Nurs Stud 1965 Apr;2(1):1-12. [CrossRef]
- Human Performance Research Group (NASA Ames Research Center). NASA Task Load Index (NASA TLX): Paper-and-Pencil Package (Version 1). Washington, DC: NASA; 1986. URL: http://humansystems.arc.nasa.gov/groups/tlx/downloads/TLX_pappen_manual.pdf [accessed 2013-02-10] [WebCite Cache]
- Saccol A, Schlemmer E, Barbosa J. M-Learning e u-Learning: Novas Perspectivas da Aprendizagem Móvel e Ubíqua. São Paulo, Brazil: Pearson Prentice Hall; 2011.
- Clay CA. Exploring the use of mobile technologies for the acquisition of clinical skills. Nurse Educ Today 2011 Aug;31(6):582-586. [CrossRef] [Medline]
- Iyengar MS, Florez-Arango JF. Decreasing workload among community health workers using interactive, structured, rich-media guidelines on smartphones. Technol Health Care 2013;21(2):113-123. [CrossRef] [Medline]
- Van Merriënboer JJG, Sweller J. Cognitive load theory and complex learning: Recent developments and future directions. Educ Psychol Rev 2005 Jun;17(2):147-177. [CrossRef]
- Rowe G, Hirsh JB, Anderson AK. Positive affect increases the breadth of attentional selection. Proc Natl Acad Sci USA 2007;104(1):383-388 [FREE Full text] [CrossRef] [Medline]
- Patel VL, Arocha JF, Diermeier M, Greenes RA, Shortliffe EH. Methods of cognitive analysis to support the design and evaluation of biomedical systems: The case of clinical practice guidelines. J Biomed Inform 2001;34(1):52-66 [FREE Full text] [CrossRef] [Medline]
- Fogg BJ, Eckles D, Bogost I, Consolvo S, Holmen E, Spasojevic M, et al. Mobile Persuasion: 20 Perspectives on the Future of Behavior Change. Standford, CA: Stanford Captology Media; 2007.
- Fogg BJ. Persuasive Technology: Using Computers to Change What We Think and Do. Stanford, CA: Morgan Kaufmann; 2002:1-312.
|ANOVA: analysis of variance|
|ICTs: information and communications technologies|
|m-OVADor: Mobile Virtual Learning Object for the Assessment of Acute Pain|
|NASA TLX: NASA Task Load Index|
|VLO: virtual learning object|
|UFSC: Universidade Federal de Santa Catarina|
Edited by G Eysenbach; submitted 26.07.15; peer-reviewed by D Barra; comments to author 11.08.15; revised version received 07.09.15; accepted 22.09.15; published 06.11.15Copyright
©Ana Graziela Alvarez, Grace Sasso, Sriram Iyengar. Originally published in JMIR Medical Education (http://mededu.jmir.org), 06.11.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on http://mededu.jmir.org/, as well as this copyright and license information must be included.