Hostname: page-component-5b777bbd6c-sbgtn Total loading time: 0 Render date: 2025-06-18T13:29:37.346Z Has data issue: false hasContentIssue false
  • English
  • Français

Eye-tracking analysis to assess the mental load of unmanned aerial system operators: systematic review and future directions

Published online by Cambridge University Press:  25 November 2024

A.C. Russo Open the ORCID record for A.C. Russo [Opens in a new window] ,
M.M. Cardoso Junior Open the ORCID record for M.M. Cardoso Junior [Opens in a new window]  and
E. Villani Open the ORCID record for E. Villani [Opens in a new window]
A.C. Russo*
Affiliation:
Departamento de Engenharia de Minas e de Petróleo da Escola Politécnica da Universidade de Sao Paulo, São Paulo, Brazil
M.M. Cardoso Junior
Affiliation:
Divisão De Engenharia Mecânica-Aeronáutica, Instituto Tecnológico de Aeronáutica, São José dos Campos, Brazil
E. Villani
Affiliation:
Divisão De Engenharia Mecânica-Aeronáutica, Instituto Tecnológico de Aeronáutica, São José dos Campos, Brazil
*
Corresponding author: A.C. Russo; Email: anacarolinarusso@usp.br
Get access Rights & Permissions [Opens in a new window]

Abstract

This article presents a systematic review on the use of eye-tracking technology to assess the mental workload of unmanned aircraft system (UAS) operators. With the increasing use of unmanned aircraft in military and civilian operations, understanding the mental workload of these operators has become essential for ensuring mission effectiveness and safety. The review covered 26 studies that explored the application of eye-tracking to capture nuances of visual attention and assess cognitive load in real-time. Traditional methods such as self-assessment questionnaires, although useful, showed limitations in terms of accuracy and objectivity, highlighting the need for advanced approaches like eye-tracking. By analysing gaze patterns in simulated environments that reproduce real challenges, it was possible to identify moments of higher mental workload, areas of concentration and sources of distraction. The review also discussed strategies for managing mental workload, including adaptive design of human-machine interfaces. The analysis of the studies revealed a growing relevance and acceptance of eye-tracking as a diagnostic and analytical tool, offering guidelines for the development of interfaces and training that dynamically respond to the cognitive needs of operators. It was concluded that eye-tracking technology can significantly contribute to the optimisation of UAS operations, enhancing both the safety and efficiency of military and civilian missions.

Keywords

eye-trackingmental workloadunmanned aircraft system (UAS)UAV operator
Type
Research Article
Information
The Aeronautical Journal , Volume 129 , Issue 1333 , March 2025 , pp. 529 - 558
Copyright
© The Author(s), 2024. Published by Cambridge University Press on behalf of Royal Aeronautical Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Fricke, T. and Holzapfel, F. An approach to flight control with large time delays derived from a pulsive human control strategy, AIAA Atmospheric Flight Mechanics Conference, 2016. https://doi.org/10.2514/6.2016-1033 CrossRefGoogle Scholar
Alaimo, A., Esposito, A., Orlando, C. and Simoncini, A. Aircraft pilots workload analysis: Heart rate variability objective measures and NASA-task load index subjective evaluation, Aerospace, 2020, 7, (9). https://doi.org/10.3390/aerospace7090137 CrossRefGoogle Scholar
Hart, S.G. and Staveland, L.E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research, Human Mental Workload, 1988, pp 139–179. https://doi.org/10.1007/s10749-010-0111-6 CrossRefGoogle Scholar
NASA, FAA Improve Air Traffic Safety, Efficiency. Aviation Space Environ. Med., 2000, 71, (4), p 463. https://www.scopus.com/inward/record.uri?eid=2-s2.0-0034024167&partnerID=40&md5=9c6afb4e4543e62212561f9bab26af06 Google Scholar
National Aeronautics and Space Administration. NASA TLX: Task Load Index, 2020. Google Scholar
Yiyuan, Z., Tangwen, Y., Dayong, D. and Shan, F. Using NASA-TLX to evaluate the flight deck design in design phase of aircraft, Procedia Eng., 2011, 17, pp 77–83. https://doi.org/10.1016/j.proeng.2011.10.010 CrossRefGoogle Scholar
Zheng, Y.Y., Yin, T.W., Dong, D.Y. and Fu, S. Using NASA-TLX to evaluate the flight deck design in design phase of aircraft, in Shuguang, Z., Shuiting, D., Yanlai, Z. and Junmin, D. (Eds), 2nd International Symposium on Aircraft Airworthiness (ISAA), vol. 17, 2011. https://doi.org/10.1016/j.proeng.2011.10.010. WE - Conference Proceedings Citation Index - Science (CPCI-S).CrossRefGoogle Scholar
Lim, Y., Gardi, A., Ramasamy, S., Vince, J., Pongracic, H., Kistan, T. and Sabatini, R. A novel simulation environment for cognitive human factors engineering research, AIAA/IEEE Digital Avionics Systems Conference - Proceedings, 2017-September, 2017a. https://doi.org/10.1109/DASC.2017.8102126 CrossRefGoogle Scholar
Sibley, C., Foroughi, C., Brown, N., Drollinger, S., Phillips, H. and Coyne, J. Augmenting traditional performance analyses with eye tracking metrics, in A. H. and A. U. (Eds), Advances in Intelligent Systems and Computing: Vol. 1201 AISC, Springer, 2021a, pp 118–125. https://doi.org/10.1007/978-3-030-51041-1_17 Google Scholar
Lim, Y., Ramasamy, S., Gardi, A., Kistan, T. and Sabatini, R. Cognitive human-machine interfaces and interactions for unmanned aircraft, J. Intell. Robot. Syst. Theory Appl., 2018a, 91, (3–4), pp 755–774. https://doi.org/10.1007/s10846-017-0648-9 CrossRefGoogle Scholar
Greenhalgh, T. How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses), BMJ, 1997, 315, (7109), pp 672–675. https://doi.org/10.1136/bmj.315.7109.672 CrossRefGoogle Scholar
Kitchenham, B. and Charters, S. Guidelines for performing Systematic Literature Reviews in Software Engineering, 2007. https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf %3EGoogle Scholar
Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P. and Stewart, L.A.S. Preferred reporting items for systematic review and meta-analysis protocols (prisma-p) 2015 statement, Jpn Pharmacol. Ther., 2019, 47, (8), pp 1177–1185.Google Scholar
Fabbri, S., Silva, C., Hernandes, E., Octaviano, F., Di Thommazo, A. and Belgamo, A. Improvements in the StArt tool to better support the systematic review process, ACM International Conference Proceeding Series, 01-03-June-2016, 2016. https://doi.org/10.1145/2915970.2916013 CrossRefGoogle Scholar
McKinley, R.A., McIntire, L.K., Schmidt, R., Repperger, D.W. and Caldwell, J.A. Evaluation of eye metrics as a detector of fatigue, Hum. Factors, 2011, 53, (4), pp 403–414. https://doi.org/10.1177/0018720811411297 CrossRefGoogle ScholarPubMed
Monfort, S.S., Sibley, C.M. and Coyne, J.T. Using machine learning and real-time workload assessment in a high-fidelity UAV simulation environment, Next-Gener. Anal. IV, 2016, 9851, p 98510B. https://doi.org/10.1117/12.2219703 Google Scholar
Roy, R.N., Bovo, A., Gateau, T., Dehais, F. and Carvalho Chanel, C.P. Operator engagement during prolonged simulated UAV operation, IFAC-PapersOnLine, 2016, 49, (32), pp. 171–176. https://doi.org/10.1016/j.ifacol.2016.12.209 CrossRefGoogle Scholar
Sibley, C., Coyne, J., Avvari, G.V., Mishra, M. and Pattipati, K.R. Supporting multi-objective decision making within a supervisory control environment, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9744, 2016, pp 210–221. https://doi.org/10.1007/978-3-319-39952-2_21 CrossRefGoogle Scholar
Coyne, J.T., Sibley, C., Sherwood, S., Foroughi, C.K., Olson, T. and Vorm, E. Assessing workload with low cost eye tracking during a supervisory control task, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10284 11th, 2017, pp 139–147. https://doi.org/10.1007/978-3-319-58628-1_12 CrossRefGoogle Scholar
Sibley, C., Coyne, J. and Thomas, J. Demonstrating the supervisory control operations user testbed (SCOUT), Proc. Hum. Factors Ergon. Soc., 2016, pp 1323–1327. https://doi.org/10.1177/1541931213601306 Google Scholar
Turpin, T., Surana, A., Alicia, T. and Taylor, G.S. Removing the bottleneck: Utilizing autonomy to manage multiple UAS sensors from inside a cockpit, 2018, 22. https://doi.org/10.1117/12.2303915 CrossRefGoogle Scholar
Devlin, S.P. and Riggs, S.L. Analyzing eye tracking data using a Markovian framework to assess differences in scan patterns, Proceedings of the Human Factors and Ergonomics Society, 2017-October, 2017, pp 1814–1818. https://doi.org/10.1177/1541931213601935 CrossRefGoogle Scholar
Lim, H.-J., Choi, S.-H., Oh, J., Kim, S., Lee, J.-W., Kim, B.S., Kim, S. and Yang, J.H. Comparison study of potential workload index in a simulated multiple-UAV operation environment, 17th International Conference on Control, Automation and Systems (ICCAS), 2017. CrossRefGoogle Scholar
Jian, L., Yin, D., Shen, L. and Niu, Y. Human machine collaborative support scheduling system of intelligence information from multiple unmanned aerial vehicles based on eye tracker, J. Shanghai Jiaotong Univ. (Sci.), 2017, 22, (3), pp 322–328. https://doi.org/10.1007/s12204-017-1838-0 CrossRefGoogle Scholar
Bektaş, K., Çöltekin, A., Krüger, J., Duchowski, A.T. and Fabrikant, S.I. GeoGCD: Improved visual search via gaze-contingent display, Eye Tracking Research and Applications Symposium (ETRA), 2019. https://doi.org/10.1145/3317959.3321488 CrossRefGoogle Scholar
Foroughi, C.K., Sibley, C., Brown, N.L., Rovira, E., Pak, R. and Coyne, J.T. Detecting automation failures in a simulated supervisory control environment, Ergonomics, 2019, 62, (9), pp 1150–1161. https://doi.org/10.1080/00140139.2019.1629639 CrossRefGoogle Scholar
Devlin, S.P., Flynn, J.R. and Riggs, S.L. How shared visual attention patterns of pairs unfold over time when workload changes, Eye Tracking Research and Applications Symposium (ETRA), 2020. https://doi.org/10.1145/3379156.3391339 CrossRefGoogle Scholar
Moacdieh, N.M., Devlin, S.P., Jundi, H. and Riggs, S.L. Effects of workload and workload transitions on attention allocation in a dual-task environment: Evidence from eye tracking metrics. J. Cogn. Eng. Decision Making, 2020, 2020, (2).Google Scholar
Niu, J., Wang, C., Niu, Y. and Wang, Z. Monitoring the performance of a multi-UAV operator through eye tracking, Proceedings - 2020 Chinese Automation Congress, CAC 2020, 2020, pp 6560–6565. https://doi.org/10.1109/CAC51589.2020.9326955 CrossRefGoogle Scholar
Planke, L.J., Lim, Y., Gardi, A., Sabatini, R., Kistan, T. and Ezer, N. A cyber-physical-human system for one-to-many UAS operations: Cognitive load analysis, Sensors (Switzerland), 2020, 20, (19), pp 1–21. https://doi.org/10.3390/s20195467 CrossRefGoogle ScholarPubMed
Devlin, S.P., Byham, J.K. and Riggs, S.L. Does what we see shape history? Examining workload history as a function of performance and ambient/focal visual attention, ACM Trans. Appl. Percept., 2021, 18, (2). https://doi.org/10.1145/3449066 CrossRefGoogle Scholar
Foroughi, C.K., Brown, N.L., Sibley, C. and Coyne, J.T. Near-perfect automation: Investigating performance, trust, and visual attention allocation, Human Factors, 2021, 65, (4), pp 546–561. https://doi.org/10.1177/00187208211032889 CrossRefGoogle ScholarPubMed
Sibley, C., Foroughi, C., Brown, N., Drollinger, S., Phillips, H. and Coyne, J. Augmenting traditional performance analyses with eye tracking metrics, Advances in Intelligent Systems and Computing, 1201 AISC, 2021b, pp 118–125. https://doi.org/10.1007/978-3-030-51041-1_17 CrossRefGoogle Scholar
Singh, G., Chanel, C.P.C. and Roy, R.N. Mental workload estimation based on physiological features for pilot-UAV teaming applications, Front. Hum. Neurosci., 2021, 15. https://doi.org/10.3389/fnhum.2021.692878 CrossRefGoogle ScholarPubMed
Devlin, S.P., Brown, N.L., Drollinger, S., Sibley, C., Alami, J. and Riggs, S.L. Scan-based eye tracking measures are predictive of workload transition performance, Appl. Ergon., 2022, 105. https://doi.org/10.1016/j.apergo.2022.103829 CrossRefGoogle ScholarPubMed
Dalilian, F. and Nembhard, D. Biometrically measured affect for screen-based drone pilot skill acquisition, Int. J. Hum.-Comput. Interact., 2023. https://doi.org/10.1080/10447318.2023.2208991 Google Scholar
El Iskandarani, M., Atweh, J.A., McGarry, S.P.D., Riggs, S.L. and Moacdieh, N.M. Does it multiMatch? What scanpath comparison tells us about task performance in teams, J. Cogn. Eng. Decision Making, 2023, 17, (3), pp 294–309. https://doi.org/10.1177/15553434231171484 CrossRefGoogle Scholar
Wanyan, X., Zhuang, D. and Zhang, H. Improving pilot mental workload evaluation with combined measures, Bio-Med. Mater. Eng., 2014, 24, (6), pp 2283–2290. https://doi.org/10.3233/BME-141041 CrossRefGoogle ScholarPubMed
Gomolka, Z., Kordos, D. and Zeslawska, E. The application of flexible areas of interest to pilot mobile eye tracking, Sensors (Switzerland), 2020, 20, (4). https://doi.org/10.3390/s20040986 CrossRefGoogle ScholarPubMed
Rudi, D., Kiefer, P., Giannopoulos, I. and Raubal, M. Gaze-based interactions in the cockpit of the future: A survey, J. Multimodal User Interfaces, 2020, 14, (1), pp 25–48. https://doi.org/10.1007/s12193-019-00309-8 CrossRefGoogle Scholar
Schriver, A.T., Morrow, D.G., Wickens, C.D. and Talleur, D.A. Expertise differences in attentional strategies related to pilot decision making, Human Factors, 2008, 50, (6), pp 864–878. https://doi.org/10.1518/001872008X374974 CrossRefGoogle ScholarPubMed
Pongsakornsathien, N., Lim, Y., Gardi, A., Hilton, S., Planke, L., Sabatini, R., Kistan, T. and Ezer, N. Sensor networks for aerospace human-machine systems, Sensors (Switzerland), 2019, 19, (16). https://doi.org/10.3390/s19163465 CrossRefGoogle ScholarPubMed
Lefrançois, O., Matton, N. and Causse, M. Improving airline pilots’ visual scanning and manual flight performance through training on skilled eye gaze strategies. Safety, 2021, 7, (4). https://doi.org/10.3390/safety7040070 CrossRefGoogle Scholar
Li, J., Oksama, L. and Hyönä, J. Close coupling between eye movements and serial attentional refreshing during multiple-identity tracking, J. Cogn. Psychol., 2018, 30, (5–6), pp 609–626. https://doi.org/10.1080/20445911.2018.1476517 CrossRefGoogle Scholar
Li, W.C., Zhang, J., Le Minh, T., Cao, J. and Wang, L. Visual scan patterns reflect to human-computer interactions on processing different types of messages in the flight deck, Int. J. Indus. Ergon., 2019, 72, pp 54–60. https://doi.org/10.1016/j.ergon.2019.04.003 CrossRefGoogle Scholar
Scannella, S., Peysakhovich, V., Ehrig, F., Lepron, E. and Dehais, F. Assessment of ocular and physiological metrics to discriminate flight phases in real light aircraft, Hum. Factors, 2018, 60, (7), pp 922–935. https://doi.org/10.1177/0018720818787135 CrossRefGoogle ScholarPubMed
Yu, C.S., Wang, E.M.Y., Li, W.C., Braithwaite, G. and Greaves, M. Pilots’ visual scan patterns and attention distribution during the pursuit of a dynamic target, Aerospace Med. Hum. Perform., 2016, 87, (1), pp 40–47. https://doi.org/10.3357/AMHP.4209.2016 CrossRefGoogle ScholarPubMed
Diaz-Piedra, C., Rieiro, H., Suárez, J., Rios-Tejada, F., Catena, A. and Di Stasi, L.L. Fatigue in the military: Towards a fatigue detection test based on the saccadic velocity, Physiol. Meas., 2016, 37, (9), pp N62–N75. https://doi.org/10.1088/0967-3334/37/9/N62 CrossRefGoogle ScholarPubMed
Lounis, C., Peysakhovich, V. and Causse, M. Visual scanning strategies in the cockpit are modulated by pilots’ expertise: A flight simulator study, PLoS ONE, 2021, 16, (2). https://doi.org/10.1371/journal.pone.0247061 CrossRefGoogle ScholarPubMed
Schwerd, S. and Schulte, A. Operator state estimation to enable adaptive assistance in manned-unmanned-teaming, Cogn. Syst. Res., 2021, 67, pp 73–83. https://doi.org/10.1016/j.cogsys.2021.01.002 CrossRefGoogle Scholar