The overarching objective of this research was to refine entirely non-obtrusive objective means of detecting and mitigating cognitive performance deficits, stress, fatigue, anxiety, and depression for the operational setting of space flight, and in doing so, provide an effective method to predict, detect, and assess decrements in behavioral health and fatigue which may negatively affect performance during space flight missions. Astronauts must maintain high-level performance while experiencing demanding workload/work schedules, extreme environmental risks, and psychosocial stressors in space (e.g., isolation, confinement). Performance during space flight can be compromised by stress, negative emotions, and fatigue. This project focused on a way to detect these behavioral states during long-duration missions through the development and validation of an objective, unobtrusive, computational model-based tracker using optical computer recognition (OCR) that reliably identifies facial expressions of negative, neutral, and positive emotions, and fatigue evident in slow eyelid closures. The feasibility of the technology was tested in the NASA’s Human Exploration Research Analog (HERA) facility.
This study had the following specific aims:
- Test the technical feasibility of acquiring facial video data of astronaut-surrogates living and working in a space analog facility for OCR analyses.
- Determine the validity of OCR algorithm accuracy for detecting positive and negative facial expressions in the HERA space flight analog.
- Determine the validity of OCR algorithm accuracy for detecting oculomotor fatigue in the HERA space flight analog.
- Identify changes or improvements to the OCR system to optimize functionality.
++ -- View more
Videos were acquired using high-definition cameras located throughout the HERA module. Facial videos were analyzed from four HERA activities that tended to evoke episodically negative facial expressions: during performance on the Robotic Onboard Trainer (ROBoT); during performance on the MMSEV Flight Simulator (Flight Sim); during performance on the computerized neurobehavioral test battery; and during performance on the Trier Social Stress Test. Video footage was converted to mpeg at 60 frames per second, and run through the OCR software executable. Each crewmember's unique trained model was developed by calibrating the OCR software executable with an individual's baseline expressions pre-mission. Data were either scored based on individual-unique expressions or by using a global model of facial expressions based on population data. The executable OCR software was used to auto-score each frame for 1 of 3 facial expressions (positive, negative, or neutral).
The results of the OCR for facial affect revealed that the OCR executable software cannot reliably track the face autonomously (i.e., without human guidance and oversight) in an environment in which the background is dense with other objects, such as the electronic equipment on the walls inside HERA. The presence of inanimate background visual objects results in OCR facial tracking errors that are not found when the video background is uniformly free of different objects. It also tended to lose accuracy when the human subject frequently shifted head position, facial direction, yawned, or grimaced. Overall, the OCR executable software tracking loss rates for the videos analyzed ranged from 25% to 99%. This included lost tracking when multiple crewmembers were being tracked by OCR. Additionally, as the software currently exists, even when the facial tracking algorithm is aligned correctly on the eyebrows, eyes, mouth, and jawline, a human scorer may still be needed in the event the algorithm falsely reports positive emotion. If the problem of facial tracking in object-dense environments can be solved, the OCR technology has the potential to help detect negative affect and stress in space flight.
Basner M, Dinges DF, Mollicone DJ, Savelev I, Ecker AJ, Di Antonio A, Jones CW, Hyder E, Kan K, Morukov BV, and Sutton JP. Psychological and behavioral changes during confinement in a 520-day simulated interplanetary mission to Mars. PLoS One.
2014. March 27; 9(3):e93298. [
Yu X, Huang J, Zhang S, Yan W, and Metaxas D. Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. 2013 IEEE International Conference on Computer Vision (ICCV). 2013:1944-51. (2013 IEEE International Conference on Computer Vision (ICCV), Sydney, Australia, December 1-8, 2013) [DOI]
Crew health and performance is critical to successful human exploration beyond low Earth orbit.
The Human Research Program (HRP) investigates and mitigates the highest risks to human health
and performance, providing essential countermeasures and technologies for human space exploration.
Risks include physiological and performance effects from hazards such as radiation, altered gravity,
and hostile environments, as well as unique challenges in medical support, human factors,
and behavioral health support. The HRP utilizes an Integrated Research Plan (IRP) to identify
the approach and research activities planned to address these risks, which are assigned to specific
Elements within the program. The Human Research Roadmap is the web-based tool for communicating the IRP content.
The Human Research Roadmap is located at: https://humanresearchroadmap.nasa.gov/
+ Click here
for information of how this experiment is contributing to the HRP's path for risk reduction.