OBJECTIVES:

Most existing interactions with robots in space exploration are achieved through teleoperations, such as the Space Station Remote Manipulator System (SSRMS) and the Special Purpose Dexterous Manipulator (SPDM, or Dextre) for extravehicular activities (EVA) on ISS. Time delays remain one of the most challenging issues in space robot teleoperations. NASA’s Human-Automation-Robot-Interaction (HARI) evidence report clearly shows that teleoperation delays can negatively affect performance if operators do not calibrate to it. Delays cause loss of situational awareness, task halt and errors, increased workload, and long-term stress and trust issues. During future deep space exploration, problems of teleoperation delays will become more severe due to the immense distances between communicators. Existing techniques for mitigating time delays in space robot teleoperation can be categorized into supervisory control and predictive feedback. Supervisory control is a semiautonomous approach where the operator gives only high-level and intermittent commands for the remote robotic system to close the autonomous actions itself. While predictive feedback still relies on continuous manual control, but adjusts the timepoints of feedback signals (e.g., haptics) based on the predicted delays to counteract latencies.
This research takes an alternative direction to investigate methods to manipulate human subjective perception of time delays in teleoperation, and human adaptation when delay patterns are less predictable and automation is not available or reliable. The goal of this research is to test if sensory manipulation, especially forms providing virtual force cues via haptic device-generated feelings of touch and resistance (paired with delayed visual cues), can help mitigate the negative influence of teleoperation delays measured by perceived presence, neural efficiency, and task performance.
This study has the following specific aims.

• Aim 1: Perform human-subject experiments to quantify how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperations.
• Aim 2: Predict the short-term and long-term benefits and risks to the operator’s functions based on neurobehavioral evidence.

++ -- View more

APPROACH:

This experiment will enroll 30 participants (15 male, 15 female) who are either right-handed or ambidextrous and between ages 18 to 35 years. This longitudinal study will expose each subject to each combination of sensory manipulation strategies and time delay scenarios (0ms control, 250ms, 500ms, and 750ms) using a virtual reality (VR) simulator for robot teleoperations over the course of one month. The sensory manipulation strategies include:

•  Anchoring: This strategy generates haptic stimulation immediately after the operator initiates an action.
•  Synchronous: The haptic feedback is intentionally delayed to match with the delayed visual feedback.
•  Asynchronous: This strategy reflects the most likely and the most challenging time delay scenarios as there are perceivable delays between action initiation and the haptic feedback, and between haptic and visual feedback.

To measure human adaptation speed to the new sensory manipulation method, each combination will be repeated 10 times during the experiment period. Randomized Complete Block Design will be used to account for potential learning effect impacts.
Quantative human models will be developed with the data obtained via two tasks. The first task will involve simulating space teleoperation with different delay types, delay magnitudes and force sense simulation fidelity and specificity. The second task will quantify and formulate the relationship between force sense simulation and perceived presence, neural efficiency, and task performance in teleoperations.
To address Aim 1, the researchers will use the VR simulator and haptic simulations to characterize human operator space teleoperation performance. The haptic simulations will include: (1) constant tactile feedback to indicate the status of an action, (2) high frequency vibrations indicative of features of the surface the robot is interfacing with, (3) sudden haptic changes to indicate contact events, (4) increased resistance to indicate inertia and virtual weight, (5) increased vibration magnitude to indicate grasping, and (6) different levels of torque to indicate rotation. Visual cues in the VR system will include first-person and third-person camera views.
Three categories of measures will be conducted before, during and after the experiment:

• 1. Primary outcome measures – the self-reported perception of time delays; subjects execute the task twice in a trial, one with a ‘test’ delay that they compare to a second with a ‘comparison’ delay.
• 2. Secondary outcome measures – adaptation to the delays measured by as the number of trials it takes a subject to report a different delay from reversal of a previously introduced delay, workload and awareness questionnaires, and neurophysiological and behavioral data at different time points;
• 3. Success metrics – movement variability/consistency regarding speed, intensity and accuracy for analyzing speed-accuracy tradeoff at various perceived delays.

To address Aim 2, the data collected for Aim 1 will be used to evaluate human performance and operator function. Data used for these analyses will include entropy of hand and eyeball movements and pupillary dilation as a compound indicator of increased cognitive load, General Linear Modeling of hemodynamic response patterns in the pre-motor cortex as an indicators of motor coordination difficulty, neural connectivity analysis data as a measure of cognitive task engagement, task speed, task errors, self-reported workload, and self-reported perception of time delays. Taken together, these data will be used to construct individual-level human models for (1) identifying the optimal sensory manipulation strategies under various delay magnitudes and (2) prediction of human functions and task performance in time-delayed teleoperations with and without sensory manipulation.

RESULTS:

The results of this experiment will provide quantitative evidence of the benefits and potential risks of force simulation for teleoperation delays along with teleoperation control system design recommendations and methods. Lessons learned from this research will inform a new training paradigm for crewmembers and ground support to improve their ability to adapt to changing environments in future deep space exploration with adaptive and assistive sensory augmentation.

++ -- View more