Evaluating Driver Performance, Situation Awareness, and Cognitive Load at Different Levels of Partial Autonomy with Dynamic Task Allocation

Introduction

This is an overview of my Master's Thesis. Here you will find an abstract of the work as well as the reserach question, hypotheses, and the experiment setup used to test the hypotheses. My full thesis can be found here where you can find the results of the experiment along with the data analysis that was conducted using R-Studio.

Abstract

The current state of the art of autonomous vehicles requires operators to be vigilant enough to take over the driving at anytime while being able to perform other tasks unrelated to the driving task. It is important for this reason to know how the autonomous system affects driving performance, cognitive load, and situation awareness.

The goal of this research was to model operator situation awareness, cognitive load, driving performance, and secondary task performance at high levels of autonomy using dynamically allocated secondary tasks. Each level of partial autonomy had its own rates where the operator should have been able to respond to secondary tasks while still staying vigilant enough to take over the driving task. There was also a rate at which the secondary tasks could be rapidly presented for brief periods of time. This rapid rate was consistent among all levels of partial autonomy. The experiment consisted of participants operating a partially autonomous military vehicle while responding to secondary task questions at the designated rates. There was a three-minute familiarization phase followed by a data collection phase where participants had to prevent the autonomous vehicle from crashing into random obstacles on the road.

After the data collection phase, participants completed a brief survey. The data collection phase was scored on the factors of driving performance, objective secondary task performance based on latency and accuracy of responses to Situation Awareness Global Assessment Technique (SAGAT) questions, subjective situation awareness from the Situation Awareness Rating Technique (SART), and cognitive load based on the NASA Task Load Index (NASA-TLX) and SOS Scale. They were analyzed in terms of how they related to the autonomy level and compared to data gathered in a similar study where tasks were presented at specific rates. The data was also used to validate the model developed in that study.

Research Question

How are operator cognitive load, situation awareness, driving performance, and task performance impacted when secondary tasks are dynamically automated based on the level of autonomy?

Hypotheses

H1: Participants’ situation awareness as measured by the Situation Awareness Rating Technique (SART) as well as real-time prompts based on the Situation Awareness Global Assessment Technique (SAGAT) will be higher than what was observed in the previous experiment at equivalent autonomy levels.

H2: Participant cognitive load as measured by the NASA Task Load Index (NASA-TLX) as well as an adapted version of the SOS scale will be lower than previously measured at equivalent autonomy levels.

H3: Participants' ability to respond to sudden road events to take manual control and avoid collisions will be higher than in the previous experiment.

H4: Participants' average secondary task latency will be lower than in the previous experiment.

Experiment Setup

The study consisted of 150 participants, each randomly assigned to one of five autonomy conditions (30 for each condition), and they took part in two drives (one practice drive and one data collection drive). Participants took part in the study through a web-based driving simulator. They were placed in a simulated environment where they operated a partially autonomous closed hatch military-type vehicle with a small viewport for looking at the road and a screen for showing the secondary tasks that were in the form of simple yes or no questions about the simulated environment. Participants were responsible for safely operating the vehicle at their assigned level of autonomy while responding to secondary task questions as quickly as possible. Three road events were encountered during the data collection drive that required intervention. Participants also encountered stationary obstacles, barriers, and traffic. Participants were evaluated during the drive on driving performance and secondary task performance, which was scored by their responses to the secondary task questions. Situation awareness data was gathered via the Situation Awareness Rating Technique (SART) in a post-drive survey as well as with real-time probes for the Situation Awareness Global Assessment Technique (SAGAT) that were implemented in the secondary task. Cognitive load was measured via NASA-TLX and a modified SOS Scale in a post-drive survey.

Autonomy Levels

Each autonomy level had all the same functionality as previous conditions but added a new feature. The features specific to each autonomy level are listed below:

Simulated Environment

Below are some screenshots from the driving simulator displaying the simulated environment.

Participant View

Here you can see the view that the participants had when they did the study.

Obstacles

Here you can see the obstacles that the participants encountered.

MT Obstacle — A truck parked on the road.

Random Events

Here you can see the 6 possible random events that the participants might have encountered

Pedestrian Event — A pedestrian crossing the road.

Barrel Event — A barrel rolling across the road.

Dog Event — A dog running across the road.

MT Event — A truck driving across the road.

Trash Event — A box of trash falling onto the road.

Ladder Event — A ladder falling onto the road.