Description: We propose to develop MonitAR, an Augmented Reality (AR) system that provides procedure completion guidance to astronauts. MonitAR will replace guidance from mission control during periods of long time delay or when communication with Earth is not possible. Astronauts using AR glasses will receive feedback from MonitAR via visual cues as they progress through procedures on the spacecraft. The visual cues will be provided when MonitAR determines the astronaut is executing a task (a specific step in the procedure) that deviates from the current procedure. MonitAR will then guide the astronaut back to completing the task in a way that fits with the procedure. During execution, the current and upcoming tasks are proactively displayed to the astronaut in a readable form. The key innovation is to apply Case-Based Reasoning (CBR) to enable MonitAR to predict the task the astronaut is beginning to execute rather than recognize it when completed. This look-ahead capability enables guidance to be provided early enough to avoid procedure/task failure. Moreover, CBR takes advantage of the astronaut?s extensive training to capture how procedures/tasks are completed and, thereby, avoid a cumbersome and brittle modeling effort. Astronaut procedures will be represented as tasks in a plan using Action Notation Modeling Language (ANML), a planning language already being used to represent astronaut procedures. By representing the procedures as plans, the different ways a procedure can be correctly executed will be captured directly from the existing procedures.

Benefits: The initial target application for this work is to support astronaut training operations. MonitAR will enable astronauts to operate and maintain numerous systems that might unexpectedly break or provide emergency first aid without support from Earth. By adding AR-enabled look-ahead capability in monitoring procedure performance, MonitAR will provide timely guidance to keep astronauts on task.

Other potential applications for MonitAR include situations where users are under stress, required to follow procedures to accomplish certain tasks, and train to maintain proficiency in them. Examples include maintenance operations on complex systems like airplanes, ships, industrial control systems, and so on. MonitAR could also be used for providing first aid in the field when a medical doctor is not present. Potential customers include DoD's Telemedicine and Advanced Technology Research Center (TATRC), FEMA for relief/response operations, and the Center for Disease Control (CDC) for epidemic control procedures. In the private sector, companies that provide industrial control systems would find MonitAR useful in handling abnormal situations in refineries, electrical and nuclear facilities, and manufacturing settings.