Unknown-unknowns are operational scenarios in AI enabled autonomous systems, such as human-in-the-loop (HIL) human-in-the-plant (HIP) systems, which are not accounted for and may represent a major challenge in the design and validation phase. In these scenarios, the operational behavior of the systems is not guaranteed to meet requirements such as safety and efficacy in presence of out-of-distribution human inputs and interactions between un-modeled faults and validated behaviors. Early detection of unknown-unknowns is paramount to prevent safety violations with potentially fatal consequences.
Researchers at Arizona State University have developed a novel framework for analyzing the operational output characteristics of safety-critical HIL-HIP systems to discover unknown-unknown scenarios and evaluate potential safety hazards. By utilizing a dynamics-induced hybrid recurrent neural networks (DiH-RNN), they are able to mine a physics-guided surrogate model (PGSM) which checks for deviation of the cyber-physical system (CPS) from safety-certified operational characteristics. This allows for early detection of unknown-unknowns based on the physical laws governing the system. This framework was validated by detecting the operational changes in an artificial pancreas due to unknown insulin cartridge errors.
The framework can discover unknown unknown scenarios and evaluate potential safety hazards before catastrophic events occur.
Potential Applications
Benefits and Advantages
-
Enables early detection of unknown-unknowns
-
Instead of verification, statistical model conformance strategy is performed which enables the system to detect unknown errors
-
Physics guided surrogate models are used to ensure that surrogate models learn essential internal processes of the cyber physical systems
-
Enables safety validated system for wider operating environments and scenarios
For more information about this opportunity, please see
For more information about the inventor(s) and their research, please see
