Test & Evaluation for Autonomous Military Vehicles: How Virtual T&E Tools Enable Comprehensive Evaluations (Part 2)
Aug 15, 2022
This blog post is the second in a two-part series covering test and evaluation (T&E) considerations, approaches, and solutions for autonomous and semi-autonomous military vehicles. Read part 1 to understand how testing and evaluation challenges differ across autonomous vehicles built for commercial and defense applications and why relying exclusively on real-world testing results in significant gaps in test coverage.
Part 2 of our series outlines the Department of Defense (DOD) approach to military vehicle T&E and new challenges associated with autonomous military vehicles. It also outlines how virtual modeling and simulation tools facilitate comprehensive T&E efforts, enabling test engineers to more effectively demonstrate the performance of an autonomous military vehicle.
The DOD procurement process places each individual armed service in charge of T&E for each vehicle that they procure. The services enforce T&E within the acquisition process for each vehicle, putting safety and efficacy requirements in place that vendors must meet or exceed to proceed through the process. The DOD Director of Operational Test & Evaluation (DOT&E) is then responsible for reviewing the results of operational T&E for each major DOD acquisition program to ensure adequate testing to confirm effectiveness and suitability for combat use.
During the prototyping phases of major vehicle programs, the manufacturer will conduct shakedown safety testing and deliver a Safety Assessment Report (SAR) to the program office. This contract deliverable comprehensively describes the inherent safety hazards of the platform and imposes controls to mitigate risk.
The SAR allows test engineers from Aberdeen Test Center (ATC), under the umbrella of Army Test and Evaluation Command (ATEC), to take possession of the vehicle and conduct their own safety assessment to determine whether the system is safe for soldiers to perform operational tests. ATC will subsequently execute performance tests to evaluate contract compliance and the selection criteria for the subsequent production phase. Additionally, once soldiers are cleared to use the vehicle in training, Army Operational Test Command (OTC) will collect data on the platform as soldiers conduct maneuvers with the vehicle, further characterizing the platform’s effectiveness.
The assessment of the safety and performance of autonomous behaviors on a military vehicle pose new and uncharted challenges on both sides:
Faced with countless scenarios to test, complex and unstructured environments, and a rigid development timeline that prescribes multiple assessments of vehicle safety and performance against requirements, testers need to rely on more than just a finite number of real-world tests.
To validate the safety and effectiveness of an autonomy stack, military autonomy programs need to build a comprehensive scenario library that covers the entire operational design domain (ODD) for the intended deployment of the autonomous system. Using this library, teams can test their autonomy stack against key performance and safety benchmarks for the scenarios that could occur in the ODD.
Physics-based virtual simulation software augments real-world tests by producing a virtual environment to continuously validate the performance and safety of an autonomy stack in a secure setting (Figure 1). With nearly limitless scenarios to test, virtual testing and evaluation software enables users to systematically build coverage without relying exclusively on costly and time-consuming real-world testing. Additionally, virtual simulation enables test engineers to evaluate the performance of an autonomy stack in scenarios that would otherwise be difficult or costly to replicate in real-world testing.
As the leading developer of virtual modeling and simulation solutions for commercial and defense autonomous systems, Applied Intuition is uniquely positioned to provide the tools needed for the Army to prove out a vehicle’s autonomy stack, supporting the deployment of autonomous ground vehicles at scale. By running millions of scenario variations simultaneously, users can identify problems well before the system is deployed in the field. And, by vastly expanding the number of scenarios, modeling and simulation solutions will convey confidence upon test engineers who would otherwise be forced to rely upon a small number of tests.
In addition to tools built to facilitate and support software-in-loop, hardware-in-loop, and even real-world testing, Applied Intuition’s solutions enable test engineers to set requirements and track testing progress, enabling Army testers to identify and address gaps in coverage quickly. AV testing and optimization does not stop at deployment: Applied’s suite of virtual modeling, simulation, and testing solutions facilitate ongoing monitoring and improvement of vehicle performance and safety.
By validating the safety and efficacy of autonomy stacks in a safe and secure virtual environment, DOD test engineers can more effectively demonstrate the performance of autonomy stacks against requirements without derailing program timelines or budgets.
Get started with virtual testing: Learn more by downloading our free V&V handbook, or request a demo to see how your team can use virtual modeling and simulation to prove out your autonomy stack.
.webp)
.webp)
