Army Cyber Researchers Develop AI Red Teaming Capabilities

As the U.S. Army transforms itself into a data-centric force enabled by artificial intelligence (AI), the service also is developing new methods for testing the security of those AI systems.

AI is expected to play a significant role in the future Army, enhancing cyber defenses, allowing human and robot teaming and enhancing next-generation command and control. To help accelerate and broaden the use of AI, including in tactical environments, the Army created a capabilities pipeline known as Project Linchpin.

In March, Army officials initiated a 100-day plan to explore ways to reduce risks associated with artificial intelligence algorithms, and they intend to follow that up with a 500-day plan beginning this summer. Linchpin is expected to be an end-to-end development and deployment environment for AI and machine learning capabilities, leveraging military intelligence data and supporting sensor requirements.

Lt. Col. Nathaniel Bastian, division chief, Data & Decision Sciences at the Army Cyber Institute located at the U.S. Military Academy West Point, described likely attacks against AI systems: manipulating training systems to change the output, poisoning the data used for a training system, and stealing AI models. To illustrate the latter example, he said that if an AI model is integrated into a drone that then gets shot down, the adversary can attempt to reverse-engineer it. “Within each of those areas, there are many different types of attacks, which exploit many different types of vulnerabilities and systems. Every system is different and data modalities are different, so being able to comprehensively do all those assessments and then train people on tools to do them does not yet exist at scale,” he said.

In addition to Linchpin, service leaders plan for the next generation of command and control (C2), which is sometimes referred to as C2Next or Next-Gen C2. It is expected to use an open architecture that is accessible with a single software application available on mobile devices or computers. C2Next capabilities will be assessed during the service’s Network Modernization Experiment in September.

One challenge with AI-enabled systems, however, is that they require new capabilities for testing their security using so-called red teaming during which experts act as adversaries seeking vulnerabilities. Col. Bastian said that new red teaming capabilities need to be developed for evaluating AI systems. “That, honestly, applies to Linchpin; that applies to everything. One of the main components is the adversarial test and evaluation, the red team, but there’s now an AI system.”

The Army Cyber Institute supports U.S. Cyber Command, Army Cyber Command, the Army Cyber Center of Excellence and West Point and receives most of its funding from other science and technology organizations, including Army Futures Command cross-functional teams, the Army Research Laboratory, the Development Command Analysis Center, and the Command, Control, Communications, Computers, Cyber, Intelligence, Surveillance and Reconnaissance Center, as well as the Office of the Undersecretary of Defense for Research and Engineering, National Security Agency, Office of Naval Research, Defense Advanced Research Projects Agency and the Intelligence Advanced Research Activity.

Additionally, the institute works with the Army program executive offices, including the Program Executive Office-Intelligence, Electronic Warfare and Sensors (PEO-IEW&S), which leads the Tactical Intelligence Targeting Access Node (TITAN) and Project Linchpin. “We previously worked with TITAN out of PEO-IEW&S to assist them with data and AI red teaming. Now, we are starting to work with Project Linchpin to help them with adversarial AI testing and evaluation. The way we’re starting to help them now is connected to their 100-day and 500-day AI implementation plan. And part of that is trying to create a framework for AI risk management, and how you quantitatively model the security risks associated with AI system vulnerabilities,” Col. Bastian reported.

“Our advisement to Project Linchpin builds upon our work for the U.S. Army C5ISR Center, where we have been developing PROTECT [Platform for Robustness and Operational Testing to Enhance Cybersecurity Technology], which is a sophisticated platform tailored to enhance the security posture of machine learning models,” Col. Bastian added. PROTECT enables users to subject their models and associated data to simulated adversarial attacks, unveiling vulnerabilities and security risks. Through comprehensive assessments and detailed reports, users gain actionable insights to fortify their models against potential threats, empowering them to proactively safeguard critical systems and data assets, he explained.

With C2Next, Col. Bastian’s team advises on the system architecture design, including network and transport elements, as well as the cybersecurity risks. “We’ve been helping lead the cybersecurity efforts of C2Next, particularly with helping develop a new assessment framework for how we’ll realistically assess C2Next, when it’s deployed someday, and how we’re going to red team the system. It’s different than classic software systems, as the C2Next capability will have data and AI components with unique vulnerabilities. Current testing exercises, playbooks, cyber tabletops, etc., don’t really exist for a future system that’s AI enabled.”

The difference with C2Next is that it will be a platform as a service, according to Col. Bastian, who added that data feeding a model could be poisoned and that the AI components generating data or making predictions have unique vulnerabilities. “With C2Next being deployed at Corps and below, as you get to a lower and lower form factor, there’s more and more adversarial risk, particularly with manipulating parts of the system.”

The Army Threat System Management Office typically performs red teaming but has never done so “from a counter-AI perspective,” the colonel explained. “Next-Gen C2 is going to have an AI major piece to it, different pieces to it. And so that’s where the modern frameworks coming out of [the National Institutes of Standards and Technology] or MITRE, defend or attack or whatever framework you want to use, don’t really address those types of unique vulnerabilities that one would want to try to find.”

He estimated it could take another couple of years to develop the necessary capabilities. “It depends. I mean, some of it already has been worked out. But in terms of a gilded system, this requires training test engineers, giving them tools to make it easy, so there’s a lot of science and technology things that need to be prioritized from a resource perspective to scale.”

Col. Bastian’s team does have some tools already available that could support AI system vulnerability assessments. For example, within the Cyber Modeling and Simulation Research Lab, the institute has a high-fidelity cyber range called Cyber Virtual Assured Network, or CyberVAN, which was provided by Peraton Labs. “It’s almost like a digital twin. It allows me to set up and configure a network configuration of varying sizes, from tactical to enterprise, and then it’s both a simulation and an emulation environment,” Col. Bastian noted. “We use that to support data generation for research, testing of capabilities, training, things like that.”

Other simulation and visualization tools in the lab include OneSAF, or semi-automated forces, for force-on-force modeling, and CyberBOSS, a technology integration tool that links OneSAF to CyberVAN, allowing more realistic network simulation and emulation.

They are also developing a capability known as ACCOLADE, which stands for Augmented Cyber Cognition with Operational Learning Automation of Deployed Expertise, an assistive automation technology to improve cyber operators’ ability to meet the increasing size, speed and complexity of tomorrow’s battlefield. The division director compared it to a cyber operators’ version of Clippy, Microsoft’s animated paper clip character that acted as an intelligent assistant for users of the Microsoft Office program from 1997 to 2003. ACCOLADE aims to “augment one’s cyber cognition through human-computer interaction data” and may also recommend the next task for a junior operator to perform. “Imagine something like that, but as an operator, now, you have this capability as part of either operations or training,” Col. Bastian said. “We also have ongoing research investigating the development of an AI-enabled security, orchestration and automated response, or SOAR, capability. You know, we can use AI for not just anomaly detection and malicious activity characterization, but also response and remediation.” Their work on autonomous cyber warfare agents uses innovative reinforcement learning techniques to develop dynamic playbooks for cybersecurity, he added.

Col. Bastian also mentioned that “C2Next is seeking to implement zero trust principles within its system architecture, particularly in terms of identity, credential and access management and the necessary tactical data tagging to enable it.” To support the effort, his team has been developing the Generative AI for ZEro-trust Labeling (GAZEL), a tool that uses fine-tuned large language models for automated data tagging to support zero trust implementation, for C5ISR’s Tactical Zero Trust program.

The Cyber Institute has a total of four laboratories. The other three are: the Intelligent Cyber-Systems and Analytics Research Lab, which aims to build, assess and deploy smart, autonomous cyber systems that enable intelligent, assured and federated decision-making; an Internet of Things Research Lab that explores the communication between smart or internet-enabled devices; and the Cognitive Security Research Lab, which develops technologies and methodologies to protect decision-making in an era of persistent social-cyber adversarial conditions and environments.