Enable breadcrumbs token at /includes/pageheader.html.twig

Securing Machine Learning Requires a Sociotechnical Approach

Considering AI interactions with humans, organizational structures and strategies may help defenders mitigate risks.

Operationalizing secure machine learning starts at the human level. Shutterstock/Gorodenkoff

In a defense environment that relies on quickly transforming vast volumes of data into information and intelligence, machine learning is a key capability. Shutterstock/Dario Rigon

The technical features of artificial intelligence introduce vulnerabilities and lend the technology to adversarial use. And securely deploying artificial intelligence depends on integration into existing organizational structures. Leveraging and securing machine learning requires a sociotechnical approach.

In 2016, contestants unleashed automated bug-hunting systems in DARPA’s Cyber Grand Challenge. The systems worked autonomously for eight hours to find and patch vulnerabilities and exploit adversaries’ weaknesses. The competition was a success. Artificial intelligence (AI) and machine learning (ML), it seemed, might be the technology needed to shift the offense-defense balance in favor of the cyber defenders. But the reality is more complex.

In a defense environment relying on transforming vast volumes and new sources of data into information and intelligence at increasing speeds, ML is a key capability. ML has wide-ranging applications, including for geospatial imaging, enterprise and predictive maintenance and in cybersecurity. To support cyber defense, ML can help monitor networks for anomalies indicative of intrusions, detect malware, discover vulnerabilities with fuzzing, create dynamic honeypots and automate known tasks. Research on AI in cybersecurity has exploded over the past decade, and ML is expected to become a key technology for businesses to counter nation-state cyber attacks.

While AI can aid cyber defense, there are also offensive applications. AI can scale existing attacks such as spear phishing, find software vulnerabilities to exploit, enhance password brute force attacks, create self-learning targeted malware or create data points that will fool other AI models. Malicious use of AI could grow existing threats, (e.g., scaling impact and driving down costs of attacks), create new threats (e.g., adversarial ML attacks), and change the nature of threats (e.g., increasing the challenges of attribution, granularity and scale of targeting).

ML introduces technical characteristics, increasing the difficulty of providing security. Defenders need to be right all the time; attackers only need to be right once. With the increased complexity introduced by ML, the possible vulnerabilities multiply. ML systems are both susceptible to traditional and ML-specific vulnerabilities. The attack surface is wide, including the ML model itself, the ML implementation, software across the ML pipeline and even the hardware. A 2020 audit of Defense Department AI systems demonstrates the vulnerability of AI to traditional known attacks targeting the broader system:

“Because DoD components and contractors did not fully implement the security controls outlined in NIST SP 800-53 and NIST SP 800-171, DoD components and contractors could become victims of cyber attacks. Malicious actors could exploit vulnerabilities on the networks and systems and steal information related to some of the Nation’s most valuable AI technologies. The protection of DoD AI data and technology is critical because AI will support military logistics, missile defense systems and medical treatments for DoD personnel.”

In addition to known security vulnerabilities, ML also introduces new challenges: There are inherent weaknesses in state-of-the-art systems, not due to error, but due to the way the AI learns. Attackers can poison training data sets altering the models subsequently trained on the data, create data points that will fool ML, steal the models and reveal hidden aspects of the training data. ML security is still in its infancy, and it is possible that research on ML robustness will advance sufficiently to be able to provide security guarantees—similar to the evolution of cryptography. However, as Defense Department members commented, securing AI will require new approaches, including new test, evaluation, verification and validation procedures. And there is concern ML security may be “a serious problem with no clear technological solutions” akin to a “cat and mouse game” with defenders and attackers.

The risk extends beyond known security weaknesses and emerging ML-specific vulnerabilities: there is also risk of emergent failures—unknown unknowns. Given the complexity and tight coupling of AI, there is an increased probability of unintentional failures. “Normal accidents” should be expected.

AI can be an enabler of better security, an attack vector and a vulnerable target. From a technical perspective, the role of ML in the cyber offense-defense balance is unclear. Analysis even shows the success of competitors in the Cyber Grand Challenge was highly due to automation, not ML methods. But even if technological considerations revealed a clear advantage, they are insufficient to assess ML’s impact on the broader cyber environment.

Looking to the history of military technology, it is clear that military capability is best measured not just by an assessment of resources but by the combination of doctrine and tactics by which the force is employed. Applying this lesson to ML security necessitates looking at factors beyond technical capabilities. The technical features of ML open and close certain possibilities, but the real challenge comes when integrating the technical characteristics of ML into the broader defense ecosystem.

Operationalizing secure ML starts at the human level. Analysts and teams working with the ML system need to trust the trustworthy results and mistrust the untrustworthy ones. Enabling users to understand the level of certainty of predictions could help them know when to act on results. But some of the technical measures designed to harden ML against adversarial attacks, such as differential privacy, inherently introduce uncertainty into predictions, are difficult for users to interpret, and even with careful documentation, could introduce challenges in tracing responsibility.

There is a need for contextual consideration of the risk tolerance of the mission when tuning models’ accuracy, precision and recall. For example, in low-risk, low-budget, fast-paced scenarios, analysts may not have time to follow up on many false positives. But in high-risk scenarios, there may be a need to decrease the rate of false negatives. Even with careful tuning, humans are prone to “automation bias,” placing too much trust in a machine’s output. Successfully operationalizing ML security requires training, functional design and contextual assessment of AI system design.

Like human-computer interaction, incentive misalignment is a known challenge to cybersecurity, and ML exacerbates the challenge. There is economic pressure to keep the false positives of ML-enabled cybersecurity tools low. And there are economic disincentives against patching vulnerabilities in ML models: gathering and curating data after a data poisoning attack and retraining a model are expensive. Additionally, Defense Department goals of sharing data and models are misaligned with goals of security. Centralizing data stores creates a high-value target for data poisoning attacks; using open-source data and models risks introducing vulnerabilities; and resharing data and models creates the potential for compromised models to propagate. ML security will have to contend with competing objectives. While technological solutions, such as detection of altered data points, differential privacy, generative adversarial networks and other robustness measures, can make ML security easier, there will still be trade-offs. Careful incentive design will be required to implement security.

ML security is developing within a changed innovation ecosystem, where the private sector holds significant AI expertise, compute, data and capital. Defense Department and White House documents have echoed this reality, stating that “the U.S. private sector and academic institutions are at the forefront of modern AI advance,” and “the lion’s share of AI spending is in the private sector.” A private sector bound to stakeholders and public perception can make a precarious partner—as evidenced when Google pulled out of the Project Maven computer vision contract. New mechanisms to harness private sector innovations, such as DARPA, Intelligence Advanced Research Projects Activity (IARPA), Defense Innovation Unit Experimental (DIUx) and In-Q-Tel, are one line of effort to advance the ML security innovation needed.

There will need to be broader shifts for the Defense Department to securely internalize ML innovation. There is opportunity to leverage procurement processes to ensure security across ML system components and adapt acquisition to support the changed speed of development. Efforts by the Joint Artificial Intelligence Center, such as the Tradewind AI acquisition platform, are steps toward necessary organizational shifts. Given the high dependence on private sector providers, there is also a need to increase cybersecurity across private sector partners. Despite a rise in attacks on ML, industry members often consider adversarial ML to be “futuristic.” Supporting private sector ML security may mean increasing awareness of the risk, updating vulnerability reporting for ML and increasing information sharing on ML failures.

As ML is incorporated into the cyber environment, it is also subject to existing U.S. cyber strategies. While deterrence underpins the current U.S. cyber strategy, deterrence has noted misfits to the cyber environment. Cyber operations are difficult to control precisely and attribute. They involve many nonstate actors and happen within an environment of constant contact. They also are challenging for cost estimation (e.g., Stuxnet), and capabilities are difficult to credibly signal. AI exacerbates the misfit of deterrence to cyber. With high dependency on data, AI intensifies already high information flows. With increased complexity, speed and scale, AI may be more difficult to control; with reliance on the private sector, AI may increase the role of the private sector as defenders, targets and security actors; and with decreasing costs to use AI, there may be a proliferation of malicious use of AI to nonstate actors. A shift from deterrence to new strategies for cyber, such as persistent engagement, was already required. New strategies will need to account for the technical features, behavior, and actors characterizing new features of the cyber environment, like ML.

The impact of AI on the cyber offense-defense balance is multidimensional and still undecided. But considering how the technical features of AI interact with the human element, existing incentive and organizational structures, and strategies can help defenders prepare to mitigate the risks and realize the benefits of AI to cybersecurity.

Morgan Livingston focuses on AI policy, with past experience including the Institute for Defense Analyses Science and Technology Policy Institute, University California, Berkeley, the Stanford Institute for Human Centered AI and the Wilson Center. This writing is solely reflective of her opinions.