Clean Data First, Develop AI Later

The Department of Defense’s (DoD’s) future in artificial intelligence (AI) will come through private companies creating models and applications, not through the department’s efforts.

“80% of the demand for AI is really satisfiable by high-quality data and a good dashboard so you can see where your stuff is. So that’s the next layer, and then finally, AI is on top of that,” said Craig Martell, DoD’s outgoing chief digital and artificial intelligence officer (CDAO). “One of the key things of high-quality data is that it’s accessible, discoverable and available to you to build an app on top of it very quickly, in a way that a combatant commander needs it,” Martell said.

Martell explained that the DoD should work to provide an environment where private contractors or the services develop their own applications using the troves of data stored but unused, as much of it may be inaccessible or in the wrong format.

“I don’t care how much data you have; if no one’s using it, it’s useless. Data is only valuable if it has customers,” Martell said at a public event in Washington, D.C. Therefore, the main long-term job of the Office of the CDAO is removing all limitations that make data unusable.

“Labeling data is the part that everybody ignores, but it is the IP [intellectual property] that matters,” Martell said. “The military IP is the labeled data. It’s also the hardest thing to do, and nobody wants to do it.” Martell added.

And to underscore the importance of getting raw information into a usable state, Martell offered his own experience after three decades researching and working in this area for the private sector.

“Anybody who’s gotten a Ph.D. in this field knows that 90% of your time is labeling your damn data, or putting it in the right format, and only 10% is actually cranking out the model,” Martell said.

The next step in the progression toward data that is standardized and available widely to users is “data labeling as a service; we’re close to releasing that broadly,” Martell said.

“If you build us a model, it’s gonna have to have a container; it’s gonna have to be containerized in particular ways; it’s gonna have to have particular hooks; it doesn’t have to use particular infrastructure,” Martell offered.

Along with this work on data, other aspects need to be updated.

“Our practices date back to the 1960s when defense dollars were the main driver of American innovation and the private sector less so. The script slipped long ago with commercial tech companies now doing the lion’s share of innovation,” said Kathleen Hicks, deputy secretary of defense, at an event hosted by Martell’s office in Washington, D.C.

Hicks called for updated contracting to current standards and to keep them at the pace of technology changes.

“We need to build contracting and policy that’s agile, responsive and fast,” Martell said and added that under new contracting systems, his office brought in a new vendor in less than two weeks.

“We want to build this AI scaffolding in such a way that vendors can show up to play, understanding the rules,” Martell said. Nevertheless, those rules have not been published yet, as they are under development.

The main hurdle is that companies need to find new ways of making money.

“Let’s talk about ways that you still can make money or different ways to make money. We want this to be an open conversation. We don’t want anybody to not make money. You just can’t make money anymore locking us up; locking us up doesn’t work,” Martell said and explained that in the future, vendors offering novel AI tools to the DoD should allow the department to have a full view of the algorithm employed.

Model retraining is another key aspect the CDAO is concerned with. Once the original training of an AI model is completed, updating with new data tends to be sidelined.

“One of the things that really struck me in industry, even more so in the DoD, it’s how we ship and forget models,” Martell said.  

Still, retraining will not be enough to keep some AIs functional, given the nature of warfare.

“If the world in the past is not sufficiently similar in the world in the future, your model won’t work. If the world changes, your model won’t work, or won’t work as well,” Martell said.

Warfare is dynamic, which makes it difficult to apply AI widely.

“Assuming that the world is going to stay the same, it’s failing,” Martell said and explained how changes, like a cratered road, can upset an AI; therefore, it is wrong to keep taking a dynamic reality off the technology equation.

While many applications will be rendered useless by changes introduced by adversaries in conflict areas, some standards will have to remain unchanged to allow joint operations.

“We’re going to have some requirements. Whatever flows through the Navy, when it goes into the combatant command, it has to look like ‘this’ and you have to be able to take things back down,” Martell said, offering that data flowing toward joint commands will have to stick to standards, and whatever each service requires specifically will be free from the CDAO’s requirements.

Another concern is the relatively fixed and small amount of AI experts with skills coveted by top technology companies. The DoD cannot compete with major corporations but has a talent pool that could potentially get the job done.

“We should build to assume that we can train AI practitioners and then provide them with the right tools,” Martell said.

This means that the CDAO will seek training specialists for specific jobs that require less expertise or that, with the right tools, can produce valuable results.

Martell’s departure from the CDAO office was announced in March, with Radha Plumb taking over the role this month.