What the DoD Can Learn from Netflix and Mazda
The U.S. Department of Defense might learn a thing or two about the software-defined world from non-defense industry companies such as Netflix and Mazda, Jason Weiss, chief software officer, U.S. Defense Department, recently suggested to the AFCEA Cyber Committee.
Weiss, who serves on the committee, relayed an incident from Mazda that he said keeps him up at night. The incident was reported by BBC News in a February 10th article.
A radio station sent an image of an album cover to the HD digital radios in 2014-2017 Mazda models in the Puget Sound area near Seattle. Those drivers who were listening to KUOW, the regional National Public Radio (NPR) station, then found that they could no longer change the station away from NPR.
“They sent a picture that did not have a file extension. Instead of it being Album Cover.jpg or ArtistPhoto.png, it was just AlbumCover. Full stop,” Weiss explained. “As a result, it ended up bricking the radio systems. Literally they could not change off of that radio station.”
The fix, he noted from the BBC News article, requires a $1,500 part, which is in short supply because of computer chip supply chain troubles. “So, there's a lot of folks driving up listenership for NPR in the Seattle area right now, courtesy of a missing file extension.”
The incident, Weiss indicated, could offer lessons learned for the U.S. military. “I have to ask myself: 'What if that was an F-35 and we were sending over a map or some sort of a photo, a GPS satellite photo, and we missed the file extension?'”
He then explained a concept known as “fuzzing,” which might be described as the introduction of chaos into the software production process. “It's basically pounding the keyboard and sending in all sorts of chaos and ridiculous data. It's pulling plugs as part of chaos engineering and literally asking if systems can survive if I unplug this server without telling anybody.”
Weiss credited Netflix for making fuzzing an instrumental part of technology production. The company uses a software tool known as Chaos Monkey that randomly disables production instances. It is one tool in a family of tools with primate-related names, which are known collectively as the Simian Army.
“Netflix actually pioneered a lot of these tools with the Chaos Monkey as the first in the Simian Army to actually validate resiliency in the production environment,” Weiss noted. “They literally do this in production every day, 365 days a year, and there's a lot of lessons to be learned there.”
The concept has implications for the department’s efforts to achieve zero trust cybersecurity. “As the department aspires to grow its zero trust maturity, we're going to have to start looking at ways of doing this in production and actually understanding that we have achieved resilient systems,” Weiss offered.
However, it would not be easy for the Department of Defense (DoD) to include chaotic elements in software development, Weiss noted. The DoD has some statutory obligations and how we separate things that makes it tricky for us to go, 'Yeah, I'm gonna unplug a server in production just to see what happens,'” Weiss offered. “But we do need to think more about how we test.”
And the department can and should make some changes. “We need to be more thorough in that testing and understand those test results can't be compartmentalized, Weiss said. “They can't be put in a box where only two people get to see that we had a bad test. That needs to be socialized with the whole team and everybody needs to understand that.”
He explained that the Defense Department is still at the beginning of its journey toward a software-defined environment and is, instead, working in a much more rigid software-controlled culture in which it is too late to make changes once someone hits the “enter” key. “DoD needs to pivot from looking at things from a software-controlled perspective to a software-defined perspective.”
In a software-defined world we recognize the value of … being able to actually go through a peer review process, going through and being very intentional about our backlog and our activities from an agile perspective, and being able to actually appreciate the fact that this is programmable, which means it's repeatable. “It's deterministic and I can even roll it back if I didn't like the outcome of that in a lot of cases.”
