Meeting DoD's Data Challenges with a Federated Data Strategy: Sponsored Content
To maintain America’s advantage over potential adversaries, the Department of Defense last year outlined a data strategy directing that military leaders must “recognize that data is a strategic asset that must be operationalized in order to provide a lethal and effective Joint Force that, combined with our network of allies and partners, sustains American influence and advances shared security and prosperity.”
In the recent DoD OCONUS (outside the continental United States) Cloud Strategy, the department outlined an “information ecosystem” required to execute on that vision across the globe: It “must include data to and from various tactical devices and mission partner environments that enable information sharing with coalition partners. … They must be available regardless of geographical location or coalition partnership.”
But the DoD has sometimes struggled with implementing a unified data strategy, hindered as much by policy, programmatic and organizational silos, as by legacy infrastructure that was never built to store the volume and variety of data now generated, and lacks the capability to route it efficiently.
To address these challenges, Deputy Defense Secretary Kathleen Hicks in June rolled out a new initiative to stage a series of “implementation experiments or exercises” that would gradually build “foundational capabilities” for data-dependent technologies like artificial intelligence, or AI, and joint all-domain command and control, or JADC2. The AI and Data Acceleration initiative, or ADA, “recognizes the challenges that DoD is facing and ... creates a concrete path forward for a mission space that has often appeared to be more rhetoric than action,” Hicks said.
Building the future force with a unified data strategy for mission
“To build the future force, DoD must unlock and operationalize data all across the department,” says Derek Strausbaugh, chief technology officer for Microsoft’s Department of Defense business, “To achieve that, they need a modern cloud infrastructure and they need to use cloud native services to catalog, manage and automate all the data discovery and the processing and transformation it has to undergo.”
The stakes could not be higher: JADC2 relies on the availability and the context of data to provide decision advantage. The military application of transformative technologies like AI and mixed reality relies on data the way conventional military power relies on ammunition.
The DoD’s data challenges are not unique, notes Strausbaugh, “Data tends to be a messy business in any enterprise. We ourselves at Microsoft—and many of our customers—have had our own data modernization journey”
“Depending on how you count it, there are hundreds or even thousands of organizations in DoD, each producing and consuming data for their own mission,” says Strausbaugh. “You can’t just impose a single set of standards top-down. The key is to be able to centrally define the objectives and then decentralize the implementation and the management.”
“This federated approach enables some centralization around the tools and techniques you’re going to employ to make sure that data is visible and available and usable across the enterprise,” says Strausbaugh.
Done right, a federated data strategy eliminates or reduces data silos, and enables teams to share data more effectively. This approach operates
on three core principles:
• Establishing federated agility through data applications—lowering the barriers to find, access and exploit data through rigorous governance, cataloguing, lineage tracking and metadata application.
• Enabling a system of intelligence through data lakes, explainable machine learning and AI models, and ML and AI ops capabilities that implement responsible and fair implementation practices.
• Building shared foundations for shared data democratization by applying Zero Trust principles to both end users, processes and machines trying to access and operate on data.
These elements enable a modernmdata platform for mission that provides analytics and insight for everyone—in the well-connected enterprise or in denied, degraded, intermittent and low-bandwith (DDIL) environments at the edge.
Meet the data at the source
“When we were wrestling with this data challenge at Microsoft,” the company found data assets were “on-prem, in multiple clouds, in software-as-a-service solutions, in private cloud environments, on third party platforms,” Strausbaugh recalls.
“To bring order from that chaos, we developed tools internally that we now offer to our customers,” says Strausbaugh. All enterprises need similar capabilities to make good use of their data: “You have to have a catalogue, because if you don’t know the data is there, you can’t use it. You have to have metadata and taxonomies because context and linkage is what makes data important. You have to have life-cycle management—if you are going to trust the data, you need to have an end-to-end understanding of where it came from. And you need analytics: You need a way to be able to monitor, measure and manage your data, and metrics for how well you are using it,” he explains.
Azure Data Catalog is an early internal tool, automating the process of data discovery, that has evolved over time to include advanced governance capabilities and became Azure Purview. Purview is a unified data governance solution that automates not just the process of data discovery, but also sensitive data classification, and provides end-to-end data lineage guarantees.
Tools like these, and the cloud infrastructure on which they run, constitute the system of intelligence—enabling data interoperability without the need for vast, unwieldy, centralized data lakes. “The reality is,” explains Strausbaugh, “you don’t actually need to move your data around. To break down those silos, what you need is the ability to converge your data in a federated fashion from all your enterprise data systems and your operational data stores. It’s a federated multitenant model based on shared infrastructure and a common set of services, that lets you operationalize the data you need.”
By not centralizing data, a federated approach also avoids another pitfall, explains Eric Brown, vice president, Microsoft Azure Global Mission Platforms, “A lot of systems produce duplicative or overlapping data. That’s just the way it is. If you centralize, you end up spending a tremendous amount of time moving, ingesting, normalizing and indexing data before you’re able to analyze. Our warfighters need capabilities that allow them to sift through data on demand, anywhere and at any classification level.”
“You need to meet the data at the source,” says Brown.
Zero trust for processing
Trust is critical for any data strategy to work and especially in the DoD, adds Vernon Weisenburg, director, Microsoft Azure Global Mission Platforms. “If warfighters are going to rely on data to make life or death decisions, it absolutely has to be reliable,” he says.
“If we’re going to deploy these technologies, whether through augmented reality goggles or human machine teaming, as envisaged in the Skyborg program, we’re going to need to trust the integrity of the data pipeline,” points out Weisenburg.
To maintain that integrity, you need a Zero Trust approach, meaning a security model that effectively adapts to the complexity of the modern environment, embraces the mobile workforce and protects people, devices, apps and data wherever they’re located.
“When people think about Zero Trust principles in relation to data, they tend to think of it in terms of data at rest, stored somewhere or data on the move, being sent somewhere,” says Weisenburg. “You also have to make data tamper-proof during processing.” Azure Data Factory provides cryptographic protection of data throughout the whole lifecycle—and at the massive scale required for DoD operations.
Accelerating advanced data workflows
Launching the ADA initiative in June, Hicks said the experiments her flyaway data teams would conduct at the 11 combatant commands will “set the stage for advanced data management platforms.” These platforms will “enable open standard architecture and the production of scalable testable and repeatable data workflows, which facilitate cross domain and cross component experimentation and development.”
With Azure as a foundation for advanced data workflows, government customers benefit from a broad range of data services that are unparalleled from ground to cloud, providing limitless scale, built-in intelligence, best-in-class security, and the freedom to gain insights from data anywhere.
For more information: Azure.com/gov