Achieving the Army's Data Imperatives at the Tactical Edge: Sponsored Content

By design, the DoD Data Strategy compels transformational change in the way data is collected, analyzed and leveraged. The mechanics may be different depending on domain or joint all-domain mission, but as referenced in a previous SIGNAL special interest editorial, the strategy’s endgame is to ensure that trusted information gets to the right destination at the right time. As the largest and oldest service at the tactical terrestrial layer of the joint force, the Army has enduring data imperatives: speed, scale and resilience. Executed diligently, these imperatives facilitate an information advantage for ground forces in garrison and in theater.  

Technology that maximizes data utility also plays a significant role as the Industrial Age Army transforms into an Information Age Army. Further, data access and sharing use cases continue to expand to the edge. For example, in a recent report, Col. Yi Se Gwon, USA, states that the way ahead for Combined Joint All-Domain Command and Control (CJADC2) and Convergence includes “adaptable technologies to get after data exchange.”  He goes on to say, “As the Army’s operational headquarters begin to develop joint interagency electronic target folders in competition, tactical formations across services will have a more comprehensive understanding of threat capabilities, vulnerabilities and capability requirements for exploitation.”

The bottom line: To “get after data exchange” and facilitate an information advantage for soldiers at the tactical edge, technology solutions must address the Army’s data imperatives individually, and holistically.

Speed to insight made simple

The Army’s first data imperative relates to speed. In the modern battlefield, Sun Tzu’s notion that “the essential factor of military success is speed” morphs into what Lt. Gen. Stephen G. Fogarty, USA, calls the golden hour of information advantage. At a recent Association of the U.S. Army event, Fogarty’s briefing noted that “whether truth or disinformation prevails in the operational environment is largely determined by one key factor—speed.” The perpetual need for speed in the military, particularly in cyber commands, is underscored by a recent Crowdstrike report that listed Russian nation-state actors, tracked as the name “Bears”, as having an average breakout time of 18:49 minutes. According to Crowdstrike, “breakout time is the critical window when an intruder compromises the first machine and when they can move laterally to other systems on the network.” In less time than it takes to watch the local news report, the Bears’ reach spread across a target’s network.

With this sort of adversarial activity occurring as fast as it does, response at speed for speed’s sake will not cut it. Speed must be combined with simple, intuitive tools that help Cyber Protection Teams (CPTs) or Mission Defense Teams (MDTs) detect, prevent and respond to threats assuredly with faster speed to insight. Tools that speed up data ingest, normalize data and humanize the user interface give CPTs and MDTs what they need to complete OODA loops faster and with greater precision. Achieving this simplified speed to insight is critical in finding hidden exploits and deterring them moving forward.

There are two schools of thought when it comes to data ingest: schema on write and schema on read. The former indexes data up front, or upon ingest, and the latter indexes data when it is used. Schema on write offers scale and performance, returning queries in milliseconds even on large amounts of historical data. Schema on read enables data to be ingested in raw form without indexing, offering flexibility for defining and executing queries in the future. These parsing strategies complement each other, and can be used in combination on various use cases. It is no longer necessary to choose one approach over the other; one or both strategies can be used as mission requirements dictate.

Additional speed and correlation accuracy can be attained by normalizing disparate data using a common schema. A common schema defines a common set of fields to be used when storing event data, such as logs and metrics. This normalization increases speed to insight in two ways:  First, a common schema makes analysts more efficient by reducing the amount of manual correlation of logs, metrics and security events from internal or third-party sources. Second, a common schema standardizes how a search engine responds. As a result, automations like machine learning, workflows and analytics remain intact.

CPTs, MDTs and others that work with data are curious in nature and crave intuitive tools that allow them to pose questions in the way that their curiosity steers them. Built-in functionality like autocomplete increases speed to insight by giving users the ability to formulate questions that are meaningful without having to know what fields are available ahead of time. Humanized user interfaces like this enable users to do what they do best, faster.

Data visibility with affordable scale

The Army’s second data imperative relates to scale. Real-time data visibility by the right person at the right time is critical, but the ability to retain and look back at older data is just as important when it comes to detecting and mitigating hidden footholds that intruders may have on a system. By no coincidence, log and metrics data retention for at least 365 days is becoming standard procedure inside and outside of the military.

Traditionally, scaling for real-time data visibility meant adding more nodes in a centralized architecture. Scaling this way has limitations because there are finite data center resources in terms of space, power and cooling, not to mention the field constraints that may arise on the tactical battlefield. Plus, this approach is often unaffordable due to the additional hardware and compute required. Similarly, maintaining access to historical data can be costly when using a centralized architecture.

Through a combination of cross cluster replication and cross cluster search, data can be sequentially replicated and indexed across remote clusters and accessed locally via search according to role-based security controls. This proven method provides speed, scale, relevance and security, and what’s more, it is affordable.

Working with older data can be costly because often back-up tapes that store data must be pulled, transported, cleaned, viewed, re-transported, and re-stored or wiped. This process is made even more antiquated by the fact that users often cannot query data, they can only view it. With frozen tier, however, data visibility is significantly expanded by storing massive amounts of data for the long haul at a much lower cost, while keeping data fully active and searchable.

Frozen tier works by using searchable snapshots to directly search data stored in the object store without any need to rehydrate it first. A local cache stores recently queried data for optimal performance on repeat searches. As a result, storage costs decrease up to 90% over hot or warm tiers and up to 80% over the cold tier.     

Architecture resilience on and off the battlefield

The Army’s third data imperative relates to resilience. Whether an internet outage occurs in garrison or devices operate offline in theater, survivability of data architecture is critical in the modern battlefield. Disconnected, intermittent and limited (DIL) environments are common in today’s operating environments. Due to the limited bandwidth, decisions are sometimes made with minimal data availability or visibility. Also, just because bandwidth is limited does not mean that adversarial techniques are deterred.  

Endpoint protection at the edge in DIL environments is a key component of resilience. Running machine learning models locally on endpoints instead of using a malware signature-based approach enables the endpoints to remain protected, whereas traditionally, disconnected endpoints would have out-of-date signatures and be at risk.  Plus, with configurable queues and distributed-by-design architecture, data and telemetry can be queued at the edge if and when network communications are down. When communications are restored, data can be seamlessly pushed from the endpoint to the cluster ensuring no data is lost due to communication issues.

In distributed DIL environments, cross cluster search can provide the data availability and visibility needed by providing query results from all other available systems. Certain remote clusters can be tagged as being more critical, ensuring that communications exist with these high value systems for the most accurate picture. By design, cross cluster search will notify users if any remote clusters are unavailable to respond to queries. Also, users can determine how long queries should take before a timeout occurs.

The modern battlefield needs agile solutions that function as well in fully connected environments as they do in DIL environments. Protection of distributed assets at the edge is possible while assuring a common operating picture through data availability and visibility.

Solutions bred in the data dimension

Taken together, the Army’s data imperatives and tactical edge requirements call for solutions bred in the data dimension with flexibility to take on new use cases in demanding, and sometimes austere environments. Armed with insight to speed up the OODA loop, the Army will prevail in any situation.

With a keen understanding of these requirements, Elastic stands ready to support our soldiers in the modern battlefield. Solutions with simple user interfaces like schema on write, schema on read, common schema, cross cluster replication, cross cluster search, frozen tier, configurable queues, and endpoint security are all available on our single technology stack, powered by search. We welcome the opportunity to demonstrate how we bring speed, scale, and resilience to your use cases.

Kevin Keeney is a Mission Support Team Leader at Elastic. Michael Young is Principal Solutions Architect at Elastic.

About Elastic
Elastic is a search company that maximizes data utility in real time. Customers worldwide use our search, observability and security stack to achieve data-dependent use cases like website search, microservice monitoring and IT/OT threat detection. Deployable on GovCloud or on premises, Elastic delivers powerful insight, no matter the mission. Contact us at army@elastic.co to schedule a demo at the AUSA 2021 Annual Meeting & Exposition in Washington, D.C.