Big Data Tools Cut Through the Fog of War
A U.S. Air Force program seeks to create an analysis tool that acts like Google Earth and TiVo.
The U.S. Air Force is using big data analysis tools to create a picture of a battlefield or area of interest that can be monitored in real time as well as stored and replayed. By merging sensor streams with data tagging and trend detection software, this capability will allow analysts and warfighters to observe, track and potentially predict enemy force operations based on their observed behavior.
The goal of the Processing Exploitation and Analysis of Large Data Sets (PEALDS) program is to enable observers to sift quickly through terabytes of collected sensor data to provide warfighters with actionable information. A major challenge facing the U.S. Defense Department and large enterprises is integrating data and understanding how data collected from multiple sensors fits into a single management enterprise, where it can be analyzed and extracted, explains Frank Purdy, vice president, corporate engagement, and head of the national security services division at Logos Technologies, Fairfax, Virginia. The firm is the prime contractor working on the program with the Air Force Research Laboratory’s (AFRL’s) Information Directorate in Rome, New York.
“At its core, PEALDS is an effort to understand big data. But more importantly, how do you harvest actionable knowledge from large data in a time sequence that meets your mission needs?” Purdy offers. With PEALDS, as the data is collected, it is indexed so that it can be cross referenced in a way to meet the user’s particular needs, he adds.
Currently, the ability to search across multiple data stores representing petabytes and exabytes of data is done manually, requiring large numbers of personnel to analyze and process useful information. PEALDS is designed to automate this process as much as possible to reduce the human footprint while collecting and acting on that information faster and in more appropriate ways, Purdy says.
Much of the work currently done by analysts in the Defense Department, government and commercial industry is looking backwards to understand what has just taken place, he explains. PEALDS attempts to move from forensic to proactive activity “to predict what will happen next,” he says.
PEALDS was awarded to Logos in April 2013 as part of a five-year, $49 million, indefinite delivery, indefinite quantity contract. Because of budget constraints caused by the continuing resolution and sequestration, operating funds for the program were delayed. As a result, the program currently is in its earliest stages, Purdy notes. The initial task orders were to set up data stores, but he explains the analytic side of the contract has not happened yet, adding that there is no timeline at the moment. “We’re in the process of making that happen now,” he says.
Logos also has a Cooperative Research and Development Agreement, or CRADA, with the AFRL, which began in late 2012. The company is using its data exploitation tools and its expertise with big data analysis to work with the AFRL to understand better how wide-area motion imagery that sensors collect can help the service develop a more predictive analysis system. Purdy says this research helps both organizations better understand how wide-area motion imagery collection works; how this information can be used to create an analytical understanding of what the behaviors are in an area; how the behaviors relate to specific events; and how analysts can make more predictive analysis based on it.
Although data fusion has been around as a concept for more than 20 years, he notes, it only recently has become possible to use it operationally. PEALDS is designed to layer different intelligence/sensor data in a temporal and spatially indexed manner to allow analysts to understand better how different events in an area of interest interrelate. This capability will help warfighters understand opponents’ behavior in a battlefield. Another goal is providing enough information to help analysts predict future events. This capability is based on understanding how a particular event occurred; whom the adversaries were; and tracking those actors when and where they need tracking/surveillance, Purdy explains.
For PEALDS, Logos is applying knowledge gained over several programs, including an effort called the Vast Imagery Storage Technique, which provides algorithms and software for the three-dimensional modeling and rendering of wide area data to extract useful information.
One of Logos’ areas of expertise is activity-based analytics, which is an important component of persistent, wide-area-based imagery, Purdy says. Among the firm’s flagship programs is an aerostat-based wide-area surveillance system called Kestrel. Deployed in Afghanistan, this system collects 360-degree imagery 24 hours a day over a city-size area, storing a terabyte of data an hour. It currently stores five petabytes of searchable full-motion video data in theater, he says.
“Our wide-area motion imagery system is Google Earth with TiVo,” Purdy explains. For example, Google Earth provides a one-time snapshot of a part of the planet. Using Logos’ technology developed for Kestrel, the system allows users to observe an area in real time. However, if analysts miss a specific event, it allows them to “TiVo back in time” through the collected data to watch an event as it took place; observe the activities that led up to the incident; and see what happened after the event. He says this provides analysts with the ability to understand and begin to predict what an opponent will do next.
The combination of map-based playback and tracking data potentially makes PEALDS an extremely powerful tool for warfighters, Purdy predicts. However, other data sources must be accessed and indexed in time and space to help analysts make decisions more quickly. “It’s a large data problem. A human being can’t look at all of that data all the time,” he explains. One tool used in Kestrel that will appear in PEALDS is “watch boxes” that enable analysts to designate areas of interest in the surveillance area. An alert is sent to users the moment anything new happens or changes in the watch box, such as people or vehicles entering or leaving the area. Movement tracks for individual or groups of vehicles and people in the box also can be displayed, he adds.
The processing power of the Kestrel system and data exploitation tools are in the software being used in PEALDS. “We consider it the secret sauce,” Purdy says. The firm also is leveraging its experience in other wide-area sensor programs, including work with the Office of Naval Research on its Lightweight Expeditionary Airborne Persistent Surveillance (LEAPS) wide-area motion imaging system. Prior to LEAPS, Logos helped develop Constant Hawk—an early persistent surveillance sensor system created to help counter the deployment of improvised explosive devices in Iraq. Constant Hawk weighs 1,500 pounds and is flown on manned aircraft. For LEAPS, this 1,500-pound payload was compressed down to a 53-pound sensor and data processing payload that was installed on an Army Shadow unmanned aerial vehicle (UAV).
Military planners soon realized that a mobile tactical force would need real-time data from a Shadow UAV-size platform, so engineers created algorithms that could supply data for up to 12 users on the ground via a datalink. This capability is referred to as “the server in the sky,” Purdy relates. It allows forces on the ground to use data in real time. This information also was stored in the payload’s memory, which was then archived for forensic analysis after the UAV had landed, he says.
In addition to PEALDS, which is an Air Force-specific program, the other military services also are working on big data efforts. Purdy notes that the U.S. Army is developing a cloud-based big data extraction program; Logos is providing the activity-based analytics part of this effort. He adds that the services are moving away from stovepiped analytics to multisource, multitheater analytics systems and technologies that allow analysts to be more effective. Logos also is involved in efforts focused on real-time analytic collaboration from rear echelon organizations because the footprint of forward units is too small for this capability, Purdy explains. “The community is changing the way it thinks about analytics. That’s the intelligence community in general, the services’ intelligence components, and it easily translates to the commercial sector,” he says.