System Fuses Many Sensors' Data Into a Single Image

An upgrade featuring a lightweight sensor and software system added to smaller unmanned vehicles can provide capabilities similar to those available on larger vehicles. As the Defense Department realigns its operational focus to the Pacific, deployable forces, such as special operations teams and Marine Corps expeditionary forces, need high-quality airborne intelligence, surveillance and reconnaissance. While such capabilities are normally provided by unmanned aircraft such as the MQ-1 Predator, smaller platforms have to meet this need for troops operating in remote areas. These smaller vehicles require fewer personnel and are easier to maintain, but they are also less capable.

The new Multi-INT sensor system uses big data analysis and processing techniques to fuse radar and signals intelligence information with the aircraft’s resident optical sensor. The resulting single screen image is available immediately to an operator, explains John Kelly, director of intelligence, surveillance and reconnaissance (ISR) futures at BAE Systems, which developed the system. Smaller types of tactical unmanned aerial systems (UAS), such as the Army’s RQ-7 Shadow, can be upgraded quickly, he adds.

At the heart of the Multi-INT system’s hardware is small, eight-pound, multifunction, all-weather synthetic aperture radar (SAR) and signals intelligence (SIGINT) gathering equipment. The radar is equipped with a change detection and moving target indication system. Change detection allows the radar to register changes between individual image scans, such as vehicles or people moving, while the moving target indicator can track selected mobile targets. Kelly contends that the radar’s small size and weight were a major breakthrough.

The SIGINT suite can detect, identify and geolocate radio frequency emitters of interest. Engineers developed antennas that could fit on smaller UAS types without interference or performance issues. The antenna elements were changed from a weight and size perspective, but the most important feature was altering how those elements were spaced, Kelly says. New design and analysis tools allowed a much smaller baseline for how antenna arrays are arranged on smaller aircraft. “There’s been a number of things over the last 10 to 15 years that have led to the ability to do this—from hardware to algorithms to design tools,” he says. Depending on the available datalink, or the size of the platform, Kelly notes, sensor data can be scaled to be processed in the air, on the ground or a combination of the two. For some types of data, such as SAR imagery, it is easier to process it in the air than to transmit raw data to the ground, Kelly says. The system uses a process called “decimation,” which is an image compression capability that strips out a degree of extraneous data in a manner similar to how JPEG images are compressed. If necessary, most of the processing and geolocation can be done in the air, and the results can be transmitted to the ground, allowing a smaller datalink to be used, he adds.

To increase the airborne processing power of the system’s sensors, Multi-INT combines commercially available microchips and graphics processing unit (GPU) cores. The processors and GPUs can be run together in parallel to provide additional processing muscle, Kelly explains. For example, depending on a UAS’s size and power needs, a five-pound quad-core processor with a central processing unit and GPUs can be installed. Such flexibility allows engineers to scale the amount of processing to suit a platform’s size, weight and power. “Ten years ago it would have been a supercomputer, and [now] it weighs less than five pounds and costs less than $20,000,” he says.

Installation of the sensor hardware is dependent on the type of aircraft. For example, in the Shadow, the radar and SIGINT systems are installed in pods, while in the RQ-21 Blackjack they are installed in an internal payload bay. These sensors have to be mechanically and electrically integrated into the aircraft. Engineers already have worked out the sensor data processing and data fusion processes—what Kelly describes as the “heavy lifting.” What remains is platform-specific component integration, he says.

On the ground, the software used in the control stations must be formatted in some type of open standard for interoperability and ease of use, Kelly says. Industry and NATO-defined mapping and interface formats are used for all of the data streams.

The software interoperates with the platform’s ground station, which is augmented with one to three ruggedized laptops, depending on user needs, Kelly says. For example, he notes, for a recent customer the company installed two laptops—one for each of the UAS’s payloads—and a 32-inch touch screen for the fused Multi-INT map data, in the aircraft’s command shelter.

The fused data is in what Kelly refers to as the presentation layer—the information is already processed and formatted to be readily displayed for the operators to use.

Once the data is on the ground, Multi-INT uses big data tools to process it. The first step is to put the raw data into a format that is easy for the system to manage and direct. A tracking algorithm allows up to three individual tracks to be run from each of the sensors. This allows each sensor to track a different target or to combine to follow a single one.

This combined data, in the form of brief “tracklets,” is then fused into a single, longer track. This process can be done without any intervention by the operator, Kelly says. From the operator’s perspective, all of the information is combined on the map display. For example, if the operator is tracking a vehicle moving down a road, the information can also include video, radar and moving target indicator data, which appears as a dot on the screen.

After tracking and fusion, cross-cueing occurs through a sensor resource manager. This algorithm allows the system to move the camera automatically to view an object that the radar has detected. This is done without the operator’s control, but the system brings newly detected targets to the user’s attention for classification and consideration. Kelly notes that the system can be set for a specified level of automation, depending on how much manual control over the sensors is required. “The computer can do a lot of the scut work without involving the humans to start,” he says.

Once the Multi-INT system detects a target, it can perform a variety of functions, such as estimating its size and location. Based on this data, Kelly notes that the camera can be cross-cued to a better viewing angle, or the radar’s waveform can be refined to tune it better to detect and track a specific type of target, such as a dismounted soldier or a vehicle. “There are a lot of things you can do intelligently before you present [something] to the operator,” he says.

Work on Multi-INT began three years ago, but its origins come from more than a decade of systems development on larger UAS platforms such as the MQ-1 Predator. It is from this research and development that the company developed the data fusion, tracking and sensor resource management algorithms used in Multi-INT. Kelly maintains that during the last 10 years, industry and the Defense Department spent hundreds of millions of dollars on the combat-proven technologies that the system is based on. “We’re all done with the proof of concept,” he says.

Two major challenges were solved over the last two to three years that made Multi-INT possible, Kelly says. The first is the ability to digitize the spectrum of video, radar, SIGINT or hyperspectral systems. “We can now build sensors, pretty much at any size, weight and power, that work really well,” he says.

The next impediment solved was the datalinks. Existing datalinks on the market are small in size and weight, use little power and are very high bandwidth, which allows data to be easily captured and transported, Kelly says. The final challenge was processing the information. BAE approached the problem through tracking, fusing those different sensor tracks into a longer-term track and then applying a sensor resource manager to interpret it.

The ongoing U.S. strategic realignment also shifts military operational policies and how they spend money on new platforms. As the Marines and special forces return to a more expeditionary type of mission, there is the need for capable UAS systems with small logistics and personnel footprints, Kelly observes. “It really was an evolution from where our customers needed to go from an affordability perspective, and also from the fact that the smaller platforms were becoming much more reliable and omnipresent,” he adds.

The smaller tactical UAS systems, such as the Shadow and Blackjack, usually have a 5- to 7-foot wingspan and can carry a payload of up to 50 pounds. They are also more numerous than larger ISR platforms, with more than 1,200 Shadows in service with the Defense Department as opposed to 200 or 300 Predators, Kelly says.

BAE delivered approximately 20 Multi-INT systems by the end of 2013, but Kelly expects the number to increase significantly in 2014.