Army Uses Advanced Systems to Understand What Soldiers Know

Soldiers from the North Carolina Army National Guard 252nd Combined Arms Battalion take part in an augmented cognition system evaluation at the Aberdeen Test Center. Program officials examined the viability of a wearable sensor system to assess soldiers’ cognitive states on the battlefield.

Research into warfighter cognition could save lives in combat and apply to multiple programs.

Information overload can stop troops in their tracks. Ongoing investigations are examining how to determine when a soldier has received too much data as well as how technology can lessen the cognitive burden of service members so they can react properly in dangerous situations. Discoveries are yielding better practices for military decision makers and medical personnel, and they have applications in fields that are a far cry from researchers’ original ideas.

The U.S. Army’s augmented cognition program is carrying on work that originally began at the Defense Advanced Research Projects Agency (DARPA) (SIGNAL Magazine, July 2006). When the Army began working with DARPA, the intent of its augmented cognition program was to demonstrate capabilities for Future Force Warrior (FFW). Augmented cognition is a separate science and technology program integrated into FFW, which is an official Advanced Technology Demonstration funded by the Army to show new capabilities integrated into Future Force combat systems.

The augmented cognition work helps commanders to understand their team leaders’ loads better, so that when leaders became overloaded, information can be passed to a second in command or delivery can be postponed. In the program, soldiers are hooked up to electroencephalograph (EEG) and electrocardiograph (ECG) sensors, which monitor various body functions such as heart rate. Psychologists can analyze the information and understand how much individuals can process before workload reaches the point of overload.

As the experiments have unfolded over the years, the focus has changed from developing technology solely for transitioning to operations to collecting information during training. Officials use the sensors in training scenarios and record the events so they can review results. Team leaders can assess through data analysis the accuracy of information they receive by comparing it to a videotape replay.

According to Henry Girolamo, advanced technology manager, integration and transition, at the U.S. Army Natick Soldier Research, Development and EngineeringCenter, Natick, Massachusetts, before the augmented cognition technology can deploy, experts must decide whether it provides commanders with information that is good enough to cause them to alter the way they communicate with their team leaders.

Girolamo became involved with the program back when DARPA was managing augmented cognition work. He saw that with FFW, soldiers would be inundated with information from sources such as unmanned vehicles, sensors, radios, displays and robots. Girolamo met with officials from FFW frequently to ensure that he understood all their interfaces and to guarantee the suite of electronics coming out of augmented cognition was compatible with FFW systems. At first, program officials informed him that augmented cognition technology would be nice to have but was not a necessity. To gain more acceptance, Girolamo began to relate the work to the warfighter’s physiological status. He says he knew the augmented cognition work had to be tied in with FFW or the effort would turn into a report and be shelved.

Girolamo’s original plan to use augmented cognition to determine when soldiers were overloaded from all of the information they received from various sources was just a starting point. The technology has found applications in many areas since then. In addition to determining whether the technology works well for leadership decision-making, the physiological information gathered by the sensors can assist medical teams who want to understand the status of soldiers from remote locations.

To improve the readings from the EEG and ECGs, developers are working to create non-contact radio frequency sensors. Artificial noise from sensors reduces the validity of the information they produce. In civilian applications, such as ultrasounds, doctors ask patients to remain still and quiet to avoid the associated problems, “but you can’t tell soldiers to do that,” Girolamo says.

The training events conducted by Army augmented cognition have proved important to the program because the realistic scenarios allow officials to evaluate what the soldier understands in terms of information load. In a test done about a year ago, program experts learned about the cognitive state of the leadership of an Army team. In the same test, personnel improved the signal processing.

According to Girolamo, the augmented cognition program has become part of the whole communications network, and all of its interfaces must be compatible with the FFW system of systems. The augmented cognition program also is complementary to the medical community’s systems and works with medics to make sure both parties remain cognizant of current efforts.

Augmented cognition studies have other applications to medicine in addition to the interest in gathering physiological information on soldiers in action and in developing non-contact sensors. Girolamo shares that a liaison from Program Executive Office (PEO) Soldier contacted him about using EEG sensors in helmets to monitor brain injuries. Soldiers can suffer from brain damage as a result of nearby explosions, but with the proper instrumentation, leaders could evaluate when to remove a soldier from a situation, how many blasts cause the injuries or how to mitigate the problem.

The uses for augmented cognition have spiraled out beyond medicine just as they have surpassed the goals Girolamo had when he became involved in the program. So when the DARPA program ended, he communicated to Army officials the value of the projects and secured their support for continuing the work. As a result, other communities of interest have approached Girolamo with possible uses for augmented cognition. The U.S. Army Tank-Automotive Research, Development and Engineering Center (TARDEC) reached out to him last April for assistance with development of a new vehicle. Program officials are determining if the tasking of soldiers on the vehicle requires two or three crew members depending on the amount of autonomy given to the vehicle. Personnel involved with the development want to allow the crew to pay attention to combat operation while the vehicle drives itself or performs other functions on its own. To best accomplish that goal, they need to understand the cognitive load on a typical two-man crew during missions. Later this year, the augmented cognition program will help assess and evaluate whether a two- or three-man crew is preferable.

The Army’s augmented cognition program is studying the viability of using sensors to assess soldiers’ cognitive state. Work is underway to create non-contact sensors that will be embedded in the helmet shell instead of worn on a headband.

Another inquiry of interest made by PEO Soldier concerned improvised explosive device (IED) detection. Soldiers often see IEDs with their bare eyes, and the office is interested in determining the cognitive status of soldiers at the exact time that they see the device and when the brain recognizes the threat. The experiments start out in a laboratory setting as soldiers look for items that are visually out of character. The augmented cognition would help judge when the soldiers detect the IED and when the brain picks up the signal.

Along the same lines is a project dealing with satellite screen shots from DARPA and the National Geospatial–Intelligence Agency. Portions of sensors can detect when people could see the targets in the shots.

Much of the testing for the Army’s augmented cognition program is conducted at the U.S. Army Aberdeen Test Center, Aberdeen Proving Ground, Maryland, including all of the sensor experiments. Girolamo has built relationships with personnel there, and a contact alerts him whenever there is an experiment in which he thinks augmented cognition could be of value. The networking has paid off. In one scenario, augmented cognition personnel had the opportunity to work with a non-mobile unit that was multitasking. “That was really beneficial to us,” Girolamo shares. In the exercise, a commander had to process a lot of information in order to rescue a downed pilot. The commander was manning radios and providing guidance to a team.

Despite the amount of information sensors provide to experts in these experiments, Girolamo states that they barely interfere with troop operations. “It’s not something that soldiers are aware of because the sensors are on their system,” he says. The warfighters carry out their mission, and information on load and activity is transmitted to their commander. Soldiers put on a harness, and the sensors are inside of the helmets. To make the output more readable for commanders, it can be color-coded. Green indicates an individual is doing well; yellow demonstrates that the soldier is taking in a lot of information but can handle more; red means a warfighter is totally engaged; and blue indicates “I’m numb.”

When the augmented cognition program runs the test for the non-contact sensors program, personnel will determine if the devices are viable for training or tactical use. Girolamo wants to determine how the non-contact sensors stand up to the “gold standard” of sensors that use contact gel. The non-contact sensors will be embedded in soldiers’ helmet shells as opposed to the headbands used now.

If the Army decides this development has value, then Girolamo plans to establish a road map for the future of augmented cognition. The road map will indicate all the interest the Army has in augmented cognition as a whole. From that point, leadership will decide what sort of funding to allocate. Girolamo explains that if funding is provided, the program may offer a multiyear contract for augmented cognition work. Depending on what sort of budget the program has to work with, officials will examine the applicability of augmented cognition for tactical operations, TARDEC projects, IED detection and traumatic brain injury.

Girolamo states that one of the problems with experimentation through DARPA is the need to be fast-paced and to obtain results quickly. He says it would be beneficial in the future to replicate a few past experiments to collect more data and to evaluate the steps taken. According to Girolamo, the research psychologists involved in the work say they want to determine how the information can be trusted when observers are not watching and not videotaping the soldier. Current experiments can accommodate watching live, videotaping and then interviewing the subjects. “This is how we correlate sensor information with reality,” Girolamo explains. If researchers learn to understand the information gathered and transmitted by the sensors, then over time, users would know they could have faith in the information provided.

“I am finding that in training, [augmented cognition] does have a significant amount of value because we can start evaluating sensor output for fidelity and resolution,” Girolamo says. “We can start to trust the sensor output.” Training also allows program personnel to determine how augmented cognition can benefit other military activities that want to be involved with the program. “In fact, I think those other programs are easier than dealing with dismounted soldiers in combat,” he shares.

At Aberdeen, more physiological than cognitive information is gathered to make sure the physical health of the trainees remains in a safe range. Girolamo says a good way to judge cognitive load is to perform a study in a combat vehicle crew compartment in which soldiers are not moving but are still processing information.

Army officials are now examining technologies from FFW and deciding which provide the greatest number of capabilities to soldiers and which will most benefit the troops going forward. Those technologies will be transferred into programs through 2015. “How augmented cognition goes forward remains to be seen,” Girolamo says. “I think one thing that might bring some attention to this is just the fact that there are many other agencies that see the viability of this technology for a lot of other activities that I had never thought of before.”

Web Resources
U.S. Army Natick Soldier Research, Development and EngineeringCenter: http://nsrdec.natick.army.mil/index.htm
Future Force Warrior: www.natick.army.mil/soldier/wsit
AberdeenTestCenter: www.atc.army.mil
Defense Advanced Research Projects Agency: www.darpa.mil