Cyber Institute Probes All Dimensions of Extended Reality

The personnel within the Realities Lab at the Army Cyber Institute located at West Point explore every aspect of extended reality technologies, developing new tools, conducting studies and asking the hard questions.

The Realities Lab is dedicated to research in what is becoming known as extended reality, or XR, a term that includes augmented, virtual and mixed reality. Extended reality technologies offer a wide range of military uses, including realistic training available virtually anywhere, modeling and simulation for weapon system development, and actual situational awareness on the battlefield.

But like most technologies, computer-generated realities can be a double-edged sword, offering benefits while potentially introducing vulnerabilities. The lab is intended to explore both sides of the XR domain.

Jan Kallberg is the man tagged with asking the tough questions at the lab. He is a social scientist with a cybersecurity background and expertise in red teaming, which requires thinking like a potential enemy. “I think mainly about surprise and what can go bad. It may sound like a small thing, but when it happens, it’s pretty significant,” he says.

Kallberg raises multiple concerns. The Army could invest large sums of money on XR technologies only to be thwarted by a commercially available $15 device. Also, warfighters could be crippled by dependency on head-up displays (HUDs). “If the HUD cripple is dependent on this equipment, can they switch and go old style? You know, fighting with regular iron sights, regular combat without relying on this augmented reality support?”

Another concern is the sensors feeding data into the XR devices could be “spoofed,” causing commanders to make poor decisions or leading troops into traps. “There have to be some sensors to feed it. So, if you feed it the wrong information, things get derailed. They can throw something out, a $15 dollar device, and you turn right into an ambush,” Kallberg posits. “As an adversary, I don’t need to get into your cloud or your networks. If I have a $15 device that can just set you off so your sensor picks it up and misleads you, that’s a higher return on investment.”

Battery life for XR technologies also could be an issue, especially in extreme conditions. “Battery life in the high north, like the Arctic, is 50 percent less. So, if you have a fight in the Arctic or in Korea at high altitude, are you going to run out of batteries? Are you going to be on a cellphone asking, ‘Do you have a charger?’”

The lab also benefits from the experience of Maj. Victor Beitelman, USA, a signal officer who has deployed with special operations forces, has worked in the defense industry and has run his own company.

His specialty is viewing XR technology with a warfighter’s eye, assessing how useful the systems would be in combat scenarios. He also solves problems and equips the lab by integrating low-cost, readily available technologies, such as the Raspberry Pi.

He once built an “off-the-shelf password cracking rig” for a training exercise and submitted all the necessary how-to documentation to the U.S. Army Special Operations Command library for tactics, techniques and procedures. Maj. Beitelman also recently conducted training on integrating different platforms, a starting point for conducting Army Cyber Institute research.

One of his current projects is to build a cyber and electronic warfare package for the 82nd Airborne Division. The cyber electromagnetic activities (CEMA) jump kit combines single-board computers and software-defined radios easily available on the open market at low cost and from multiple manufacturers. It provides a limited capability for 24 to 48 hours until more capable equipment arrives. “We attach antennas to it, and it allows us to sweep the spectrum and figure out who’s on the battlefield and gain a picture of our own presence in the electromagnetic spectrum in that combat environment,” Maj. Beitelman reports.

The CEMA jump kit isn’t the only system being developed at the lab. Capt. Victor Mitsuoka, USA, a research scientist and logistician with signal, intelligence, coding and electronic warfare experience, is one of the original developers of a radio frequency visualization project aptly named RF Viz. It essentially detects radio frequency emissions and indicates on a virtual reality system where those emissions occur.

The system can detect enemy or friendly radio transmissions. If a commander sees his tactical operations center “lit up like a Christmas tree,” for example, he will know to take action. “That gives the commander a visualization that he needs to practice radio discipline. He needs to tell the soldiers to stop using their cellphones because all of that radio transmission is being used for radio direction finding, and then the opposing artillery can put rounds on that target,” Capt. Mitsuoka explains.

Capt. Mitsuoka worked on the radio detection sensor nodes and the visualization for the simulation. He also built the network to mesh the sensors together, the data collection system and the data translators to take the data from the sensors and pump it to systems such as the Microsoft HoloLens and Bohemia Interactive’s Visual Battle Simulation 3, which provides a comprehensive virtual training environment for tactical training, experimentation and mission rehearsal for land, sea and air.

“With your avatar soldier, you can see where the sensors are relative to you, and based on what they were pointing at and the direction of that radio signal, you could also triangulate where the radio signal is coming from,” he reports.

Aryn Pyke is studying whether it is better to have the RF Viz system integrated into a heads-up display or onto a tablet computer. Michael Boyce, meanwhile, continues to optimize the code, achieving significant improvements in the last two years.

Pyke’s background is in cognitive science and computer engineering. Her goal is to make the displays easy to use while identifying tradeoffs between platforms. Tablets require soldiers to look down and use their hands, both of which can be a detriment during combat. Goggles, meanwhile, reduce peripheral vision, which also is important on the battlefield. Recent versions of the HoloLens have improved, so Microsoft is addressing the issue, Pyke notes.

“One of the projects that we worked on had to do with allowing the augmented reality tool to hypothetically show you the location of invisible threats, in this case, enemy radio signal sources,” Pyke says, referring to RF Viz. “You can’t see those with the naked eye, but they can be detected by an antenna array, and that information can be superimposed in that location on your augmented reality system.”

Perhaps not too surprisingly, cadets think the goggles are cool, while more mature soldiers tend to prefer tablets. “There’s a certain familiarity with using equipment that you’ve used for several years like tablets and cellphones, and there’s sometimes a certain skepticism or discomfort with using new technologies, so part of the research is to explore what subjects’ attitudes are, and how much training or what type of training might be necessary to make subjects feel comfortable with new technology,” she explains.

Another challenge with current XR systems is that they work best when used in a familiar room. “There’s a certain calibration that’s expected, so augmented reality, as it stands, can work best in an environment where you’re always going to be using it in the same room, and there are certain benchmark locations … where the goggles can orient themselves,” Pyke states

While it can be overcome by supplementing GPS information, that need for familiar surroundings poses a challenge to using XR on the battlefield, but Pyke says the challenge is not “an insurmountable one.”

In addition to his work optimizing RF Viz code, Boyce works with the Army Simulation and Training Technology Center, Orlando, Florida, on the Cyber Battlefield Operating Simulation System, or CyberBOSS. The system, built by Dignitas Technologies LLC, facilitates cyberspace effects and actions across training systems to provide in a multidomain operations environment.

Boyce is a human factor psychologist. His work at the Army Cyber Institute focuses on cyber training for brigade or higher commanders and leaders. He brings an information operations perspective and explores technologies for best displaying readable data.

His current focus is on visualizing large-scale training events. “One of the things we are looking at is taking a lot of the existing simulation capabilities that we have, such as our augmented reality and virtual reality systems down in Orlando and making that scalable,” he offers. It’s one thing, he says to talk about 10 soldiers, another thing to talk about 10,000 with radios, cellphones and other equipment.

Boyce has helped conduct a study using the Battlespace Visualization and Interaction system. “We’ve combined with the Air Force to do multidomain operations training, so we have ground assets and air assets, and we’re trying to see how we can better understand the complex battlespace through simulation. That’s through HoloLens, so we’re looking at the effects of altitude and situational awareness on cadets’ abilities to make tactical decisions,” he says, adding that he likely will report results this summer.

Maj. Beitelman raises the possibility that future units with 3D printers will repair or even build their own XR devices, such as virtual reality headsets. “If the knowledge base is there within the Army to manufacture your own, really the sky’s the limit. As long as you have the components and a soldering iron, you can put it together,” he asserts. “Leveraging those things with the low-cost computers and systems … it’s absolutely possible for units to be able to build these things.”

The build-your-own device approach may be essential in the future, the major indicates. “Units that don’t innovate are going to have a hard time keeping up. Budgets go up and budgets go down, so if a unit knows how to organically produce things and can share, and it’s functional, I think that’s value added for the taxpayer as well as for the defense of the country.”