When Mixed Reality and Military Collide

Mixed reality technologies, including augmented reality or virtual reality, are changing the way the U.S. Navy and Marine Corps handle system training, operations, maintenance and prototyping. Augmented reality is a combination of computer-generated images partially overlaid on a real-world view. In virtual reality, which has been around for decades, a user’s vision is obscured and totally replaced with computer-generated graphics. Both technologies visually employ degrees of computer-generated information. With the help of the Navy’s Battlespace Exploitation of Mixed Reality laboratory in San Diego, the services are looking into both types of mixed reality applications using low-cost commercial off-the-shelf technologies.

One such innovation, called the GunnAR Unified Gunnery System, uses an augmented reality helmet as part of a weapon system. GunnAR takes information from a naval gunnery liaison officer and weapon system and displays it in the helmet of a sailor on deck running the naval gun. The technology, which evolved from a sailor’s award-winning idea, improves upon the traditional way of getting the information and authorization to fire to the gunner from “decades-old radios and sound-powered phones, which are hard to hear over the din of machine guns and through required ear protection,” according to the laboratory, also known as BEMR.

GunnAR itself was created using virtual reality prototyping technologies. The ability to create anything virtually, then test it out virtually and get immediate input from warfighters, is allowing novel ideas to move to prototyping in a short amount of time with a small amount of funding, explains Heidi Buck, BEMR laboratory lead.

“So if I were to build that helmet system in the real world, I would have to do live-fire exercises. I would have to stand on the deck of a ship,” Buck says. “And you know all of this is really expensive and time-consuming, with big exercises that I would have to coordinate. Instead, I can bring the experts virtually onto the deck of the ship and have them look at different designs. They can rehearse how the gun system would work, use the compass, look at everything in their field of view and quickly redesign on the fly to improve the gunnery system, almost as if they are standing there in the real world.”

Housed in the Navy’s Space and Naval Warfare Systems Center Pacific (SSC Pacific), BEMR is considering cutting-edge mixed reality hardware and software in the gaming space and the so-called immersive technology field. BEMR is charged with exploring, evaluating and applying the technologies for Navy, Marine Corps and other Defense Department activities, as virtual reality and augmented reality work optimally in different applications. “We are focused on all that low-cost gear and how they should use it in the next five, 10, 15, 20 years,” Buck says.

Setting BEMR’s technical direction, running day-to-day operations and hosting demonstrations, among other responsibilities, Buck is one of the founders of the laboratory, which began three years ago with an infusion of funding and equipment from the Office of Naval Research. BEMR has grown quickly into a transformative tool in the Space and Naval Warfare Systems Command’s (SPAWAR’s) arsenal of command, control, communications, computers, intelligence, surveillance and reconnaissance (C4ISR) capabilities.

“You can imagine walking into an empty room and seeing how augmented reality can be used to visualize the battlespace,” Buck says. “We can control drones or a fleet. Virtually, we can create anything and display anything. And now that we have this rapid prototyping environment in virtual reality, we can prototype solutions really easily. That allows us to show these concepts to stakeholders, and it enables us to get capabilities out to the fleet a little faster.”

By helping warfighters prepare for dangerous, very expensive or infrequently performed tasks, mixed reality capabilities are proving to benefit training. BEMR employs what Navy officials call a virtual equipment environment, which provides the services with a virtual parts assembly and disassembly engine. It is used to virtualize any maintenance training course on any device. Another tool, the Aircraft Carrier Environment Simulator, allows for training of any position in the air department for specific aircraft carriers. In addition, 360-degree videos displayed in 360-degree viewers familiarize warfighters with tasks and build their experience before performing an activity. For example, the laboratory recently helped Navy SEAL teams train to dock a swimmer delivery vehicle (SDV). Using 360-degree technologies to maneuver the SDV virtually in a dry deck shelter on a submarine allowed the SEALs to train for the highly complex, dangerous effort without the risks or costs of actual SDV docking operations.

While people easily can understand virtual training, it is a little harder to grasp how mixed reality technologies can be used in system maintenance, prototyping and operations. But again, the technologies are proving useful. In particular, augmented reality works well in maintenance applications. “Let’s say you are on a ship, and there is some big piece of equipment that has failed or is broken somehow,” Buck shares. “You can put on augmented reality glasses and see information overlaid on top of it. Then, the platform can provide step-by-step instructions, like how to turn something on, take it apart and fix things. Using augmented reality is like having a virtual expert or a remote expert looking over your shoulder, guiding you through how to do a task.”

For operations, mixed reality technologies are improving warfighter targeting and performance, as the Navy has seen from the GunnAR system. “Here, augmented reality is really good because a gunner can look out at the ocean, at ships or any targets and overlay augmented reality, and in the gunner’s field of view are the firing commands, a compass and any other information the gunner may need. You can use that imaging capability anywhere on the ship. You can walk onto the bridge and see in the field of view the important information that you need to know,” Buck says.

Using virtual reality for prototyping “is really valuable to us,” she confirms. “Say, you have to build a large naval system. You can either start with a lot of engineers, a lot of design reviews and start building this giant system, or I can build it virtually in virtual reality. We can have experts virtually come on the ship and give feedback. We can look at different virtual designs of an object without physics or restrictions or costs of the stuff in the real world. In the virtual world, none of that exists. You have no boundaries in the virtual world, so we can build things very quickly and prototype things really quickly.”

BEMR’s motto now is to build in the virtual world first. Get everyone to agree on a design, obtain approval on the design, then go out and physically build the system, Buck implores.

“This offers potentially huge costs savings for the Navy because you are actually testing it, trying it out and agreeing before you even go into production,” adds Patric Petrie, SSC Pacific staff.

To support these mixed reality efforts, having the right technology is important, and to that end, BEMR is taking a look at everything, Buck acknowledges. The laboratory wants advanced gloves to track hand movements as well as gloves and vests that provide haptic feedback. It wants eye-tracking technologies. “We are going to need all sorts of hardware and equipment,” she admits. “I’m not in the business of building headsets. I don’t want to do that. So any new company with a headset is really, really interesting to me. And I want to reach out and talk to all of those companies, try out their headsets and see what the pros and cons are.”

However, for a laboratory whose mission is to harness commercially available technology, finding what will support the distinctive needs of the Navy or the Marines sometimes is a challenge. “On one hand, I look at it and think, ‘If I am just patient, the industry will develop the stuff,’ because everyone wants the better, more immersive experience,” Buck laments, noting that most mainstream consumer augmented reality or virtual reality products are not built for harsh military environments.

“The Navy has all of these unique conditions, like going from blinding light on deck of a ship to then walking into a dark room,” the laboratory lead says. “So your augmented reality headset needs to be able to adjust to that quickly and reliably. The average consumer is going to be using a headset walking around their house or down the street. In comparison, the gunner on deck of a ship is going to experience vibrations, loud noises, sea spray, weather, and the roll and pitch of a ship.”

The Navy has special agreements with industry to help pull the technology along for its military needs, including cooperative research and development agreements. BEMR has an agreement with DAQRI, an augmented reality helmet technology firm in Los Angeles that helps supply hardware for the GunnAR system. “In fact, we have gone to some of the big-name companies and talked to them about our unique issues,” Buck says.

Officials from the laboratory plan to visit virtual reality theme parks and other immersive experiences around the country to gain insights on new technologies. The growing laboratory also finds itself in the unique position of hiring artists and engineers to produce simulation artwork, 3-D models and animation. “We are working with Hollywood set designers, movie designers, game designers and other artists,” Buck notes.

However, the best information so far is coming from the throngs of warfighters visiting the laboratory. On average, BEMR has 3,000 visitors a year, and at one point, 6,000 warfighters came through for technical demonstrations. “We are getting so many visitors to the laboratory that they very easily start to see the benefits of mixed reality and how this is going to be used in the future,” Buck states. “Having all the visitors come through allows us to collect ideas from warfighters in all different services and job titles. We are able to collect those ideas, vet those ideas, prototype those ideas and improve on those ideas.”

Additionally, the laboratory’s success has quickly caught attention and support from top leaders. One admiral who visited said, “It is like Star Trek. When you are faced with problems you’ve never even considered, if someone can visualize it for you, you can come up with a solution,” Petrie says.

To Buck, it is all about BEMR “really getting our finger on the pulse of what problems there are in the Navy and other services that might be solved by this low-cost mixed reality technology.”