Augmented Reality Goes Airborne

A technology that harnesses augmented reality will enable the U.S. Air Force to train in the air for initial pilot qualification, dogfighting, refueling and maneuvering. The A-TARS augmented reality platform, or Airborne Tactical Augmented Reality System, developed by California-based Red 6, provides virtual opponents, such as the fifth generation Chinese J-20 and Russian Su-57, for pilot’s aerial dogfighting training runs, as well as other aerial digital assets to develop and enhance different piloting skills.

The Air Force issued a $1.5 million Small Business Innovation Research (SBIR) Phase II contract to Red 6 in November to deliver a prototype of the A-TARS system for its Pilot Training Next (PTN) program, following a successful SBIR Phase I effort that began in early 2019.

Through a patented display technology, the A-TARS system puts virtual aircraft and other visual cues into an airborne pilot’s helmet, integrated as part of a special visor. The current display system has a 105-degree field of view and is integrated on top of a standard HGU-55 helmet system. The tool also is daylight capable and supports any pilot skill level. For example, A-TARS lets a pilot “see” a digital tanker flying its route next to the real aircraft for virtual mid-air refueling, allowing the pilot to practice maneuvering and docking to receive fuel. “We are able to put virtual airplanes into the real world, into the pilot’s line of sight,” explains Red 6’s CEO, Daniel Robinson, RAF (Ret.).

The ability to train top U.S. military aviators in air-to-air combat usually requires real pilots acting as opposing aerial fighters. Robinson, the first non-U.S. pilot to fly the F-22 Raptor with the U.S. Air Force, saw firsthand the deficit of so-called red air opponents. “For the U.S. Air Force alone, it is a multibillion-dollar-a-year problem and is proving increasingly problematic to solve, given the chronic lack of front-line pilots and an aging fleet of aircraft that are no longer capable of simulating the most current threats,” he says. “Let’s say you take off in an airplane. You go for an hour or two hours. How many repetitions can you get within that two-hour cycle?” he ponders. “[With A-TARS], the efficiency within the individual sortie skyrockets, which gives the opportunity for students to get a greater look and more times going through these training scenarios, it’s just ‘practice makes perfect.’ The more and more you can expose them to that, the more and more training value you can get within the individual sortie, and that has a massive compounding effect.”

In addition, the system “can do some more radical” training scenarios, Robinson offers. “For example, when we train and we do air-to-air combat training, we always have rules of safety in place,” he says. “We’re not allowed to get within a thousand feet of each other, and we can’t take head-on gunshots because it leads to collisions and really dangerous things. But that is not how we fight for real. So how do you do that in training? With this tool you can. You can start to give these students a look at how it would actually be in real life.”

Robinson says that they are working closely with the Air Combat Command (ACC) and the Air Education and Training Command (AETC), including PTN officials, to flush out the specifics of the platform’s initial training scenarios. “What I’ve always wanted to avoid was developing A-TARS in a vacuum just by ourselves,” he states. “The idea is that we co-develop it with them, to solve their biggest pain points that they’ve got. This allows us to not just help reduce costs and increase efficiency but improve within the training cycle itself. And there’s some really interesting thoughts that are coming out of PTN about how the technology could further evolve.”

Red 6 began its technology process pitching the A-TARS platform during the Starburst Aerospace Accelerator’s annual East Coast Selection Committee event in 2018 at The Mitre Corporation in McLean, Virginia. At that point, the company had raised $2.55 million in seed round of financing led by Los Angeles-based Moonshots Capital, along with support from Austin’s Capital Factory, IronGate Capital Advisors, as well as other investments.

The Air Force took notice, especially when Robinson presented his idea for the AR technology to AFWERX—the service’s innovation arm—and to the Air Force Research Laboratory (AFRL). In February 2019, Red 6 Aerospace ran feasibility demonstrations for the service, showing the augmented reality platform in a static environment—in the cockpit of a test airplane parked in a hangar, which led to the SBIR Phase I award.

“We had representatives from the test pilot school, AFRL, ACC and AETC,” Robinson says. “They all came to see it. They were able to sit in the cockpit, put a helmet on that we designed, and through the visor system we built they were able to see augmented airplanes around them but also still see the real world.”

Since the SBIR Phase II award in November, Red 6 has been refining the technology and conducting aerial demonstrations for stakeholders. SIGNAL Magazine attended one of those demonstrations in California and met with Robinson and some of the Red 6 Team: Chief Technology Officer (CTO) Nick Bicanic; Chief Strategy Officer (CSO) David Reeths, a former Navy Intelligence Officer and Air Force Strategic Planner; Chief Product Officer (CPO) Glenn Snyder; and Director of Business Development Lt. Col. Tom Mackie, USMC (Ret.). In addition, Chris Randall is the director of aircraft operation.

The next step for the company, Robinson says, is to integrate the A-TARS technology into an Air Force plane by July, installing it either in a T-6 or T-38 with the guidance of the ACC, AFRL and PTN.

“It makes most sense for us right now to go into the T-6,” Reeths adds. “All of the U.S. fixed-wing flying components use variants of the aircraft for their initial flying training, as do a number of key allies or partners.” Having the use of the T-6 for A-TARS is ideal, especially from a cost standpoint. “Being able to go into a platform that has a much cheaper per hour operational cost, like a T-6, where it is a prop versus a jet aircraft, you can do a lot more testing of that at lower costs, as well as exploring what other training can be moved from expensive jet aircraft because of A-TARS,” the CSO says.

Additionally, Red 6 will be working on furthering the capabilities of the A-TARS display technology, Robinson says. “What I want is a very elegant solution which is built into the visor on your helmet. So, when you’re flying around, and you’ve got your helmet on, this would be an additional visor that just drops down. When finished, the visor system would have 150-degrees field of view, with a super-high resolution, low-latency display. That’s what we are developing over the next 18 months.”

The creation of A-TARS as a mid-air tool for pilots naturally required an aircraft, and for that, Robinson built a unique plane: a Berkut 540 propeller that is fully aerobatic and capable of 300 miles per hour and 9 gravitational force equivalent, or 9 Gs.

“Even though it’s a very basic aircraft, it is a very sophisticated plane performance-wise to fly in,” Robinson states. “We had to integrate A-TARS in our own aircraft because we are extremely nimble, innovating rapidly and doing aerial testing of the system on a near-daily basis, including doing multiple flights in a single day.”

The augmented reality platform, meanwhile, came largely as an inspiration from Snyder’s previous, groundbreaking work. A computer prodigy, the CPO subsequently used his digital reality and coding skills in Hollywood on the Transformers, Star Wars and Ender’s Game movies, and he was director of pipeline development for Los Angeles-based Imaginary Forces. His virtual reality (VR) and immersive technology experience led to the development with Adam Amaral of a car racing VR system, called Virtual Drift, which enabled one driver on a real racetrack in the United Kingdom to race against another driver on an actual track in the United States at the same time.

“Both racers wore virtual reality-based helmets but were driving real cars, around real race tracks,” Robinson explains of Snyder’s technology. “They met each other simultaneously in this virtual world and raced each other around a real track from opposite sides of the ocean. It was an incredible first in VR, and technically very significant. As soon as I saw it, I started connecting the dots and I asked him, ‘Is this possible in airplanes rather than cars?’ He said, ‘Yes, it is quite possible, although complex.’”

Robinson recognized that virtual reality, or the complete digital representation of the real world, was not feasible for pilots training in the sky. However, augmented reality, or the overlay of some virtual elements onto the real world, offered promise. Red 6 had to solve the problem of using augmented reality outdoors and in highly dynamic environments, such as jets traveling hundreds of miles per hour and performing rapid maneuvers, with Snyder evolving the technology, he says.

In order to create A-TARS, Snyder had to spend many hours flying in the Berkut 540 to test the AR components, sometimes flying four to five flights a day. “These days, most programmers are developing for platforms the size of laptops or cellphones,” Synder explains. “In those cases you’re able to test on the platform tens if not hundreds of times per day from your desk. With A-TARS, our test platform is the Berkut and we needed to ensure the technology would work in extreme cases, so I had to get used to programming while upside down very quickly. In order to keep up with our extremely tight time line, Dan and I were flying sorties seven days a week because it was the only way to test modifications to the software and hardware. By the time we had our first PTN officers up for a demo, I had about 150 hours in the backseat.”

Moreover, adroit pilots were a necessary factor in building out A-TARS. Col. Mackie, a 31-year Marine Corps veteran and Top Gun graduate, flew more than 3,000 hours as an F/A-18 fighter pilot for 21 years, active duty and reserves. Since last year, Robinson and Mackie have been taking Snyder and the other team members on countless sorties to hash out the systems’ features. Naturally, the Air Force identified specific criteria for A-TARS and the air demonstration.

“[In November] we started flying the air demo with external parties, including people from Pilot Training Next,” Robinson says. “We’ve been flying this version for the last few months, preparing for the initial external validation and for when the Air Force was first able to see this, and they were very excited about it.”

In addition, the company is speaking with the other services’ flying components about potential training uses, as well as possible operational applications. “We’re talking about beyond training now, where you can use this operationally,” he offers. “There are huge amounts of benefits to using this operationally, which ultimately makes everybody more prepared. And if frankly, the more prepared you are for war and the more capable they are as a fighting force, the less you actually have to fight.”