An F-22 Pilot in a Real Jet Just Took on China’s J-20 in Augmented Reality

This article was originally published Nov. 16, 2020, on Sandboxx News. Follow Sandboxx News on Instagram. 

By Alex Hollings

An American veteran F-22 pilot just took on a virtual representation of China’s Chengdu J-20 stealth fighter from the seat of a very real aircraft in flight. The Chinese fighter was projected on the pilot’s augmented reality helmet-mounted display, giving a real pilot in a real aircraft the opportunity to train against a seemingly real foreign opponent. According to the two firms responsible for the test, this marks the first time such an engagement with a virtual adversary has ever been achieved.

In a story first covered by Thomas Newdick at The Warzone, U.S. companies Red 6 and EpiSci announced the successful test of their augmented reality training program earlier this week. Their veteran F-22 pilot participated in the test from the stick of a Freeflight Composites Berkut 560 experimental aircraft. Inside the cockpit, he wore a helmet that included an augmented reality display that projected the image of his opponent, China’s J-20, in his field of view. The enemy aircraft was not controlled by another aviator, but was instead a reactive adversary controlled by EpiSci’s Tactical AI technology.

“With this first-ever within-visual-range dogfight against an AI bandit, EpiSci’s Tactical AI demonstrated the ability to work on a real aircraft, with flight-ready hardware and sensors,” said Chris Gentile, EpiSci’s Vice President for Tactical Autonomous Systems.

“While fielding autonomous systems that control fighters may be in the future, this system is ready now to bring next-generation capability to our training programs, providing immediate benefit to the USAF’s ability to develop and maintain world-class fighter pilots. By introducing them to this technology now, they’ll be even more prepared to use a range of unmanned tools in the future.”

The AI-enabled, augmented reality J-20 pilot leveraged EpiSci’s experience in previous virtual dogfights. The firm was one of eight teams that competed in the U.S. Defense Advanced Research Projects Agency’s AlphaDogfight trials, which pitted a real U.S. Air Force F-16 pilot against virtual fighters crewed only by artificial intelligence. Unlike this recent test, the AlphaDogfight trials were held entirely within a virtual world, with the human pilot managing his aircraft through a virtual reality headset and game controls. The AI pilot handily defeated its human counterpart an impressive five times in a row, though it’s important to note that the test’s virtual environment and unusual methodology favored the AI in a way real combat may not have.

Red 6 worked in conjunction with EpiSci’s AI pilot with their Airborne Tactical Augmented Reality System (ATARS), which was responsible for the pilot interface that projected the virtual opponent into the pilot’s augmented reality view, effectively serving as the point in which the real pilot and aircraft could interact with the virtual J-20 opponent.

“Red 6’s ATARS system enables real pilots to fly real airplanes connected together in an augmented world,” explained Dan Robinson, CEO and founder of Red 6.

“With the additional integration of Tactical AI into our platform, we are now able to interact and respond to any threat aircraft. This opens spectacular possibilities for training.”

This technology could offer American fighter pilots a significant advantage in any potential 21st century conflict. While aggressor squadrons offer a very realistic simulation of the combat environment, there are real limitations to what can be accomplished through the use of aggressors. Aggressor squadrons, sometimes called “adversary squadrons,” are made up of aviators and aircraft used specifically for the purpose of simulating enemy forces in war games. Aggressors, many of whom are veteran military aviators working in the private sector, fly actual combat aircraft in simulated missions against U.S. forces.

Here’s a video of an aggressor A-4 Skyhawk running for its (simulated) life as a U.S. Air Force F-22 Raptor closes in.

While training against one another and aggressor aircraft offers American aviators with a very realistic facsimile of combat operations, this approach to training often does not offer the ability to fly against the sorts of jets American pilots will actually square off against in a near-peer conflict against a nation like China. The use of the J-20 for this test speaks to that limitation, as American pilots have never had to square off against an enemy force operating their own stealth aircraft before.

While F-117 Nighthawks have been seen in the skies over America’s southwest in recent months, prompting some to wonder if they are serving in an adversary capacity, the “stealth fighter” was never actually a fighter at all. The F-117 was designed to carry a relatively small payload of two bombs deep into enemy territory while avoiding detection. Aircraft like the J-20, on the other hand, are purpose-built to conduct fighter operations at long distances, including engaging enemy fighters. Put simply; the F-117 just isn’t a good stand-in for 5th generation fighters like the aforementioned J-20 or Russia’s Su-57.

Virtual adversaries, on the other hand, can be programmed to fly just like we anticipate enemy fighters to, using publicly sourced information about their aerodynamic capabilities and projected stealth profile. If leveraged by the U.S. Air Force, these AI models could be updated using classified intelligence regarding opponent aircraft, potentially making the virtual fighter’s tactics and performance even more accurate. That experience against enemy fighters could make all the difference when it comes to the split-second decisions fighter pilots are faced with in a dogfight.

“Dogfighting, or Basic Fighter Maneuvers as we call it, is an incredibly complex and dynamic environment. The most difficult part is perceiving what the adversary is doing,” F-35 Pilot instructor Justin Lee told Sandboxx News.

“You’re looking for minute changes in their lift-vector which foreshadows their next move. That’s why it’s important to have good vision (which can be corrected with glasses or surgery).”

This sort of training is just one of a number of areas where artificial intelligence is likely going to dramatically change the world of military aviation. While AI-controlled virtual aircraft can change the way fighter pilots square off against opponents in training, artificial intelligence will eventually also supplement the pilots themselves. While DARPA’s AlphaDogfight trials pitted an AI pilot against a human one, the stated goal of their Air Combat Evolution (ACE) endeavor was to help pilots grow more accustomed to sharing the sky, and even the cockpit, with AI.

“Regardless of whether the human or machine wins the final dogfight, the AlphaDogfight Trials is all about increasing trust in AI. If the champion AI earns the respect of an F-16 pilot, we’ll have come one step closer to achieving effective human-machine teaming in air combat, which is the goal of the ACE program.”

Artificial intelligence can supplement human pilots within the same aircraft in a number of ways: from more capable auto-pilot systems to taking quick action when an inbound missile is detected. However, in order for pilots to be able to fully leverage the capabilities AI can offer, they need to learn to work with these systems, to trust the decisions they make, and to adjust their approach to combat operations. The more simple tasks the AI can take over for the human pilot, the more focus that pilot can place on complex tasks like managing the battles space.

Pilots can also soon expect similar AI systems to control autonomous wingmen flying alongside them into battle. Program’s like the U.S. Air Force’s Skyborg are already aiming to field armed UCAVs alongside crewed aircraft. By using low-cost attritable platforms like the Kratos Valkyrie, Skyborg drones could support fighter operations by taking orders from human pilots and using AI to quickly determine the best ways to achieve their objectives. These drones could engage ground targets, fly ahead of the manned fighter to increase its sensor range, or even sacrifice themselves to save the the pilot and his or her aircraft.

“Pilots of the future will need to be comfortable teaming with AI — and training is the ideal place to introduce this technology,” said EpiSci’s Gentile.