Virtual Humans Keep It Real

The Virtual Afghanistan virtual reality tool enables soldiers to immerse themselves into simulated dangerous environments as treatment for post-traumatic stress disorder (PTSD).

Superheroes and genuine heroes benefit from U.S. Army-funded research.

U.S. Army soldiers have something in common with Superman and Spider-Man: they all benefit from Army-funded virtual reality research being conducted at the University of Southern California Institute for Creative Technologies. The Oscar-winning research has made digital characters look more realistic in movies such as Avatar, Spider-Man II and Superman Returns, among others, and it also helps soldiers cope with post-traumatic stress disorder. It also is used to train service members for a variety of missions and situations, including countering both improvised explosive devices and insurgency operations, as well as tactical intelligence gathering. The institute’s research, which rapidly is taking the “virtual” out of virtual reality, also helps teach soldiers such traits as leadership, cultural awareness and relationship building.

The Institute for Creative Technologies (ICT) was founded in 1999 using Army funds. The goal is to optimize progress in virtual reality technologies developed by the movie and video gaming industries with advances being made in military modeling and simulation. The benefit for the Army is more realistic training. Advances made at the ICT can help soldiers at every stage of their careers, from the time they are recruited until they return to their civilian lives. ICT officials estimate that about 50,000 troops have used tools developed by the institute.

One major research area focuses on virtual humans. Unlike traditional avatars, which users control, these characters are computer programmed with artificial intelligence to understand language and respond appropriately, and to carry on a conversation unassisted by a living human. That conversation can include flashes of humor and realistic body language and facial gestures. In fact, some characters have been mathematically programmed with basic human emotions. “Virtual humans are artificial intelligence-controlled humans. They are meant to act and respond as a real human so that you can have a real conversation with them,” says John Hart, the Army’s ICT program manager. “These virtual humans have emotions.”

Emotions are important in part because they impact communication and decision-making. Emotions also make the virtual humans more believable, which improves the training experience. In the Cognition and Emotion Project, ICT researchers are developing models that allow synthetic characters to derive an emotional response to events in the world and respond with behaviors consistent with that emotional state. The project uses virtual characters to investigate how people interpret emotional behavior and how those interpretations influence memory and decision-making.

The ICT is working with the university’s medical school to train students to conduct therapy sessions. “One of the characters is a rape victim. The students are supposed to talk to her and find out what’s going on, but she initially doesn’t want to talk. She’s very reserved, very shy and kind of scared, so you see these emotions in both the face itself, how she reacts and [in] her tone of voice,” Hart explains. “Even though students know the virtual human is a computer-controlled character, they accept the character as a real, live human being.”

Two closely related projects—Virtual Iraq and Virtual Afghanistan—help soldiers cope with post-traumatic stress disorder (PTSD). The projects use exposure therapy, in which the soldier, guided by a trained therapist, confronts traumatic memories through a retelling of the experience. The system is used across the country, especially in medical centers such as Walter Reed. ICT researchers built on technology developed for the Xbox gaming system and the combat tactical simulation scenario Full Spectrum Warrior. The current applications consist of a series of virtual scenarios specifically designed to represent Middle Eastern cities and desert road environments. In addition to the visual stimuli presented in the virtual reality head-mounted display, directional three-dimensional (3-D) audio, vibrations and smells can be included in the simulation.

Now, rather than relying exclusively on imagining a particular scenario, a patient can experience it again in a virtual world under safe and controlled conditions. In addition, the current generation of young military personnel, having grown up with digital gaming technology, may actually be more comfortable with participating in virtual reality treatment than in traditional talk therapy. “With a clinical psychologist, the soldier can go in and do these multiple exposure therapy treatments, where they’re able to put on a head-mounted display. They have a weapon. They can be put into a Humvee, either as a driver or as a passenger, and they have the sights, the sounds, even the smells. This is being used in an experimental phase right now, but it’s showing some positive results,” Hart reveals. “We have to run a larger set of clinical trials. We’re working to do that right now. We’re running it in small doses and collecting data as far as the effectiveness of this type of treatment.”

ICT research also helps train military leaders to make sounder decisions in complex situations. The UrbanSim project, for example, is a personal computer-based virtual training application for practicing the art of battle command in counterinsurgency and stability operations. It includes a game-based practice environment, a Web-based multimedia primer on counterinsurgency doctrine and an array of scenarios. UrbanSim allows trainees to plan and execute missions in the role of an Army battalion commander. Users lead the battalion in a variety of missions, including maintaining stability, fighting insurgents and reconstructing the civil infrastructure. “It’s almost like a SimCity-type thing where they’re given an area of land—that’s their area of responsibility—and they have various things going on. Maybe there’s crime, maybe insurgents, maybe some water pump stations are not working, or schools are not open for some reason, so they’re going to have to make decisions as far as what actions to take,” Hart explains. “What we’re really trying to do is to show them that there are second or third order effects, maybe positive and maybe negative, to the decisions they make. It lets them practice the art of battle command in a safe environment.”

These digital renderings of the face of actress Emily O’Brien are part of the process for creating a realistic, three-dimensional virtual image of a living person.

UrbanSim uses an underlying socio-cultural behavior model, coupled with a story engine that interjects events and situations based on the real-world experience of former commanders. It provides guidance and after-action review capabilities. The system is being used and evaluated in a number of Army training contexts, including courses at the School for Command Preparation and the Command and General Staff College, both at Fort Leavenworth, Kansas. ICT officials are working with the Program Executive Office for Simulation, Training and Instrumentation (PEO-STRI) to deploy the system nationwide. Earlier this year, the Army started the process of fielding the system service-wide. “We’re working with PEO-STRI to deploy it on a set of gaming computers that are being established at these battle simulation centers throughout the Army,” Hart says.

Future technologies may enable soldiers to train on the go by building applications for mobile devices, such as smartphones and tablets. The devices could be used, for example, to allow soldiers to watch a training video and then discuss it in a classroom environment or to take online correspondence courses. “The idea is that eventually you’ll see a smartphone in the hand of every soldier, so that’s a new area [of training] just starting to take hold,” Hart says.

Tactical questioning is an area of training that teaches soldiers to socialize with local citizens and to gather intelligence. Another program focuses on carrying out bilateral negotiations. In both cases, soldiers interact with virtual humans as training for real situations. “Those are two that we’ve transitioned over to PEO-STRI,” Hart says.

One of the most realistic virtual characters is known as Digital Emily, a collaborative project between the ICT and Image Metrics, a Santa Monica, California, company specializing in 3-D facial animation. The project used cutting-edge lighting techniques to develop a 3-D digital rendering of the face of soap opera actress Emily O’Brien, who is best known for her Emmy-nominated work on The Young and the Restless. The process is an advance over an ICT lighting technique that won a Scientific and Engineering Award at the 2010 Academy Awards Oscars ceremony. Digital Emily is practically indistinguishable from video of the live actress. “Our scientists have been recognized as [being at] the top of their fields in graphics and virtual humans research, contributions to the artificial intelligence community, trauma psychology, emotion modeling and social simulation. And in a sign that we’ve really made it in Los Angeles, we’ve even received an Academy Award, which was given to one of our researchers, Paul Debevec, for his advances in creating believable digital faces. His techniques and technologies have been used in Avatar, The Curious Case of Benjamin Button, Spider-Man 2 and many other blockbuster films,” says Randall W. Hill Jr., ICT executive director.

Another character, Sergeant Star, has been used for recruiting purposes. He is a life-sized character projected onto a semi-transparent screen and is fully capable of conversing with his human counterparts. His chest moves with each breath; his eyebrows arch when appropriate; he shrugs; and he jokes about his being a moving, thinking, talking and incredibly handsome soldier. The character answers questions on topics including Army careers, training, education and money for college. He also handles queries about the technology behind his development and explains how his creation fits in with plans for future Army training environments.

The art of storytelling is a major component of virtual reality, according to Hart, because soldiers need to immerse themselves in the story of the training simulation, just as they would immerse themselves into a book, movie or video game. The institute is working to improve the technology behind the art of digital storytelling. For example, if soldiers using a particular simulation system need to take a right at a fork in the road and they go left instead, the system can, unbeknownst to the user, move whatever they were supposed to run into so that they run into it anyway. In addition, the institute is soliciting true stories from soldiers that can be incorporated into training scenarios, making the training that much more believable. Hart cites a quote by author Ursula K. Le Guin on the importance of storytelling: “There have been great societies that have not used the wheel, but there have been no societies that did not tell stories.”

WEB RESOURCES
Institute for Creative Technologies (ICT): http://ict.usc.edu/
ICT projects: http://ict.usc.edu/projects
Program Executive Office for Simulation, Training and Instrumentation: www.peostri.army.mil/

VIDEOS
Sgt. Star: http://ict.usc.edu/projects/sergeant_star/C40
Digital Emily: www.youtube.com/watch?v=bLiX5d3rC6o