Military Humanizes Virtual Population

The Institute for Creative Technologies (ICT) is using AI.implant, developed by Engenuity Technologies Incorporated, to build an urban environment as part of the institute’s Integrating Architecture project. The result will be a “serious video game” that helps warfighters rehearse urban warfighting and negotiation skills they have learned in the classroom.

By employing artificial intelligence, the U.S. Army is raising video games to a new level to create virtual communities populated by hundreds of thousands of fully developed characters. Combining computer game technology, a little show business magic and the expertise of some very clever research engineers, these training tools can be used to practice traditional warfighting techniques as well as to rehearse new skills such as conducting effective negotiation and understanding cultural influences.

This next generation of “serious gaming” is built on projects that enable soldiers to experience training that is so compelling they react as if the virtual world is real. The Institute for Creative Technologies (ICT) is among the organizations developing these immersive training tools. The institute, part of the University of Southern California in Los Angeles, opened its doors with Army funding in 1999. Early ICT work includes the Full Spectrum series of video games: Full Spectrum Warrior, Full Spectrum Leader and Full Spectrum Command. Computer game developers, Army instructors and ICT researchers built the games to focus on the leadership skills that commanders at the platoon, squad and company levels need. As with all the serious games the institute creates, students first learn through traditional instruction methods then use the games to practice their new skills.

Dr. Michael van Lent, project leader at ICT, heads the Integrating Architecture project and the Social Simulation and Cultural Representation project, among others. He admits that using simulation technology for training is not new but explains what sets ICT’s work apart. “The key difference that ICT brings to this work is a strong link to the entertainment industry, which is why we’re in Los Angeles. So that means both Hollywood and the computer game industry. All of the immersive training prototypes we develop here include both academic research in fields such as artificial intelligence and technologies invented from the entertainment industry—movies, computer games, game engines and that entertainment industry ‘pixie dust’ that gets people so caught up in these things,” he says. Prototypes are evaluated in classrooms and if proved effective, are handed off to Army programs where they can be fielded to the service’s schools.

One of the training tools the ICT created blends the movie experience with an instructional infrastructure available on the Web. Called Army Excellence in Leadership, or AXL, the 15-minute movie plot was written by institute personnel with input from soldiers returning from Afghanistan and Iraq. Described by van Lent as “not the typical Army instructional movie,” the film features characters and even a third-act plot twist. After watching the movie, student soldiers move to the Web page and view interviews with some of the characters in which they provide their perspectives of the plot events. This integrated product leads students through a deeper understanding of the material and, because it is Web-based, allows pupils to tap into comments and conclusions from other students. “So, you’re benefiting from the larger community of people who have used this training tool,” van Lent explains.

One of the most recent ICT serious games is geared toward training lieutenant colonels and colonels. ELECT BiLAT enables them to practice how to prepare for and conduct meetings with locals in a cross-cultural context. For example, the game simulates an officer meeting with a local police chief, government official or cleric in the area of operation, allowing the officers to rehearse the encounter to understand the cultural niceties and negotiation nuances.

The first ICT product of the Army Technology Objectives program, ELECT BiLAT was displayed at the Army Science Conference last November. It is scheduled to undergo a classroom evaluation early this year.

Building on these accomplishments, the institute’s researchers now are undertaking a project as intricate as the human genome. It is called the Integrating Architecture initiative and involves the complicated, unpredictable marvel known as human behavior. ICT’s goal is to create a densely populated urban environment that comprises hundreds of thousands of characters moving and acting simultaneously.

To tackle this challenge, ICT researchers turned to a company that has been involved in the simulation business for 20 years. Engenuity Technologies Incorporated, Montreal, started in the avionics sector designing human-machine interfaces for aircraft simulators. The company also developed a simulation suite that creates virtual environments for training.

For the past five years, the company has been working in the video gaming industry, building games for systems such as PlayStation 2 and Xbox. ICT is using one of Engenuity’s more recent commercial technologies, called AI.implant, to create a virtual urban environment. AI.implant offers user-friendly tools that allow game developers—or in ICT’s case, researchers—to create unique, engaging characters as well as crowd interaction.

According to Dr. Paul Kruszewski, chief technology officer, Engenuity, AI.implant’s graphic user interface simplifies the programming process so that subject matter experts in human behavior can design characters themselves. “Humans are not important when doing the aircraft simulators, but when you get to the urban fighting environment, you need better humans because that’s where the fighting occurs. It’s one thing to design a virtual soldier; it’s another to train the entity to act like a soldier. What we’re providing is the brain behind the simulation. AI.implant populates the situation,” Kruszewski explains.

The work has focused on how to bring about control in an area such as an Iraqi or Afghan village. Kruszewski says that soldiers can predict when they will be attacked with some certainty if they are properly trained and know the cues. “If you drive into a village and people are coming up to you, it’s probably OK. But if you drive into a marketplace and everyone starts sitting down or moving away, you may have a problem,” he offers.

Van Lent shares that ICT decided to use Engenuity’s artificial intelligence product because it offers effective solutions to two of the most difficult, time-consuming problems that must be overcome to develop the urban environment. The first is path planning, or figuring out what path a virtual character needs to take to get from point A to point B when people, buildings and cars are in the way. “AI.implant has a good solution for that path-finding problem,” van Lent explains.

The second problem AI.implant solves is steering. Once the path has been determined, the researchers can use the technology to create movement orders for the entity so it can follow the path.

“The challenge for us is creating a densely populated urban environment. We want a large town or city that has hundreds of thousands of characters moving around. That obviously means you have hundreds of thousands of individual characters, each trying to figure out how to get from point A to point B then doing it. This is why it’s important for us to have this high level of optimization in the path finding and steering,” van Lent maintains. To generate another component of the virtual world—behavior—the ICT team will use its own artificial intelligence research, he adds.

The German firm Krauss-Maffei Wegmann GmbH, which manufactures training equipment and simulators for a variety of military vehicles, chose to use AI.implant to update its product line offerings. One of the military’s new requirements is a simulation that allows troops to train and practice Military Operations on Urbanized Terrain, or MOUT.

“It’s what the guys at AI.implant call the ‘neutron bomb syndrome’ where you have this city that looks great, but there really isn’t anyone living in it. We’re trying to have the full city that’s got the full number of people because we’re not trying to just entertain but to educate. To do that, you need to model what’s going on in the entire city because even though activities are spatially distant, they could impact what’s going on in the student’s neighborhood. If you’re trying to give students a realistic environment in which to learn, you can’t pretend that most of the city is asleep most of the time,” van Lent offers.

ICT researchers have a much more ambitious goal than filling a virtual city with automaton characters that can go from one place to another without bumping into each other. Artificial intelligence especially will come into play when the development team moves to a level beyond AI.implant in which complete personalities will be crafted for each of the characters.

This will involve examining many facets of life as a human, van Lent explains. “Who are they? What are they doing? What are their jobs? Who are their families? Who are their friends? What’s their political party? What religion are they? What ethnicity are they? Where do they come from, and how does that affect and influence their behavior and what they choose to do and who they choose to talk to? And most importantly, how do they react to the actions of the U.S. soldiers in the simulation environment?” he asks. The goal of the work is to develop a training tool for senior officers so they can understand the second- and third-order effects of their actions in the neighborhoods of cities such as Baghdad or Fallujah.

ICT has started working with behavioral scientists to guide the initiative and ensure that specific aspects of human behavior are included in the artificial intelligence model. “Obviously one of the big aspects of this is that the 200 or whatever number of common activities here in the U.S. are different than the 200 common activities in Iraq, which are different than the 200 common activities in Afghanistan. And even in a Sunni neighborhood in Iraq, those 200 activities would be different than in a Shiite neighborhood. So we are starting to tie into social scientists, sociologists and cultural anthropologists to help us,” van Lent says.

“From an artificial intelligence perspective, there’s been very little work on modeling that kind of stuff in the artificial intelligence model. Most artificial intelligence models assume that all people are the same: They’re all trying to do the optimal actions to achieve their goals. One of the things we’re working on as another project is trying to create artificial intelligence models that explicitly model ‘I’m from this culture, and you’re from that culture, and I have assumptions about your culture, and you have assumptions about my culture, and I have assumptions about how you view my culture.’ All of this has a big influence in how we behave when we’re interacting,” he explains. ICT calls this area of study computational social science, which van Lent views as an offshoot of artificial intelligence.

Another challenge the ICT team faces is what van Lent calls “handling the emergence.” Once 100,000 people each have identities and the choice is made about what actions they will take, the researchers will turn the virtual world on. Van Lent says that the team will not be able to predict or even control very well what will happen. The challenge for ICT researchers is to make sure that the environment still goes in directions that produce useful training experiences for students.

While the institute went to the commercial sector for some of the technology, the researchers also needed to learn as much as possible from warfighters returning from Iraq to design a realistic experience. For instance, one officer relayed the story about his decision-making process for imposing a curfew in a town. He was concerned that a curfew would cause the local population to resent the United States. But after the curfew was imposed, crime in the town decreased significantly and the residents were grateful. “So imposing the curfew actually had the opposite effect than what the officer was concerned about,” van Lent offers.

Helping U.S. soldiers work with local residents requires the ability to teach students to manage the interrelationships between different cultures. For example, when a local resident wants to meet with a U.S. officer, the officer must determine whether to see the resident or have a subordinate do so. If the officer meets with the resident, that person achieves a certain status that the officer may or may not find desirable. “That’s where we’re heading with this, and obviously to do that you need this large but fairly realistically modeled population. AI.implant is the building block that’s helping us move all these people around the environment,” he adds.

Van Lent is excited about bringing this virtual city to life. “Having a city where there are hundreds of thousands of people moving around and having the different cultures model and the different political factions model, social relationships model and family model are intriguing. What’s exciting is just watching this big sandbox, if you will, turn over. There’s so much going on, and there are so many layers interacting in so many ways that you can’t really predict what’s going to happen. You can almost imagine it like the ultimate ant farm or the ultimate dollhouse. You set it up and you turn on the switch, but you never quite know what’s going to happen and what direction it’s going to go in,” he relates.

The urban environment training tool prototype is scheduled to be in the classroom for evaluation in 2008.

Web Resources
Institute for Creative Technologies: www.ict.usc.edu
Engenuity Technologies Incorporated: www.engenuitytech.com
Army Excellence in Leadership: http://projects.ict.usc.edu/axl