Live and simulated onslaughts lead to improved defenses.
Coordinating a mission execution during exercise BLACK DEMON are (foreground, l to r) Senior Airman Patrice S. Mobley, USAF; Tech. Sgt. Thomas R. Harry, USAF; and Staff Sgt. Gary D. Smith, USAF. The exercise tested Air Force information security on both a simulated range and live networks. In the background (r to l), Capt. Aaron N. Lamb, USAF; 1st Lt. Christopher K. Roberts, USAF; and Capt. Darryl Mosley, USAF, prepare the next scheduled mission for execution.
People and equipment rise to the occasion when military computer networks are attacked, according to evaluators at a recent U.S. Air Force exercise. A two-week event that tested experts on both native Air Force networks and a simulation range produced some surprises in the capabilities of humans and hardware.
The exercise spanned the Air Force network enterprise command and control hierarchy from its lowest level—the network control center—up through the regional major commands’ network operation security center to the Air Force computer emergency response team. Ultimately, it reached to the Air Force Network Operations Security Center (AFNOSC).
Known as BLACK DEMON 2004, the exercise was hosted by the 23rd Information Operations Squadron in the Air Force Information Warfare Center at Lackland Air Force Base, Texas. A total of 14 bases took part in black demon at 22 locations, which provided the added benefit of multiple data points so analysts could see which efforts worked best across the Air Force. The worldwide exercise included personnel in Europe and Hawaii who worked concurrently.
“We have taken an Air Force tried-and-true mechanism—flag-level exercises—and applied that to our network defense,” declares Lt. Col. John D. Bansemer, USAF, 23rd Information Operations Squadron commander. “That methodology has proved enormously successful.”
While the exercise involved only a small subset of the network cadre—fewer than 500 participants—its lessons will be imparted to upgrade training for more than 31,000 Air Force network professionals.
“We are starting to solidify the ability to train our network operators to be able to defend against very realistic threats,” the colonel says. Instead of using tabletop sets, BLACK DEMON adds realistic ranges and simulators as part of the infrastructure. The result is that participants learn many new ways of solving network problems and new capabilities inherent in existing gear.
One element that sets black demon apart from many other exercises is that it has an intelligence component, explains Maj. Aras P. Suziedelis, USAF, director of operations at the squadron. Participants receive intelligence scenarios that complicate the problem that they must solve. “It really tests their tactics, techniques and procedures and gives them an opportunity for realistic training,” he says.
Col. Bansemer reports a significant increase in the level of participation since the last BLACK DEMON exercise, which was in 2002. This year, all nine major Air Force commands took part with about 450 people who planned or executed the exercise. In addition, organizations such as the Air Force Communications Agency (AFCA), the Air National Guard and the Air Intelligence Agency participated, he relates.
The AFCA worked closely with the squadron, placing personnel at every range location, Maj. Suziedelis says. The agency built the ranges used in BLACK DEMON, and they are used year-round.
The AFNOSC served as a sponsor for BLACK DEMON, Col. Bansemer relates, because part of the exercise included refining command and control (C2) procedures. He notes that, in some cases, the AFNOSC “did too good a job” of tightening procedures. Some of these tightened C2 procedures actually prevented accomplishing other training objectives.
The exercise was conducted both on live networks and in a simulated range. It was scripted carefully down to minor details so that analysts would know exactly what transpired throughout the several days of the event. Maj. Suziedelis emphasizes that this scripting focused on only aggressor activities and did not limit responses by participants.
BLACK DEMON was split between the live and simulated venues for two reasons. First, exercise planners sought to train as realistically as possible, so the operational networks provided participants with real-world experience on familiar settings and equipment. The second reason involved a desire to examine extensive attacks that could cause problems on live networks. To avoid participants jumping back and forth between different venues, officials made the range the exclusive home of the exercise for its final week.
The exercise began on the live networks where personnel work regularly. The first two days of BLACK DEMON on these networks gave the operators various stimuli to determine how they would react on their operational arena. On the third day, the exercise was moved off the live networks and onto the range networks.
Col. Bansemer relates that the range allows planners to be more aggressive in exposing the operators to a variety of threats. The efforts on the live networks could not go as far lest they inflict damage to these active operational elements. The range permitted far more adventuresome scenarios that reflected possible real-world threats.
The 11-hour time zone difference between Europe and Pacific players helped provide a degree of veracity with real-world conditions. An information attack can be felt around the world simultaneously or in near real time, so planners replicated that scenario.
|Senior Airman Michael Sebel of the 868th Communications Squadron, Network Security Flight, monitors the automated security incident monitor/common intrusion detection director and Symantec intruder alert at the Air Mobility Command Network Operations Security Center, Scott Air Force Base, during the live-play portion of BLACK DEMON. This portion, which led off the exercise, featured red-team probes on the live network.|
The red team initiated a wide variety of attacks that were based on real-world scenarios from intelligence reports, relates 1st Lt. Christopher K. Roberts, USAF, an infrastructure and technology expert at the squadron. When the exercise took place on the live networks, these attacks largely took the form of reconnaissance probes. The exercise on the range featured more significant types of attacks such as denial of service, failure of network defense tools and insider threats, including unauthorized or malicious users. Other hazards included malicious logic and loss of firewall protection.
Many of these threats are familiar to readers of newspapers and news magazines, the lieutenant allows, so the planners applied them within the context of a scenario presented to the players. For example, several scenarios featured a malicious user inside the network moving among different systems to exfiltrate data. The exercise did not feature a red team trying to enter a system and alter data, he notes.
Some scenarios required a crew person to trace a designated Internet protocol to a specific location. Performing this live is more complicated than doing it on a smaller range, Lt. Roberts explains, so it was run on the live network. The reconnaissance attacks, which also were run on the live network, were highly taxing in that their subtlety made them difficult in many cases to discern.
Col. Bansemer allows that the exercise deliberately allowed the red team to win on some occasions. The intent was to give the stimulus to the network defenders, which was an important aspect. In other cases, the aggressors were handicapped because planners were focusing on achieving specific training goals.
Observers found room for improvement in a couple of general areas. Tactics, procedures and equipment use all were subjected to constructive criticism. Training is another area that will be improved as a result of BLACK DEMON findings, the colonel notes.
The observations that are still emerging from BLACK DEMON fall into several categories, Col. Bansemer relates. One observation that emerged from the exercise is that participants’ reaction times improved significantly, he says. Whereas early in the exercise participants may have taken an hour to respond to a problem, as BLACK DEMON progressed they began to address problems more quickly.
Ultimately, some participants were responding to red team attacks so quickly that planners did not have an event waiting in the queue for up to 30 minutes. “One lesson for us was to have more things in your hip pocket to hit these operators with because they are going to get good very quickly—they have a very quick learning curve,” Col. Bansemer states.
A related area involves training. Observers noticed several areas where they can focus training so that operators can respond better. Planners literally misjudged their ability to adapt to changing conditions, the colonel reports. Officials had to redesign the exercise on the fly because these operators were able to adapt so quickly.
Maj. Suziedelis adds that some of these operators were “world-class network administrators and defenders.” Some locations would have an operator who was almost mythical in stature among peers for knowing a router forward and backward. This operator would know what was possible on that router and would react to problems accordingly.
In fact, many locations featured individuals who knew their systems “like the back of their hands,” the colonel adds. These personnel helped uncover hardware and software capabilities that experts had never imagined being applied to network defense purposes. “From a technology standpoint, we have very good equipment, and we may just need to use some of the features that we didn’t even know existed on the equipment,” he exclaims.
Analysts saw how these operators were using existing field equipment in unique and effective ways. The exercise permitted a solution observed at one site to be implemented across the range of Air Force systems.
The same applies to tactics, the colonel adds. About 30 different ways of applying a tactic to a specific situation are, pending evaluation in a laboratory environment, poised for servicewide implementation.
One leading lesson was a recognition of the range of threats facing the network. Consequently, participants understood the need for an improved security posture for use on a regular basis and not just during an attack.
Another key point hammered home was the importance of knowing a network inside and out. One benefit of that knowledge is that defenders will recognize the difference between normal operations and abnormal situations that might be indicative of an attack. And, they will understand the full capabilities and the tools available to defend the network.
The exercise highlighted several network management and security processes that can be improved by increased or enhanced automation. Automating these processes can reduce planners’ reliance on placing personnel in the right place at the right time.
BLACK DEMON did not employ live computer viruses in the range. However, it did generate specific virus scenarios so that their stimuli—attack indicators—appeared to network defenders as if actual viruses were invading the architecture. One discovery was that some systems did not have patches for a malicious logic virus that has been circulating for years.
While lessons learned from BLACK DEMON 2004 still are being compiled, officials are planning the next iteration. It will follow a series of smaller AFCA simulator training exercises throughout the year. The next black demon will build on these and the recent exercise with the goal of generating a “Red Flag” for network defense.
The next exercise likely will provide an order-of-magnitude leap in realism, the colonel offers. Experience gleaned from this event puts planners much closer to their goal of having a range that is as close as possible to the real world.
Lt. Roberts notes that participants can expect hardware changes and new software that will make the simulations more realistic. These changes will affect the range in ways that personnel would observe on a live network during an actual attack. And, operators will be challenged much harder next year.
“We are really looking forward to putting the operators through the hoop in the future,” Col. Bansemer says.