Challenges Emerge Reinventing the Joint Forces Command Network

When the U.S. Joint Forces Command folded, its networked functions did not disappear.

When the U.S. Joint Forces Command was disestablished nearly a year ago, transitioning its network proved more complicated than just flipping a switch—or even what was anticipated in various scenarios. Experts found that the nature of the command’s disestablishment brought to light significant incompatibilities in Defense Department networks.

The Joint Forces Command (JFCOM) did not disappear; it largely broke up into different parts that then were reassigned to other parts of the Defense Department. Elements did not cease functioning, nor did they morph seamlessly into their new parent organizations in the Joint Staff. Both equipment and processes needed to be coordinated. The network needed to continue functioning throughout the changeover. And, this complicated transition even brought unknown JFCOM network partners out of the woodwork.

With JFCOM scheduled to be disestablished by August 4, 2011, network experts realized that its network and its personnel had to be redistributed to the proper elements without suffering any loss of function or service. The command’s large information technology infrastructure comprised 300 servers, dozens of firewalls and intrusion detection devices, and a demilitarized-zone architecture. And, the infrastructure itself included more than 300 people. That total had to be reduced to 56 people supporting the new infrastructure.

Some technologies and capabilities could be leveraged from the Joint Staff, which would help reduce the strain on support personnel. Other technologies needed to remain local, explains Brian Jones, chief engineer, Joint Staff Office of the Chief Information Officer (OCIO), Hampton Roads, Virginia. And, some elements, including a large part of the network infrastructure, were designated for termination because they no longer were needed on a local basis.

Jones developed a plan of action and milestones (POA&M) document that featured more than 20,000 line items. About 3,000 users from the JFCOM transition worked on unclassified and classified networks, and they had to be moved to the Joint Staff Information Network. The network also had to transition from an old 2007 portal to Microsoft SharePoint 2010.

All of these efforts involved terabytes of information, and the entire activity had to be completed by March 31, 2012—without any loss of service or function.

The OCIO team obtained 10-gigabit communications pipes that connected Hampton Roads to the Pentagon. It also established trust between the Joint Staff network and the JFCOM network so it could move data from one network to the other. It stood up more than 60 local servers on the Joint Staff network in parallel to the existing JFCOM network to provide both nonsecure Internet protocol router network (NIPRNET) and secret Internet protocol router network (SIPRNET) authentication, file, print and email services in Hampton Roads. Maria Urrutia, division chief, Joint Staff OCIO, Hampton Roads, notes that the team had to move people from 21 buildings in Norfolk, Virginia, to four buildings in Suffolk, Virginia.

Along with these efforts, the OCIO team had to obtain permissions and rights within the Joint Staff network so that it had the authorities to administrate the servers. The Joint Staff’s network system had not been designed with the concept of remote sites in mind, Jones relates.

Jones says that joining the two networks represented a significant risk to the Pentagon. The Joint Staff network personnel did not know what the JFCOM network comprised, and understanding that took some time before approving the migration steps. The OCIO team obtained the needed trust from the designated approving authority (DAA), which for the Joint Staff runs out of the Joint Staff Support Services Office in the Pentagon. A legacy DAA in JFCOM was transitioned to the Joint Staff in September 2011, and the two DAAs orchestrated the migration. “It took a lot of work to get that signature to allow us to join the two networks together so that we could move the people gradually and systematically, as opposed to cold-turkey offline,” Jones says.

Among the major challenges facing the network transition was a change in operating systems from Windows XP to Windows 7, relates Lt. Col. Bradley Stumpf, USA, branch head for customer support, Joint Staff OCIO, Hampton Roads. Office software also had to be upgraded. “We knew we were going to have to touch every single person to do their NIPRNET migration and their SIPRNET migration,” he says.

For the command’s Norfolk campus, the NIPRNET migration was ready about a month before the SIPRNET equivalent, which dashed plans to do them concurrently. By performing the first NIPRNET migration on one of the directorates, the group learned useful lessons for the remaining effort. The colonel relates that the team was trying to migrate about two dozen users per day.

One month after the NIPRNET migration began, the SIPRNET effort was ready. So, for about two months the team conducted dual migrations, although no group of users had both networks migrated simultaneously. The SIPRNET migration provided unique challenges in areas such as 802.1x, port security and detachable hard drives, Col. Stumpf says. Jones notes that, in addition to changing the users’ computers, the team also was changing their Internet protocol (IP) addresses.

Managing the user data—which proved to be more voluminous than anticipated—was one of the biggest challenges, Col. Stumpf allows. Some users had 50 gigabytes of personal data that they expected to be moved for them. The team had to move all of that data up to the network, re-image the user’s box, and then pull that data back down to the user’s new computer. In hindsight, the team should have limited the amount of data that it would move and required users to burn their excess on CDs, he concedes.

One self-inflicted problem was that the team began the migration for the 1,700 personnel at the Suffolk compound right after the Christmas holiday break, the colonel admits. The team had only one full day to prepare for the migration, and then it encountered different technical issues than those at the Norfolk campus. These issues largely centered on differences between the two compounds’ infrastructures.

“We had a lot better success in Norfolk because, when we migrated people, we did it by location,” Urrutia relates. “The people were there; we could go from one desk to another.” At Suffolk, people slated for same-day migration were scattered throughout buildings. Technical experts had to disperse to address all of the migration needs, and they often had to gather equipment in a different location to perform necessary changes. Eventually, the problems were ironed out in a couple of weeks, and the team ultimately was able to increase its migration count to 50 users a day.

Several technology incompatibilities arose with the changeover. Some of JFCOM’s information technology was more advanced than its counterparts on the Joint Staff Network, and those differences had to be ameliorated.

Urrutia relates that the OCIO team had challenges with virtual desktop environment technology. The Joint Staff network was running on 32-bit PCs while the JFCOM machines were 64-bit PCs. Col. Stumpf elaborates that, in addition to the virtual desktop infrastructure, incompatibilities included different models of laptops, desktops and ClearCube blades.

JFCOM had been in the process of moving to a desktop virtual environment, so some of its infrastructure already had been modernized by the time of the command’s disestablishment. But, having a partially implemented plan meant that the team had to maintain two environments while transitioning to a third. That migration continues, Jones notes. By the end of 2012, the result will be 2,000 desktops replaced with virtual equivalents.

The team built the Joint Staff network in Hampton Roads from scratch, Jones relates. This includes the Windows 7 client image. All the initiatives that were on the Joint Staff’s back burner moved up as part of the changeover when the transition team showed how to incorporate them. The Joint Staff has adopted the 64-bit PCs for its network, and it is using the Hampton Roads image.

The team dealt with some hurdles when it faced the need to archive large amounts of data left over from JFCOM activities. The Joint Staff had no capacity for archiving these vast amounts, so the transition team resorted to planned storage capacity that was available. However, much of the database capability resided on the JFCOM portal, and it was used for purposes ranging from internal storage to sharing with external partners. The Joint Staff does not permit external access to its portal, which required a cultural change from the collaborative environment that defined JFCOM.

Jones explains that the team had to develop a plan for dealing with the different portal data-sharing requirements. Ultimately, the team enhanced the Joint Staff requirements by focusing on Intelink, which the Joint Staff uses.

One of the unexpected developments that emerged during the transition involved discovering the types of customers on the network. During the transition, the team was surprised to receive calls from various organizations about which they had no idea were on the network. “We had Navy cyberforces, NGA [National Geospatial-Intelligence Agency] and NCIS [Naval Criminal Investigative Service] on our network,” relates Cynthia Dilley, systems operations branch chief, Joint Staff OCIO, Hampton Roads. “I thought, ‘Holy smoke, who allowed these people to get on our network?’ Our network at the time was so sprawling and huge, it was amazing how it just grew and grew.”

Transitioning these hidden organizations was difficult, as many of them literally had lost contact with their sponsors or did not even know whom they were supporting. “When we asked some of them who their sponsor was, they were dumbfounded,” Dilley recalls. “Many of them were scrambling to find their sponsor.”

Jones also notes that everyone in a facility “is drinking from the same NIPRNET.” Yet, each organization has its own dedicated pipe to obtain that same NIPRNET. These are not separate networks, but they do involve redundant pipes. “Why don’t we make life easy—whoever is providing the water to the building provides the NIPR and SIPR for anybody that is sitting in that building?” he offers. “Then, you won’t have to build entire infrastructures to support each organization.”

One of the most important keys to the transition’s success was buy-in by leadership, the team offers. Dilley allows how all of the leaders supported the effort, although one general did express concern about the plan’s logistics. Team members opted not to make any changes lest they need to change the plan for everyone else, so it proceeded as originally designed.

While some transition funds were set aside for technology acquisition, the OCIO team tapped its existing budget to acquire any needed new technology. Jones relates that the group simply shifted focus for JFCOM life-cycle budget items. The downsizing meant that funds for purchasing new computers or servers, for example, could be re-allocated to purchases such as a new storage area network that helped provide a good foundation for transferring data seamlessly.