Fewer Data Centers Can Be a Good Thing
Careful planning can cut costs and offer other benefits.
According to Vivek Kundra, the federal chief information officer, the U.S. Defense Department was operating 772 data centers as of July 30. 2010. The Office of Management and Budget defines a data center as any room that is greater than 500 square feet and is devoted to data processing. Kundra called for a 38 percent reduction in the number of data centers by 2015. Though such calls are driven by budget considerations, the metric of counting how many data centers can be eliminated is misleading. From a budget standpoint, only the reductions in the Defense Department’s $36.3 billion fiscal 2011 information technology expenses will matter.
Sun Microsystems, Hewlett-Packard, Dell, Microsoft and Silicon Graphics now offer complete data centers in standard 40-foot x 8-foot and 20-foot x 8-foot shipping containers, occupying 320 and 160 square feet, respectively. Such data centers have a capacity of up to 29.5 petabytes of storage and up to 46,080 central processing unit, or CPU, cores of processing power. Such data centers would not be included in any of the Office of Management and Budget statistics.
The current costs of petabyte files ranged from $117,000 to $826,000 per petabyte. It is declining steeply. Therefore, the petabyte costs of a single container—the single largest hardware cost—would be up to $24.4 million. The estimated 100 petabytes needed to support most initial Defense Department cyber operations would be close to $90 million, which is well within the range of affordability if one considers that the total department budget for information technology operations and maintenance was $25.1 billion.
Perhaps the greatest omissions in the Defense Department data center count are the installations that are operated in the contractors’ names. In addition, the department has a large number of locations that are not designated as data centers but nevertheless are running many servers where each can cost less than $500 per year. Small clusters of servers are costly because of their expense for operators, electricity and overhead. Excluding small locations from consolidation plans overlooks an additional savings potential.
Current server virtualization technologies offer a larger multiple of the computing power than was available 10 years ago. Consolidations through virtualization so far have achieved 10:1 reductions in the number of servers hosted in data centers. Just shrinking the number of servers will be sufficient to reduce the need for data center floor space. Defense Department components therefore may be tempted to use server consolidations in lieu of data center reductions.
Counting the number of servers is not a good metric and should not be an index of data center size. Server consolidation without virtualization can replace poorly utilized $500-per-year servers with a poorly utilized $5,000-per-year mainframe unless the entire software environment in the data center is overhauled as well. That will take additional effort than just facility consolidation.
Therefore, any plans for proposing consolidation of data centers to a much smaller number of secure sites must show how the total cost of ownership of operations and maintenance will be realized. The net cash-flow break-even point for delivering the savings also must be determined. Because Defense Department accounting does not show depreciation expense, any cash-flow projections must include the costs of new capital expense as well as any write-offs for discarded equipment. Plans that focus only on data center facility counts are insufficient.
The greatest gains in data center consolidations come not from a reduction in the hardware costs—which are less than 10 percent of operating costs—but from cutting applications operations and maintenance expenses of personnel plus data center overhead. Personnel account for more than 50 percent of costs, and data center overhead accounts for more than 20 percent of costs. Every data center incurs the costs for software maintenance support, help desk staffing and supervisory overhead expense. An excessive concentration on the shrinking of hardware costs, as well as on power efficiencies and real estate charges, will overlook the major savings opportunities in the areas of direct and indirect labor.
Any discussion of data center consolidation also must include provisions for backup, fallback and service availability. Stand-alone, hard-to-defend and under-funded sites, with low-capacity utilization, will result in service levels below acceptable levels for cyber operations, which are at least 5 minutes of total downtime per year and less than 200 milliseconds of latency. Reconstitution of a failed data center, as is currently done, cannot take place in hours or even days. This type of hold-up would be intolerable for cyber operations.
The primary objectives for data center consolidation, therefore, should be the improvement in reliability, performance and exceptional levels of information security. Cost reductions are necessary but only secondary.
The analysis of data center consolidation savings must include an evaluation of related communications costs. The speed of propagation of transactions can be counted in a few milliseconds anywhere in the world. A reduction in the number of data centers will be feasible only if greater bandwidth is available to connect every major data center with direct links over fiber optic channels rather than via the Internet.
Any Defense Department commitment to reduce the number of data centers should be taken only after examining its consequences. For example, all data centers support a dedicated number of customers, who then are linked to their respective servers with unique local area network (LAN) and wide area network (WAN) connections. One cannot just rip out servers and mainframes without the careful re-engineering of whatever links are in place already. The existing LAN and WAN connections are supported by hundreds of small subcontractors who provide maintenance and operating services, each using the best of custom-made technologies that they have found to be affordable. None of this connectivity is standardized. Anyone attempting data center consolidation cannot just pick up the existing connecting links without conducting further examination and testing at every point during the transition from many data centers down to just a few.
Any of the more than 700 Defense Department data centers have multiple versions of operating systems, homegrown security appliances, unique communication processing methods, incompatible device interfaces and different network management consoles. It is unlikely that all of the procedures in place are documented adequately. Any data consolidation effort will have to develop a plan for how such variety can be synchronized to operate in the new environment.
Resident contractors retain much of the knowledge for operating the data centers, particularly under failure conditions. Potentially, the department’s data centers can include up to 2,814 different server versions connected to approximately 1,811 listed versions of customer clients that are managed by nearly 1,210 versions of operating systems. If all of these data centers are folded into a cloud, all of this diversity must be reduced first.
It is not known how much of this variability is present in existing Defense Department data centers, but it is estimated that the department’s legacy systems remain a large repository for obsolete software or applications that have not been upgraded for years. In addition, suppliers have contracts that either will have to be liquidated or be paid off for their unexpired terms. In any migration to a consolidated environment, the Defense Department also will have to deal with conditions in which legacy operators have installed unique procedural steps to preserve their lockup on the conduct of the department’s business.
Every data center contractor attempting to merge data centers into an infrastructure-as-a-service or a software-as-a-service cloud environment—as currently proposed by the Defense Information Systems Agency (DISA)—first will have to contend with the complex issue of how to migrate out of an existing operating environment into a cloud. DISA, or a cloud equivalent set up by the Army, Navy, Marine Corps or Air Force, then will have to define a way of handling customers in tightly prescribed ways after any consolidation takes place, as well as during every step taken during the intervening migration process.
DISA now has staked out a claim to become the leading provider of cloud computing services to the Defense Department for both unclassified and classified data. This will require major new investments. There are at least 36 ways to implement cloud services. The Cloud Security Alliance has published 192 cloud evaluation guidelines. The criteria include: cloud computing architecture; governance; risk management; legal matters; compliance and audit; lifecycle management; portability and interoperability; operations; business continuity; disaster recovery; incident response; remediation; encryption; key management and access management.
Any cloud computing evaluation will have to consider technology acquisition from cloud vendors such as Amazon, AT&T Synaptic Hosting, BlueLock Virtual Cloud Computing, Enomaly, GoGrid, Google, Hosting.com, Microsoft Azure, NetSuite, Logica, Rackspace Cloud, RightScale, Salesforce.com, Terremark vCloud Express and Unisys Secure Cloud. Each of these offers combinations of features and functions with varying degrees of lockup. Which one of these technologies will offer the Defense Department a choice of smoothly moving computer processing from one cloud vendor to another is a matter that will require choices as to who offers the greatest application portability.
Before proceeding with a large number of data center mergers, particularly into DISA, planning must be devoted to the disruption that department users will suffer during migration. Any plan to restructure data center operations first should start with investment in the engineering that would dictate the standards chosen and how the new data center environment will function. Dictating the cuts in data centers without also detailing how the transitions will support desktop hardware, desktop software and communication processing is risky. Many of the millions of existing Defense Department desktops are custom-connected to what are purely local arrangements between the servers and the respective customers.
Discussion about data center consolidation also must include the organization and setup of network operations centers (NOCs) for managing the interoperability and for assuring the security of the networks that connect all data centers. With the growth in the size of computer networks, the importance of central monitoring of operations rises as well. A large number of networks already manage hundreds or even thousands of servers. It is likely that the Defense Department already operates more than 100,000 servers.
Such large aggregations of equipment require real-time monitoring of security status, uptime, latency, capacity and response time. The Defense Department will have to rely on NOCs for monitoring all networks as well as for managing all of the tens of thousands of attached assets. To achieve interoperability and conserve the services of scarce personnel, a wide range of fully automated diagnostic routines will have to become standardized across all NOCs.
The tools available for NOC operations will become a major investment for the department. Such tools will represent a major share of intellectual capital that will be necessary for data center consolidation to take place. Consolidation also will require a tight integration between NOC operations and the Global Information Grid (GIG). Such coupling among the GIG and the NOCs will have to be included in the entire data center consolidation program. Ultimately, it will require moving the accountability for most status monitoring from the individual data centers to the NOCs. The NOCs then become the first line of defense for assuring the security of cyber operations.
There are economies of scale available from the consolidating of data centers into complexes with more than 500,000 square feet of computer floor space. Huge data centers show a dramatic lowering in the costs of information processing and in decreases in manpower expenses, while also increasing the reliability and latency of what ultimately would become one of many Defense Department server farms. Further savings can be realized by lowering the costs of electric power to run computers and air conditioning. The annual cost of electricity now exceeds the depreciation cost of computer hardware.
Firms exist that build and equip data centers. Commercial data center operators build efficient, large facilities and then lease them either as totally dedicated facilities, or as “cages” available for partial occupancy. In the case of major companies such as Apple, Microsoft or Facebook, the computer configuration in the server farms is standardized to meet the company’s proprietary system requirements. For example, Sabey Corporation currently is building a 350,000-square-foot data center for Dell. CoreSite offers several locations with finished “wholesale” data center space for users who seek turnkey space that can be deployed quickly. Equinix operates 22 data centers in the United States, seven in Europe and five in Asia. Some Equinix data centers are larger than 200,000 square feet. Rackspace operates nine data centers, which includes managed hosting. Interxion operates 28 sites in Europe.
Existing Defense Department data centers have been constructed over the past 30 years in small increments, each funded by separate contracts. Even the DISA megacenters, authorized in 1992 by Defense Management Review Decision (DMRD) 918, do not have the scale that matches anything that is under construction now. The department’s data centers do not reflect the economies of scale that have become available because of optic communications and new virtual software. Meanwhile, the cost of hardware has been shrinking at the rate of 18 percent per year, while the expenses for electricity and operating manpower have been rising to meet a growth in demand for services. The net result is that the existing proliferation of Defense Department data centers is not affordable.
The existing data centers have their origins in separate contracts, which dictated how computing facilities would be acquired and organized. It also is clear that the current number of data centers cannot support the increasing demands for security and reliability without an increase in budgets. With capital costs for constructing economically viable data centers now approaching $500 million, it is unlikely that such capital would become available with the prospective budget cuts and the relatively short schedule available to the Defense Department to arm itself for cyber operations.
The Defense Department fiscal 2010 budget for operations and maintenance of information technology was $25.1 billion. This makes the leasing of Defense Department-dedicated data centers—constructed by any one of the many commercial firms—a manageable proposition. It may be the way in which data center consolidation into clouds can be prepared for the time when information warfare will mandate a different way of configuring information technology operations.
The next installment in Paul A. Strassmann’s series on defense information technology will address whether multiple acquisitions can support cyber operations in the May issue of SIGNAL Magazine.
WEB RESOURCES
Cloud Security Alliance: www.cloudsecurityalliance.org
Federal information technology management reform plan: : http://cio.gov/documents/25-Point-Implementation-Plan-to-Reform-Federal%20IT.pdf
