signalarticles

A multi-agency big data initiative offers an array of national advantages.

U.S. government agencies will award a flurry of contracts in the coming months under the Big Data Research and Development Initiative, a massive undertaking involving multiple agencies and $200 million in commitments. The initiative is designed to unleash the power of the extensive amounts of data generated on a daily basis. The ultimate benefit, experts say, could transform scientific research, lead to the development of new commercial technologies, boost the economy and improve education, all of which makes the United States more competitive with other nations and enhances national security.

Big data is defined as datasets too large for typical database software tools to capture, store, manage and analyze. Experts estimate that in 2013, 3.6 zettabytes of data will be created, and the amount doubles every two years. A zettabyte is equal to 1 billion terabytes, and a terabyte is equal to 1 trillion bytes.

When the initiative was announced March 29, 2012, John Holdren, assistant to the president and director of the White House Office of Science and Technology Policy, compared it to government investments in information technology that led to advances in supercomputing and the creation of the Internet. The initiative promises to transform the ability to use big data for scientific discovery, environmental and biomedical research, education and national security, Holdren says.

Currently, much of generated data is available only to a select few. “Data are sitting out there in labs—in tens of thousands of labs across the country, and only the person who developed the database in that lab can actually access the data,” says Suzanne Iacono, deputy assistant director for the National Science Foundation (NSF) Directorate for Computer and Information Science and Engineering.

An industry group is racing to draft commonly accepted best practices for security and privacy. 

A coalition of information technology companies has banded to create industrywide standards to help both the public and private sectors manage large information datasets. This effort is tapping some of the best minds in computing to work with datasets that now are reaching into the realm of petabytes and exabytes in size.

Whether these large datasets come from harvesting the buying habits of customers on a store’s website or from a military fleet of drones performing aerial reconnaissance over Iraq or Afghanistan, managing huge volumes of information can be compared to trying to take a sip of water through a firehose. New ways must be developed to manage and secure large volumes of information when traditional database applications fall short. In some cases, the inability to provide timely analysis of an extremely large dataset may result in a missed business opportunity or, as in the case of the intelligence community, failure to capture vital information for security purposes. In addition, the limitations of existing analytic tools when it comes to big data may hamper a business or government organization’s ability to spot future weaknesses, or to be proactive about important decisions.

An unprecedented choice allows soldiers to use communications and intelligence assets in more meaningful ways.

Military operational decisions are moving further down the chain of the command, and a group of Stryker soldiers has taken a large step toward improving the training small units receive. Troops with this battalion had a chance to practice with capabilities never before available to them in an environment that simulates combat better than any facility they have at home. The results are new levels of preparation and confidence for whatever challenges they may be called on to handle next.

Home based in Hawaii, the 1st Battalion, 27th Infantry Regiment, 2nd Stryker Brigade Combat Team, 25th Infantry Division traveled to Korea to conduct the training event Wolfhound Maul. The unit is nicknamed the Wolfhounds. Contained within that effort was a smaller combined arms live fire exercise (CALFEX) that gave the unit’s three subordinate companies a chance to run full-spectrum operations with new assets in stressful surroundings. Maj. Christopher Choi, USA, operations officer for the battalion, says that as his team searched for the best resources to conduct the training it wanted to provide, it realized Korea offered benefits not available in Hawaii. After a careful cost-benefit analysis, decision makers chose to approve the travel to the Korean peninsula. Almost the entire battalion made the trip, with its companies experiencing the CALFEX training one at a time over a period of a month.

In many cases, haste makes waste as the U.S. Army wrestles with the inherent contradictions that emerged as it tries to speed new information technologies to warfighters.