Laboratory Kicks Research Up a Notch

The Blue Gene system, developed by IBM, has been installed at the ANL and will assist researchers and engineers with computations needed for more detailed simulations.

Computing capabilities to improve the scientific scale of problems tackled by a factor of 100.

Researchers at Argonne National Laboratory are crafting the building blocks for future technologies that will increase computing speeds, enhance collaboration and advance the fields of materials science and biology. Today, the facility that produced the first sustained nuclear chain reaction in 1942 continues to develop cutting-edge capabilities through its work with academia and industry. At the same time, it is employing some of the latest technologies to refine its modeling and simulations work that will affect advancements in energy, the environment and national security.

Argonne National Laboratory (ANL), Argonne, Illinois, has made substantial progress in the technologies that the laboratory itself uses in research and engineering while developing capabilities that will help other organizations advance their research and development work. Dr. Don Joyce, deputy laboratory director, the ANL, explains that some of the most exciting work is in the area of materials, which he describes as an overlap of computation, physics and the standard material science of biology. The laboratory is developing a new center, called the Center for Nanoscale Materials, that will utilize its advanced photon source, intense pulse neutron source, microscopy center and biology center to better understand proteins. “That understanding will drive work on self assembly further forward in the nano area because self assembly is a biological process that we’re mimicking,” Joyce relates.

“What we’re looking at in the materials level is more basic research than applied research. We are working to understand how to make nanomaterials rather than how they are going to be used. We hope and anticipate that, with a fundamental understanding, people are going to look at the research and say, ‘Aha!’ and apply it. And we will certainly be looking for that ourselves as well,” he adds.

The laboratory has been involved in biotechnology for some time, originally studying the effects of radiation on humans. Currently, the ANL features a robust structural biology group that can determine the structure of proteins very quickly and apply that information to a variety of areas. For example, if a person arrives at an emergency room with an illness, the molecular structure of the organism causing the illness could be analyzed using approaches such as the enhanced photon source so an antidote could be developed, Joyce explains.

Additional efforts in the biological arena include work with the University of Chicago. Together, the two organizations are building a regional bio-containment laboratory as part of their support of national security. This is a field that the laboratory plans to develop further, he says. “Quite a few organizations are involved in this area. Part of the difference between standard universities and us is that we are better structured to handle it. We are making sure processes, policies and procedures are followed, and we always know the cure for whatever bio-contaminant we are working on,” Joyce notes.

The laboratory is involved in other nanotechnology work as well as the ultra- nanocrystal diamond project focuses on the ability to grow diamond films with a uniformity that has never been achieved before on a nanoscale. This material would be used in microelectromechanical systems. Ultra-nanocrystalline diamond films are deposited on a silicon plate using a chemical vapor deposition method that Argonne developed. This results in freestanding diamond structures that are as thin as 300 nanometers. One simple application for this material would be the wear surfaces in pump seals; however, researchers at Argonne began the research because of what they believe will be a need in the future for more durable components for moving mechanical assemblies that could be used in microsatellites and spacecraft sent into deep space.

In the realm of fundamental physics, the ANL is currently competing to be the site for the rare isotope accelerator. Joyce explains that much information is available about elements that are considered common today. For example, radioactive materials have a half-life that ranges from fractions of a second to several years. But the half-lives of rare isotopes are billionths of a second, and although they are not commonly observed, there is plenty of evidence that they exist, he notes. “It is important to be able to measure the properties so we can better characterize them in the models of stellar evolution, for example. And, for practical applications regarding national security, the data would be important for weapons stockpile searches. How do you model weapons? Rare isotopes also may play a role there. It may not be the highest priority for the NSA [National Security Agency]; they have a lot to take care of. But they wouldn’t complain if someone measured it,” he states.

The type of research the ANL conducts heavily relies on significant advances in computational science, Joyce says, and Remy Evard, chief information officer at the ANL, relates that the laboratory has been involved in two projects that will help its own researchers as well as scientists worldwide. Evard is responsible for information systems at the ANL and is personally involved in research about high-performance computing and computational simulation.

In the computing space, Evard shares that the ANL deployed a Blue Gene system last December. “Blue Gene is a product out of IBM Research that not too long ago was acknowledged as the fastest computer in the world. Argonne is deploying one to focus on the research aspects of computational simulation. On the software aspects of it, we’re working with IBM and several other institutions to develop software that many sites could use,” he explains.

At ANL, Blue Gene is a testbed system that the laboratory plans to grow into a larger system and use for a range of computational activities in areas such as climate simulations and the modeling of stars, reactors and different types of infrastructure during emergencies. “We have close to 100 different kinds of scientific engineering applications running at the laboratory that are all getting use out of it,” he says.

Blue Gene is an order of magnitude faster and less expensive than comparable systems, Evard notes. This allows the ANL’s researchers and engineers to tackle larger problems, but rack space, power and staff requirements are significantly reduced, he says. “It sounds dull, but it’s enormous. … Because IBM is using a technology called system-on-a-chip, you can take a whole computer, put it on a chip then put multiple chips inside a rack. IBM has been very clever about how they’ve done this, and they’re working with the research community to modify the underlying software supporting the applications, and that’s what we’re doing,” he explains.

For example, the ANL is supporting work on the message passing interface (MPI), which is a standard interface used in modeling and simulation. Laboratory engineers are porting the MPI to Blue Gene to enable the use of several applications without making substantial changes. Because it is a platform that supports existing applications, it is much more cost-effective and will have a broad impact, Evard maintains.

A single rack of the Blue Gene system is comparable in capability to 20 racks of a standard IBM system. “This system doesn’t give us any new capability. It’s making the system the same physical size as the other computers we have but increasing computing by a factor of 20. I think with that we will be able to basically improve the scientific scale of the problems we work on by a factor of 100,” he explains.

One example of the type of computations the ANL scientists will be able to perform once the Blue Gene system is fully operational involves climate analysis. Today, this work is accomplished by dividing the Earth’s surface into a grid of squares. More advanced computing capabilities will allow researchers to reduce the size of these squares, resulting in better simulations. “This will give us the ability to improve the magnitude of what we’re doing by a factor of 20 or 100, depending on the simulation,” Evard notes.

The goal at the ANL in the computational area, which Evard thinks is the most exciting area at Argonne, is to install a petaflop system, which is 1,000 trillion floating point operations per second. The average today is 10 teraflops. “One rack we have today is theoretically 5 teraflops. If you were to go from a teraflop system to a petaflop system, you would increase your computing capability by a factor of 1,000,” he says. As a result, a computation that would take 17 years to complete today could be done in one week with a petaflop system, he explains.

ANL scientists are working with researchers around the world to accomplish this goal, and Evard and his associates expect the Blue Gene system to have a petaflop capability in the next three to four years. While IBM continues to develop the hardware as well as the software, the ANL and its partners are working to build the layer of software that will be situated between the operating system and other system software, he states.

In addition to increasing computational capabilities through improving individual systems, the ANL is heavily involved in helping build grid computing resources. The primary focus of the work is the Globus Toolkit. Developed by a consortium, the toolkit is the enabling technology used as the basis for most grid computing. It allows people to share computing power, databases and other tools securely online across corporate, institutional and geographic boundaries, and it includes software services and libraries for resource monitoring, data discovery and file management.

Globus is packaged as a set of components that users can employ either alone or in combination to develop applications. It was developed to facilitate collaboration while still allowing participants in the grid to control their access to their own resources.

A project called Access Grid, developed by the ANL’s Futures Lab, uses multimedia tools, interactive environments and interfaces to grid middleware and visualization environments that support collaboration across more than 150 institutions worldwide. It can be used in a range of ways—from large-scale conferences to individual meetings.

Grid computing environments will facilitate several new applications. For example, smart instruments such as electron microscopes, particle accelerators and wind tunnels could be coupled with remote supercomputers, users and databases to enable interactive online comparisons with previous work. Collaborative engineering could be accomplished with high-bandwidth access to shared virtual spaces that allow the interactive manipulation of shared datasets.

Evard explains that the current focus of the grid computing work is on determining how to use computers in the best possible way for science and engineering. “What is the most effective way to use data stored in one facility and a computer at another facility, and how do you do that in a secure way? How do you arrange it as a common knowledge place? That’s what’s driving the grid,” he says.

Joyce points out that work that involves increasing computational capabilities is extremely important to many of the laboratory’s projects because the ANL depends on simulations. “Making progress in any area relies on accurate simulation, and to do simulation, you need computational ability. In the fundamental physics area, and specifically nuclear physics, by doing simulations, we can better understand what experiments we need to do rather than doing experiments and then analyzing them, then doing another experiment that provides better focus,” he explains.

Web Resources
Argonne National Laboratory: www.anl.gov
IBM Blue Gene: www.research.ibm.com/bluegene
The Globus Alliance: www.globus.org