NIST Releases Better Bug Catcher
National Institute of Standards and Technology (NIST) researchers recently released a new and improved bug catching system designed to more efficiently find software glitches during the development process. The Advanced Combinatorial Testing System (ACTS) generates plans for efficiently testing every combination of six or fewer interacting variables rather than the more commonly used “pairwise” approach to testing software, which checks combinations of only two variables.
 NIST’s software for testing computer systems—ACTS—takes advantage of research that shows that virtually all software failures appear to be caused by six or fewer interactions. |
The new version includes an improved user interface and a better method of specifying relationships between parameters for testing. “The new version has better features for handling constraints in the system under test. For example, the user can specify that if parameter x is a certain value, parameter y must be a particular value. Or they could specify that parameter y is not a certain value,” explains Rick Kuhn, a NIST computer scientist who helped develop ACTS. These constraints can be complex, but a simple theoretical example is that one application may run out of file space and another may use an input that is too long. If those two things happen independently, they each simply trigger an error message, but if they happen simultaneously, the entire system could cash.
Kuhn says the U.S. Air Force 46th Test Wing at Eglin Air Force Base, Florida, has used ACTS in their test and evaluation work and are now rolling out the methods to other Air Force organizations. In one case, the Test Wing had a system with a large number of variables provided by multiple vendors. Variables included voice, video, Global Positioning System, range finders and communications links. Concerned that plugging in the various technologies in the wrong order might crash the system, the Air Force tested as many combinations as possible for plugging in the array of technologies in various orders. “We compressed as many sequences of events together as possible in as small a number of tests as possible. That wasn’t too complex,” says Kuhn.
Lockheed Martin Corporation also is using ACTS to test some of its own software under a Cooperative Research and Development Agreement recently signed with NIST. The two organizations will jointly publish a research paper on the project once it is completed. “That will be a real nice contribution from them to the field to make sure these methods are useful. In terms of tech transfer, we need valid data from real-world applications, not just some guys in a government research lab recommending people use this,” says Kuhn.
NIST first released the ACTS in 2008 and distributed it freely to 465 organizations and individuals in government, industry and academia. Current users are primarily from the information technology, defense, finance and telecommunications sectors. Potential users can request ACTS online.
NIST reported in 2002 that software bugs cost the economy nearly $60 billion even though 50 percent of software development budgets are devoted to testing. Testing every possible variable is not practical. A system with 34 on and off switches, for example, would require 17 billion tests. NIST researchers have found, however, that between 70 and 95 percent of all system crashes are caused by only two interacting variables, and practically 100 percent of software failures are caused by no more than six. Using ACTS, all three-way interactions can be evaluated using only 33 tests, all six-way combinations with 522 tests, rather than 17 billion.
Along with the new version of ACTS, the NIST team has released an electronic textbook tutorial, Practical Combinatorial Testing, that introduces key concepts and methods for testing combinations of variables that may affect software. It also explains the use of software tools for generating combinatorial tests. The textbook is designed for undergraduate students in computer science or engineering, and Kuhn says he believes it is the first textbook on the subject.
