Studying software crashes in a variety of applications from medical devices to Web browsers, from the National Institute of Standards and Technology (NIST) researchers obtained hard evidence to support long-held conventional wisdom: most software failures result from simple events rather than complex ones.
Even for Web browsers containing hundreds of different variables, most failures were caused by interactions between just two variables. Nonetheless, in the applications that the researchers studied, additional failures could result from interactions of up to six variables.
Based on that insight, NIST together with the University of Texas, Arlington went beyond the popular practice of “pairwise testing,” or exploring interactions between only two variables at a time, and designed a method for efficiently testing different combinations of settings in up to at least six interacting variables at a time. Their technique resembles combinatorial chemistry in which scientists screen multiple chemical compounds simultaneously rather than one at a time.
For example, imagine a word-processing program that features 10 different text formats. Certain combinations of settings (such as turning on superscript, subscript and italics at the same time) could cause the software to crash. Trying all possible combinations of the 10 effects together would require 1,024 tests. However, testing all possible combinations of any three effects requires just 13 different tests, thanks in part to the fact that if the tests are selected judiciously the 10 different variables allow you to explore 120 combinations of “triples” simultaneously.
The new tool generates tests for exploring interactions among the settings of multiple variables in a given program. Compared to most earlier combinatorial testing software, which has typically focused on testing interactions between just two variables, the tool excels at quickly generating efficient tests for 6-way interactions or more.
The researchers plan to release the tool early next year as open-source code.
MEDICA.de; Source: National Institute of Standards and Technology (NIST)