Effective Test Data Selection

While it is easy to define criteria and generate tests, remember that pairwise is a way to maximize coverage while minimizing tests. It is crucial to carefully select criteria which adequately covers all desired combinations of values while minimizing the generated tests. Using values that don’t change test coverage of the system may unnecessarily increase the number of unnecessary cases.

Example 1

For example, 70 values distributed across parameters in two different ways can produce different results:

  • 10 params with 7 values each (10*7) generates 86 tests (Cartesian > 282 Million)

  • 7 params with 10 values each (7*10) generates 146 tests (Cartesian of 10 Million)

More doesn’t always mean more

Though it seems counterintuitive, adding more parameters or more values does not always mean more tests. Unfortunately, there is no formula to determine how many tests will be generated from criteria, the only way to know is to generate. But, you can know the fewest number of tests that will be generated; just multiply the size of the two largest columns together.

Example 2

For example, 7 parameters in one set, and those same 7 criteria with an additional 25 criteria.

  • 2 parameters of 10 values each and 5 parameters of 2 values each generates 100 tests (Cartesian of 3,200)

  • those same 7 parameters plus 25 more of 2 values each still generates 100 tests
    (Cartesian > 3 Billion)

What does this all mean? That revealing the highest percentage of defects at minimum cost (minimum test cases with maximum test coverage) requires careful analysis of the values' scope.

  1. use the smallest possible value set per criteria name to provide coverage

Using values that don’t change test coverage of the system under test significantly increases the number of useless cases. The better you understand the data, the easier to define parameters that ensure all crucial test coverage parameters are defined. And that you aren’t generating excessive and unnecessary tests.

It is important to remember that data selection is crucial to minimize testing.

Read more about Pairwise on Kimputing Blog!