Pairwise Testing – Why it Matters
Every company or person wanting to prove something needs testing to prove it. Whether you’re developing software, verifying the accuracy of mechanical devices, or confirming the effectiveness of medical breakthroughs, you’re testing. But to test effectively, you need to answer some basic questions: what to test, how to test, how much to test, and when to start. Let’s answer those questions.
What to Test
Why not test every possibility? We all know that testing everything is impossible except in the most simple cases. For example, 30 fields with 3 values each doesn’t seem that complicated, but testing everything would require over 205 trillion (with a “t”) tests! Pairwise can do this in just 24 tests. What is Pairwise?
Pairwise testing is based on the well-established principle that most defects trigger with either a single input parameter or an interaction between pairs of parameters. Pairwise testing then “simply” pairs all your test criteria into as few tests as possible. Let’s see what this means with two simple examples using letter/number combinations to make it easier to read (you can substitute any value for those below; the process is the same.)
Sample 1 – 3 criteria of 3 values each
| Criteria | Values | ||
|---|---|---|---|
| a | a1 | a2 | a3 |
| b | b1 | b2 | b3 |
| c | c1 | c2 | c3 |
| Pairs to be Tested – 27 | |||||
|---|---|---|---|---|---|
| a1, b1 | a1, b2 | a1, b3 | a1, c1 | a1, c2 | a1, c3 |
| a2, b1 | a2, b2 | a2, b3 | a2, c1 | a2, c2 | a1, c3 |
| a3, b1 | a3, b2 | a3, b3 | a3, c1 | a3, c2 | a3, c3 |
| b1, c1 | b1, c2 | b1, c3 | b2, c1 | b2, c2 | b2, c3 |
| b3, c1 | b3, c2 | b3, c3 | |||
Highlighted pairs are for reference between Sample 1 and Sample 2 tests.
Tests for all possibilities – 27
| Pairwise Tests – 9 | ||
|---|---|---|
| a1, b1, c1 | a1, b2, c2 | a1, b3, c3 |
| a2, b1, c3 | a2, b2, c1 | a2, b3, c2 |
| a3, b1, c2 | a3, b2, c3 | a3, b3, c1 |
Sample 2 – 3 criteria of 3 values each, 1 criteria of 5 values, and 3 criteria of 2 values each
| Criteria | Values | ||||
|---|---|---|---|---|---|
| a | a1 | a2 | a3 | ||
| b | b1 | b2 | b3 | ||
| c | c1 | c2 | c3 | ||
| d | d1 | d2 | d3 | d4 | d5 |
| e | e1 | e2 | |||
| f | f1 | f2 | |||
| g | g1 | g2 | |||
Pairs to be tested – 168 (similar to 3×3 process)
Tests for all possibilities – 1,080
| Pairwise Tests | ||
|---|---|---|
| a1, b1, c1, d1, e1, f1, g1 | a1, b1, c1, d4, e2, f2, g1 | a1, b2, c3, d3, e2, f2, g2 |
| a1, b3, c2, d2, e1, f1, g2 | a1, b3, c3, d5, e1, f1, g1 | a2, b1, c1, d5, e2, f2, g2 |
| a2, b1, c3, d2, e1, f2, g1 | a2, b2, c2, d1, e2, f2, g2 | a2, b2, c2, d4, e1, f1, g2 |
| a2, b3, c1, d3, e2, f1, g1 | a3, b1, c2, d3, e1, f2, g1 | a3, b2, c1, d2, e2, f1, g2 |
| a3, b2, c2, d5, e2, f2, g1 | a3, b3, c3, d1, e2, f2, g1 | a3, b3, c3, d4, e1, f1, g2 |
Note that Sample 1 reduces your tests from 27 down to 9, while Sample 2 reduces your tests from 1,080 to just 15!
Adding more criteria with varying values complicates things even more, making it impossible to produce these tests on your own. But technology can produce them for you.
How to Test
“How to test” refers to testing manually or with automation. Regardless of which, the benefit is the same: fewer tests mean less time, thus reduced cost for both people and hardware. All while improving your coverage. As simple as that, buts let’s dive a bit deeper.
Manual testing takes a person’s time, our most valuable commodity. Some testers are faster, while others may be more accurate, but time is the ultimate bottleneck. To combat this, we often reduce testing or eliminate regression testing once tests pass. Then fill that time with more random and exploratory testing, but how much confidence do we have in those tests?
Pairwise testing eliminates this randomness with a strict requirement: test every pair. The level of confidence gained from this is not attainable with random and exploratory testing, so pairwise testing both reduces testing effort and increases confidence through increased coverage.
Automated testing benefits in the same way. Even though hardware costs aren’t as high as person costs (people are slower and more expensive), hardware costs are real. But the more significant advantage is the shortened execution windows. Continually adding regression tests should be a natural part of development, but more tests mean more time needed to run the tests. Reducing that growth keeps the execution windows shorter, hopefully within your physical constraints.
How Much to Test
We need to find the balance between enough testing and the effort required to test. While Pairwise helps generates every test needed for pairs, this isn’t always good. You likely need to add required tests for complicated scenarios and constraints to eliminate invalid tests. This is different from random and exploratory testing because you add known conditions to generate specific test cases. Read our upcoming blog, “Pairwise Testing – Choose Your Data Wisely,” to learn more.
Ultimately, you decide how much to test. But isn’t it better to know the principles behind your testing strategy rather than just random testing and hoping for the best?
When to Start
The sooner you start, the sooner you’ll gain the advantage. While there is some effort to define the criteria, it’s not onerous and quickly reduces your testing effort while improving coverage. Since it’s easy to update test criteria as needs change, starting early is never a wasted effort. You get value when you use pairwise, so why not start now?
And the Savings Are?
The ultimate goal is to save money; reducing effort while increasing quality does both. Let’s take a couple of examples and assume that 100 tests provide ample coverage for each scenario (not possible). Let’s also assume that each test takes 10 minutes to complete.
Scenario 1 – Web
A website test with 16 different parameters (as shown below), some required tests, and several constraints. Testing this produces over 759 pairs and more than 4.4 million possible combinations.
| Field | Value |
|---|---|
| Device | Mobile, Desktop |
| Mobile OS | iOS, Android, n/a |
| Mobile Browser | Chrome, Android, Opera, Safari, n/a |
| Mobile Connection | edge, 4g, wifi, n/a |
| www in front | yes, no |
| Protocol | http, https |
| Cookies | enabled, disabled |
| IP Version | IPv4, IPv6 |
| Desktop OS | Windows, Linux, Mac, n/a |
| Desktop Browser | Chrome, Firefox, Internet Explorer, Opera, Safari, n/a |
| Desktop Connection | wifi, cable, n/a |
| Server Caching | on, off |
| Server Minification | on, off |
| Window Dimension | wide, tall |
| Window Size | large, small |
| Language | English, non-English |
Scenario 2 – Large
More criteria can grow things exponentially. Consider 60 parameters with 2, 3, and 4 values spread evenly. This results in 15,910 pairs with more than 4*10^27 possible combinations (I can’t comprehend that number).
As described above, testing all possibilities or manually creating pairwise tests for either scenario above is impossible. But pairwise generation produces a minimum number of tests while guaranteeing maximum coverage with the following results:
| Scenario | Params/ Values | Pairs | Cartesian Tests | Pairwise Tests | Reduced from 100 | Time Saved |
|---|---|---|---|---|---|---|
| Web | 16 / 38 | 759 | 4.4 Million | 26 | 74 | 12+hrs |
| Large | 60 / 180 | 15,910 | 4*10^27 | 34 | 66 | 11 hrs |
Scenarios 1 and 2 each reduced the test count significantly while guaranteeing paired coverage. And we save between 11 and 12 hours every time we run either suite based on 100 tests, each taking 10 minutes to run.
How Do I Get Started?
Download and start using AnkrPt Pairwise today, for free!
Watch for our upcoming articles Pairwise Testing – Choosing Your Data Wisely, andPairwise Testing – By The Numbers, to get into more detail and learn how to better take advantage of Pairwise testing.
“Maximizing coverage while minimizing tests” to “Maximizing confidence while minimizing cost,” a true win-win. AnkrPt Pairwise.
Enjoy, and happy testing!
Kim Filiatrault
Automation Enthusiast
Watch or read our other posts at Kimputing Blogs. You’ll find everything from Automated testing to CenterTest, Guidewire knowledge to general interest. We’re trying to help share our knowledge from decades of experience.
