Configuration Files

AnkrPt Pairwise uses simple csv files to define how tests generate. These files contain up to four (4) different sections as described below.

Sections

Section Required Description

CRITERIA

yes

identifies the criteria used to generate tests

NEGATIVE CRITERIA

no

defines negative criteria used to generate negative tests

REQUIRED TESTS

no

identifies specific tests to generate. All fields need not be included.

CONSTRAINTS

no

define limitations to be enforced during test generation

Basic Rules

There are some basic rules when editing your csv (comma-separated values) files:

  • section keywords are uppercased and define the beginning of each section,

  • each section can only occur once,

  • section order is vital as each section builds upon previous ones,

  • only the CRITERIA section is required; all others are optional,

  • all values must be separated by a delimiter (regional defaults),

  • if you don’t want the regional delimiter, a runtime parameter lets you override it,

  • you can edit with Excel but must retain the csv structure; we do not support native Excel,

  • blank lines are ignored so can be used for readability.

CRITERIA Section

The Criteria section defines the valid data used to generate tests. The structure of the Criteria section is:

CRITERIA (1)
Name, value1, value 2, "value3", ... (2)
1 CRITERIA tag identifying the start of the Criteria section.
2 a criteria entry with the criteria name followed by a list of valid criteria. The name is in blue below.

Let’s see an example of a CRITERIA section with valid criteria.

criteria example
Figure 1. CRITERIA section example
Please refer to Using Excel for additional considerations for working with csv files in Excel.

This structure makes it easy to easily define your requirements while still allowing complex testing conditions. Some basic rules for Criteria are:

  • The Criteria section starts with the CRITERIA tag.

  • There must be one, and only one, Criteria section.

  • Each criteria row defines the criteria name (in blue above) and corresponding valid values for that criteria.

  • At least two criteria are required.

  • Each criterion can only be defined once. If criteria are defined multiple times, you must merge them.

  • Each criterion must have at least two values with all values unique for a given criteria.

  • All names and values are automatically trimmed (removal of leading and trailing spaces).

  • To use the delimiter within values, use quotes as per normal csv processing.

  • Blank lines are ignored.

NEGATIVE CRITERIA Section

It is not enough to just do 'happy path' testing, you should do negative testing as well to ensure the system is stable even with harmful parameters. AnkrPt Pairwise does this with NEGATIVE CRITERIA as follows:

NEGATIVE CRITERIA (1)
Name, value1, value 2, "value3", ... (2)
1 NEGATIVE CRITERIA tag identifying the start of the Negative Criteria section
2 each row identifies the name of a valid criterion followed by a list of invalid values

Let’s see an example of a NEGATIVE CRITERIA section with invalid criteria.

negative criteria example
Figure 2. Negative Criteria Example

It is similar to the CRITERIA section except for negative values, and you must refer to an existing criteria name from the CRITERIA section. Some basic rules for Negative Criteria are:

  • The Negative Criteria section starts with the NEGATIVE CRITERIA tag.

  • There must be one, and only one, Negative Criteria section.

  • Each negative criteria defines a criteria name and corresponding negative values.

  • Each criterion must reference an existing valid criteria from the CRITERIA section.

  • You cannot define a negative value if it is already a valid value for that name.

  • Each negative criteria are optional.

  • All names and values are automatically trimmed (removal of leading and trailing spaces).

  • To use the delimiter within values, use quotes as per normal csv processing.

  • Blank lines are ignored.

By default, Pairwise generates as many negative test cases as there are negative values (one invalid value per Test Case). But you can change this to condense multiple negative tests into as few tests possible producing tests with multiple negative conditions on a single row.

REQUIRED TESTS Section

Generating pairwise tests ensures all pairs are tested, but not necessarily a specific test that you need. Required tests are a quick and easy to ensure a specific test case is produced. Let’s look at the structure.

REQUIRED TEST (1)
ID, Name1, Name2, Name3, Name4, Name5,... (2)
1,value1,,,,value5,... (3)
1 REQUIRED TEST tag identifies the start of the Required Test section.
2 A label line making it easier to match your fields to the required value. The first column is a "Test ID" column.
3 The id of the required test followed by values for each name.

Let’s see an example of the REQUIRED TEST section.

required tests example
Figure 3. Required tests example

Required tests are a little different from the Criteria sections but refer to those sections for data. Some basic rules for Required Tests are:

  • refer to values defined in either the CRITERIA or NEGATIVE CRITERIA sections.

  • undefined criteria are filled with criteria values during generation.

  • at least three criteria are required otherwise you’re just doing pair generation which is already done.

CONSTRAINTS Section

Constraints are an extremely important and powerful part of AnkrPt Pairwise. They let us define limits while building test cases, making it possible to ensure proper tests. Constraints use the following structure:

CONSTRAINTS (1)
When [criteria] is/isIn/isNot/isNotIn [value] (2)
And/Or [criteria] is/isIn/isNot/isNotIn [value] (3)
Then [criteria] is/isIn/isNot/isNotIn [value] (4)
And [criteria] is/isIn/isNot/isNotIn [value] (5)
1 CONSTRAINTS tag identifying the start of a Constraints section.
2 When defines the beginning of a constraint
3 And and Or define additional criteria for the constraint
4 Then is the beginning of the Then section for its associated When
5 And defines additional Then criteria. Or is not allowed with Then.

Let’s see an example.

constraints example
Figure 4. Constraints example

Some basic rules of the Constraints section are:

  • every constraint requires both a When and Then statement.

  • Or is not allowed with the Then clause.

  • And and Or follow basic programming logic with Or tested first followed by And.

Constraints are built with keywords, criteria, and values (including negative ones) based upon basic logical conditions using a "When this Then that" structure.

Keywords

Table 1. Constraints keywords

Keyword

Description

When

defines the beginning of a Constraint

Then

defines the beginning of the Then clause

And

logic operator to And your criteria

Or

logic operator to Or your criteria

Is

lets you define a single value that the condition must match

IsIn

defines multiple possible values that the Constraint must match one of

IsNot

defines a single value which the Constraint cannot be

IsNotIn

defines a list of values which the Constraint cannot match any of

  • THEN conditions should be based on explicit requirements.

  • WHEN and THEN are defined once per Constraint; usage of both is mandatory.

  • you must use defined criteria.

  • REQUIRED TESTS are less important than CONSTRAINTS - it is required to resolve conflict before test generation.

  • OR is not allowed to define multiple values for the same criteria, use isIn or isNotIn instead.

It’s essential to use CONSTRAINTS wisely otherwise you might accidentally eliminate some needed pairs from test generation. You should verify the dependencies between constraints to avoid the risk of omitting essential pair combinations.
To better understand the impact of your constraints, pairs not considered valid for tests is provided in the generation summary.

Putting it all together

Now let’s put everything together to see a complete sample Criteria csv file.

all together
Figure 5. Putting it all together