CatWalk User Guide

Launch CatWalk

The Java bundle for CatWalk is an executable JAR file that launches the tool out-of-the-box. You should see this (click on any image to zoom):

CatWalk welcome screen

Load the Test Class

In the Test Class panel, make sure that the Binary Location is pointing to the root binary folder for the compiled class files that you want to test. This may be called C:/Eclipse/MyProject/bin in Eclipse.

In the Test Class panel, then select the Java package tree leading to the class you want to test. Make sure that the Qualified Name only refers to the full package-qualified name of the test class.

Launch the Locations dialog in the Custom Settings panel, and decide where you wish to save any oracle files. By default, this is under the directory in which you launched CatWalk.

Protocol analysis of a BankAccount

Pick the Test Settings

In the Test Settings panel, the Convention options are:

• STATIC to test a static class, or the static API of a normal class with some static methods;
• NORMAL (the default) to test a normal class or enum type, ignoring inherited Object methods;
• CUSTOM also including equals(), hashCode(), toString(), getClass() from Object;
• COMPLETE also including all standard system methods inherited from Object.

In the Test Settings panel, the Strategy options are:

• EXHAUSTIVE to explore sequences consisting of all interleaved method combinations (costly);
• MUTATING to extend only those sequences that modify the state of the test class (better);
• ALGEBRAIC (the default) to extend only those sequences that reach a new state on this path;
• METAMORPHIC to extend only those sequences that reach a state not seen on any parallel path.

Algebraic exploration of a BankAccount

In the Test Settings panel, the Modality options are:

• INSPECTION statically analyse the test class's public method API;
• SIMULATION also dynamically analyse the class's algebraic structure;
• EXPLORATION (the default) also execute a test suite according to the chosen Strategy and Depth;
• VALIDATION also create a test Oracle, in which you will be asked to confirm selected test outcomes;
• GENERATION also export a JUnit test suite (see more details later in this guide).

In the Test Settings panel, the Test Depth spinner sets the maximum test sequence length (default value 3).

For test classes with very large APIs, start low and increase as time and memory permit. Depths of 6-7 are reachable in a class with 10 public methods. Under some test settings, you may explore all possible tests.

Exhaustive validation of a BankAccount

Run CatWalk

Once you have loaded a test class and picked your test settings, it's time to run your tests.

In the Test Execution panel, hover the mouse over Wilfrid the Cat. If he raises one paw, then you're ready to go. Click on the Cat, and he'll start walking. So long as the Cat is walking, the tool is operating. Once he stops, then the tool has finished, and you can view the results.

If ever Wilfrid slows down, or jitters in his gait, then the tool is approaching memory exhaustion. This need not be a problem - the tool may recover. If memory runs out, it will display all results obtained so far. You may choose to abort a test run by clicking on Cancel.

View the Results

In the large Results tabbed pane, you may view the results of a test run. Click on the appropriate tab to see the following:

• Protocol Analysis to see the public constructors and methods of the test class, or the public constants and methods of an enumerated type;
• Algebraic Analysis to see the primitive, transformer and observer operations of the test class. Primitives visit a new state. Transformers revisit an old state. Observers just inspect state.

Algebraic analysis of a BankAccount

• Test Cycle #0 to see the results of executing all constructors, under all distinct input partitions;
• Test Cycle #1 to see the results of executing all constructors and method sequences of length 1, under all distinct input partitions;
• Test Cycle #N to see the results of executing all constructors and method sequences of length N, under all distinct input partitions;
• Histogram Analysis to see how many input partitions were found for each operation; and how frequently partitions of each size were found;
• Test Summary to see how many tests executed nominally or raised exceptions; how many tests were passed, or failed, or unchecked; how many distinct tests were picked from the total sequence permutations and input value-populations.

Histogram analysis for a BankAccount

Test Summary for a BankAccount

Export to JUnit

If you wish to export a JUnit test suite, there are a few more things to set up. First, ensure that your Java project includes the JUnit library and CatWalk as an external library (otherwise, the generated JUnit test-drivers will have missing Java dependencies).

In the Locations dialog, make sure that the Source Location folder is pointing to the root source folder for the project you want to test. This may be called something like C:/Eclipse/MyProject/src in Eclipse;

In the Locations dialog, select the package under your source root where you want JUnit files to go (otherwise, test drivers will be added to the test class package). Make sure that the Package Name only refers to a package directory tree. After this, you can safely use the Modality: GENERATION.

Pushing CatWalk to memory limits

Configuration

Launch the Configuration dialog in the Custom Settings panel to review rarely-changed configuration options.

• Sampling: Switch between the LINEAR, FIBONACCI and GEOMETRIC input sampling methods. Linear is slow, Fibonacci is default and Geometric is fast, when reaching for the limits of an input space.
• Max Inputs: Set the maximum number of sampled inputs for each method argument, within which range you expect to reveal all distinct behaviour. By default, this is set low at 20.
• Tree Depth: Set how deeply to compare object states; 0 is shallow equality, 1 is one-reference-deep object-tree equality, etc.
• Probe Depth: Set how long probing test sequences need to be to analyse the algebraic properties of the test class.
• Max Results: Set how many distinct return values you wish to treat as qualitatively different behavioural responses.
• Max Failures: Set how many distinct raised exceptions you wish to treat as qualitatively different behavioural responses.
• Min Regularity: Set how soon to start predicting a regularity linking an input with a given result. The default is after two occurrences.
• Trivial Testing: Uncheck the box to stop testing methods with trivial null argument values, for one of their input partitions.