JWalk software testing tool suite

Lazy systematic unit testing for agile methods

You are here: JWalk Home / User Guide /
Department of Computer Science

The JWalk User Guide

JWalkTester exercising the Full state

The JWalk 1.1 Tool Suite comes with two finished testing tools, JWalkTester, the flagship GUI-based unit testing tool, and JWalkUtility, the classic command-line driven unit testing utility. Also, the unbundled software contains the JWalker tool kit for building custom JWalk applications. Familiarise yourself with the common JWalking concepts below, then refer to the specific guides for each tool or toolkit (see margin).

JWalking Concepts

What is JWalking? Well, it is a fast and effective way to do Java unit testing. All the tools need is a compiled Java class to test, and some indication of what test strategy to follow. You may inspect the test class's interface, explore its algebraic structure or state-space, exercise sequences of methods to examine its visible behaviour, or train a test oracle to recognise correct or incorrect test outcomes. Once a test oracle has been trained, testing is fully automatic. Also, if the test class is modified and recompiled, the oracle can be re-trained in a short time. Oracles can also learn from superclass oracles, but adapt quickly to the subclass, learning about combinations of local and inherited methods. The important thing is that the tools determine the right method sequences that need to be tested, and explore the test class completely, to bounded depths.

Test Class

One unit, a compiled Java class, may be tested at a time. This is known as the test class, which must always be supplied by the tester and uploaded by the testing tool. Other test parameters have suitable default settings, which the tester may choose to override. The test class could be a single small component, or a large subsystem delegating to other classes, in which case tests may also invoke methods on the connected classes. If you want to replace production classes with stub- or mock-objects, then refer to the section on how to create Custom Generators for JWalk.

The test class may exist in the default package, or in a package under the current test class directory (by default, the current working directory), or in some other package known to the Java CLASSPATH. The test class must always be loaded using its package-qualified name: if you get the path wrong, the tools will report failure to find or to load the test class. The test class directory may be changed in some tools.

Test Strategy

The exploration of the test class can follow different test strategies, which range from the brute force breadth-first exploration of all method combinations, to focusing on primitive algebraic constructions that reach all low-level concrete states, to the sophisticated analysis of high-level states and transitions.

  • protocol strategy: creates test sequences corresponding to every possible interleaved ordering of the test class's public constructors and methods (all method protocols);
  • algebra strategy: creates test sequences corresponding to every novel construction, that is, sequences which drive the test object into different concrete states (all algebraic constructions);
  • states strategy: creates test sequences corresponding to the state cover, transition-cover and n-switch transition cover, which drive the test object through its high-level state machine (all states and transitions).
The algebraic structure is detected using low-level object state comparison, which identifies when methods change state, leave state unchanged, or cause an object to revisit an earlier state. The high-level abstract states are detected using natural state predicate methods supplied by the test class. For example, if a Stack class provides predicates isEmpty() and isFull(), three states will be discovered: {Empty, Default, Full}. Some test classes may only have a single Default state, whereas others may have only named high-level states.

Test Modality

The different test modalities control whether you wish merely to inspect or analyse the test class, or also explore its behaviour by executing different test sequences, or also validate its behaviour by training a test oracle, which is later saved in an oracle data file:

  • inspect modality: performs a static analysis (and optionally a dynamic analysis) of the test class, to reveal the public API, or algebraic structure, or state space;
  • explore modality: also exercises the test class by generating and executing test sequences according to the current test strategy, displaying results for the tester to view;
  • validate modality: also validates the outcomes of each test sequence, comparing the actual result against the result predicted by a test oracle, which is trained interactively and saved when testing is over.
A test oracle is saved in, and reloaded from, an oracle data file, which is placed in the oracle directory (by default, the test class directory). The file is a plain text file, whose name consists of the short name of the test class, followed by the extension: .jwk. For example, an oracle for the class org.jwalk.test.Stack will be saved in the file Stack.jwk in the current oracle directory, which can be changed in some tools.

Depth Settings

The testing algorithms used by JWalk perform bounded exhaustive exploration of the test class's state space. All exploration and validation is carried out to a bounded depth. Furthermore, the initial dynamic analysis, and the comparison of object states, are also affected by custom depth settings.

  • testDepth: controls the maximum length of test sequences generated during exploration (and validation) - test sequences of every length up to this limit are exercised, counting initial object construction (or a state-cover sequence) as depth zero;
  • probeDepth: controls the maximum length of probing sequences generated during the initial dynamic analysis (in certain strategies) - set this higher to discover missing states and lower to avoid memory exhaustion;
  • stateDepth: controls how deeply objects are examined when their states are compared for equality - set this to zero for shallow equality and higher for deeper equality (of object trees), especially if a mutable object structure appears not to change state.

Value Generators

The JWalk strategy algorithms compute what method sequences to execute. Before execution, each constructor or method must be supplied with suitable test inputs. These are supplied by objects known as Generators, which are capable of synthesising values of given Java types. Generators must synthesise values in an exactly repeatable, deterministic order (rather than randomly). This allows the JWalk tools to learn the correct test outcomes, when the same test inputs are presented in subsequent test cycles. There are two kinds of Generator, described by the Java interfaces:

  • MasterGenerator: produces a standard series of quasi-unique test values (and objects) in a monotonically increasing sequence;
  • CustomGenerator: produces a custom series of test values (and objects) under the explicit control of the tester.
The tester may supply their own CustomGenerators, to control the context of testing and the order in which test values are presented to the test class. For more details, refer to the section on how to create Custom Generators for JWalk. CustomGenerators are compiled Java classes, loaded from the current generator directory (by default, the test class directory), which may be changed in some tools.

Convention on Inheritance

The JWalk strategy algorithms exercise all public methods of the test class, including those inherited from ancestor classes, all the way up to the root Object class. However, interleaving all of Object's built-in methods is sometimes a nuisance, so these methods may be selectively excluded by convention. There are three conventional settings:

  • standard - exclude all of Object's methods
  • custom - include some of Object's methods
  • complete - include all of Object's methods
In the custom setting, the following four of Object's methods are included: equals(), hashCode(), toString() and getClass(). If the test class is in fact the root class java.lang.Object, this overrides any convention, and all of its methods are included.

Standard Files and Directories

The JWalk tools need to read and write certain data files during their normal operation. These files are few in number, known to the user, and exist in standard locations under the installation directory. The files that JWalk routinely reads and/or writes are the following:

  • the JWalk license file, JWalkLicense.txt, which must live in the working directory from which the tools are launched;
  • the oracle data file, which is created for each test class when the tools are run in validation mode, and exists either in the test class directory, or in a separate oracle directory specified by the tester;
  • the redirected input file, input.txt, which is created for each test class directory, if input is redirected by uploading the custom RedirectInGenerator.
Of these, the oracle data file and the redirected input file may be safely deleted, when no longer required. The oracle data file may be edited by hand (so long as the format is respected) to cause JWalk to learn or forget parts of a specification. The redirected input file may be edited to reflect the console input required by the test class. The JWalk license file must not be changed in any way and may not be transferred, except to a new working directory on the same machine.

Indiscriminate File Creation

Warning: If the JWalk tools are used in an unguarded way, it is possible to excercise user-defined classes that create or destroy files and directories indiscriminately. If user-defined classes have methods that accept File or Stream parameters, then the JWalk tools will attempt to create instances of these. In principle, exercising an object that creates or deletes files and directories could cause arbitrary havoc to the filesystem, but in practice, this is mitigated in two ways:

  • all created directories and files should appear below the current working directory, and have automatically generated names corresponding to the conventional strings emitted by the JWalk tools;
  • the tester may take control of how and where files and directories are created, by supplying a CustomGenerator that intercepts the creation of File and Stream types.
We have tested the worst consequences of this by exercising Java's File class exhaustively, to depth 15. This causes the creation of a directory named aubergine under the current working directory, which is then populated by hundreds of temporary files, with strange synthesised names. Apart from the time taken to create so many files, no long-term damage was suffered and later this directory was removed.

JWalkTester analysing a ReservableBook

JWalking Method

The following describes the general approach followed when exploring and testing a prototype class using the JWalk 1.1 Tool Suite. This method applies both to the finished testing tools JWalkTester and JWalkUtility; and to any application built using the JWalker tool kit.

Inspecting the Test Class

After loading the test class, you may choose to inspect it. Depending on the test strategy chosen, you will obtain a different analysis:

  • protocol inspection: performs a static analysis of the test class, showing its public constructor and method interface (if the test class is an enumerated type, its constants are displayed);
  • algebra inspection: also performs a dynamic analysis of the test class's algebraic structure, classifying its operations into primitive, transformer and observer operations;
  • state inspection: instead, performs a dynamic analysis of the test class's high-level state space, using the natural state predicates supplied by the class to help identify states.

To illustrate algebraic analysis, a ReservableBook class provides a number of methods, which are classified. The initial constructor and methods issue() and reserve() are classified as primitive, because they drive the book into all of its different concrete states. The methods discharge() and cancel() are classified as transformers, because they return the book to an earlier state. The methods getBorrower() and isOnLoan() are classified as observers, because they do not modify the book's state.

To illustrate high-level state analysis, if a ReservableBook class provides the two state predicate methods isOnLoan() and isReserved(), then the state detection algorithms will find four high-level states: {Default, OnLoan, Reserved, Reserved&OnLoan}, by seeking to reach concrete states in which each predicate returns true or false. The Default state is the one in which no predicate returns true. The other states are named after the predicates which return true. The shortest test sequences reaching these distinct high-level states are collected in the state cover test set, which forms the starting basis for state-based testing (see below).

Exploring the Test Class

Next, you may wish to explore the behaviour of the test class in some detail. You might do this to check that you haven't overlooked interesting combinations of methods that might yield unusual results. You will start with a low testDepth setting, and increase this as you grow confident that the shorter sequences behave correctly. The results you get depend on the test strategy chosen:

  • protocol exploration: executes all constructors followed by all interleaved combinations of methods (including inherited methods), producing some unusual sequences in which the same methods are repeatedly invoked, or are invoked in unexpected orders;
  • algebra exploration: executes all constructors followed by only those methods (including inherited methods) that drive the test object into new concrete states, producing sequences that grow the test object and observe its behaviour at the edges;
  • state exploration: executes the state cover (see above) extended by all interleaved combinations of methods (including inherited methods), producing a set of test results for each high-level state, explored to each depth.

Algebraic exploration is perhaps the most compact strategy to use at first, when investigating all the expected behaviours of the test class. For example, the prototype ReservableBook is explored in this mode, revealing an interesting combination at testDepth 3:

   ReservableBook target = new ReservableBook();
         target.reserve(Borrower Borrower#1);
         target.issue(Borrower Borrower#2);
      ==> Borrower#2 : NORMAL  (unchanged)
which fails to take into account the business rules of reservation. The programmer may quickly fix the code so that books may only be issued to the borrower that reserved them.

Protocol exploration is good for identifying boundary cases like null operations and for checking that observer-methods do not unexpectedly modify the object's state. For example, when the ReservableBook is explored, this yields some interesting sequences that include:

   ReservableBook target = new ReservableBook();
      ==> void : NORMAL  (unchanged)
This makes the programmer think about what should happen if a book is discharged, before being issued on loan to any borrower. Shoud this be ignored (a nullop), or should it raise an exception?

State exploration is good for ensuring that an object behaves robustly in all of its high-level states. For example, a resizable Stack is explored, and the following unexpected exception is raised, when methods are exercised in the Stack's Full state:

   Stack target = new Stack();
         target.push(Object Object#1);
         target.push(Object Object#2);
         target.push(Object Object#3);
         target.push(Object Object#4);
         target.push(Object Object#5);
         target.push(Object Object#6);
      ==> ArrayIndexOutOfBoundsException#1 : EXCEPTION
showing that the Stack failed to resize itself as expected upon the sixth call to push(). The programmer may go back and fix the code so that the Stack resizes itself properly.

Validating the Test Class

Finally, you may want to construct a test oracle that learns the correct specification of the test class incrementally. In this modality, the testing tools will load an oracle data file, seeking this in the current oracle directory, which by default is the same as the test class directory (the location from which the test class was loaded). If no data file exists, a new one will be created. The only exception to this is if an oracle data file was created for the superclass of the current test class, in which case this forms the basis for the new test oracle.

When executing in the validation modality, the tools will interact with the tester to ask questions about specific test outcomes. The tester may respond with one of three signals (the input method will depend on the tool):

  • yes: to confirm the test outcome, in which case the oracle remembers this as a correct result;
  • no: to reject the test outcome, in which case the oracle remembers this as an incorrect result;
  • quit: to cancel the current test series, in which case the tools abort, saving the current results.
You will start validation at small testDepth settings, progressively increasing this as you grow confident that the results are correct. The tools are able to predict many more test results than the tester had to confirm (this is one of the main benefits of lazy systematic unit testing). Once the tool has learned a single confirmed result, it will be able to validate this, and several more outcomes, completely automatically. If the code for the test class is modified, the oracle will realise that something has changed, and will ask the tester to reconfirm just those key outcomes that appear to be different, predicting further changes where it can.

The best combination of settings to use when constructing a first test oracle is algebraic validation, since this focuses on the most important test cases describing the behaviour of the test class and presents the fewest new cases to the tester for confirmation (this may train a complete oracle, in some cases). An excerpt from the test results after training a test oracle for a Stack to a testDepth of 1 may look like this:

   Stack target = new Stack();
      ==> true : PASSED  (confirmed)
   Stack target = new Stack();
      ==> 0 : PASSED  (confirmed)
   Stack target = new Stack();
      ==> EmptyStackException#1 : PASSED  (confirmed) 
   Stack target = new Stack();
         target.push(Object Object#1);
      ==> void : PASSED
Here, the tester has confirmed some observations on an empty Stack, but the tool has automatically passed the final operation, which returned void as the tool expected. However, the tester had to confirm the exceptional result of pop(), since the tool identifies this as unusual, or possibly unexpected. If the same test series is run again, all these results will be predicted automatically.

If the tester now switches to a different setting, such as protocol validation, the same trained test oracle will be used again, and is able to validate vastly many more test cases, for example:

   Stack target = new Stack();
      ==> true : PASSED
   Stack target = new Stack();
      ==> 0 : PASSED 
   Stack target = new Stack();
      ==> EmptyStackException#1 : PASSED
   Stack target = new Stack();
         target.push(Object Object#1);
      ==> void : PASSED
The JWalk tools are able to determine that isEmpty() does not modify the concrete state of the Stack, so they may use the results that the tester confirmed earlier, and apply these to the longer sequences presented here. Note that the tester did not have to confirm any of these outcomes! However, at greater depths, the tools may eventually find new cases that could not be predicted from the existing oracle. For example, the following sequence:
   Stack.target = new Stack();
      ==> Object#2
Must be confirmed, since the tools have never seen a Stack before containing just the top element Object#2. Typically, after confirming a only few tens of outcomes by hand (which takes comparatively little time), the tools are able to validate literally thousands of test cases. See the JWalk Publications Page for information about the tools' excellent performance.

Regent Court, 211 Portobello, Sheffield S1 4DP, United Kingdom