The JWalk User Guide

The JWalk 1.1 Tool Suite comes with two finished testing tools, JWalkTester, the flagship GUI-based unit testing tool, and JWalkUtility, the classic command-line driven unit testing utility. Also, the unbundled software contains the JWalker tool kit for building custom JWalk applications. Familiarise yourself with the common JWalking concepts below, then refer to the specific guides for each tool or toolkit (see margin).

What is JWalking? Well, it is a fast and effective way to do Java unit testing. All the tools need is a compiled Java class to test, and some indication of what test strategy to follow. You may inspect the test class's interface, explore its algebraic structure or state-space, exercise sequences of methods to examine its visible behaviour, or train a test oracle to recognise correct or incorrect test outcomes. Once a test oracle has been trained, testing is fully automatic. Also, if the test class is modified and recompiled, the oracle can be re-trained in a short time. Oracles can also learn from superclass oracles, but adapt quickly to the subclass, learning about combinations of local and inherited methods. The important thing is that the tools determine the right method sequences that need to be tested, and explore the test class completely, to bounded depths.

Test Class

The test class may exist in the default package, or in a package under the current test class directory (by default, the current working directory), or in some other package known to the Java CLASSPATH. The test class must always be loaded using its package-qualified name: if you get the path wrong, the tools will report failure to find or to load the test class. The test class directory may be changed in some tools.

Test Strategy

The exploration of the test class can follow different test strategies, which range from the brute force breadth-first exploration of all method combinations, to focusing on primitive algebraic constructions that reach all low-level concrete states, to the sophisticated analysis of high-level states and transitions.

• protocol strategy: creates test sequences corresponding to every possible interleaved ordering of the test class's public constructors and methods (all method protocols);
• algebra strategy: creates test sequences corresponding to every novel construction, that is, sequences which drive the test object into different concrete states (all algebraic constructions);
• states strategy: creates test sequences corresponding to the state cover, transition-cover and n-switch transition cover, which drive the test object through its high-level state machine (all states and transitions).

Test Modality

The different test modalities control whether you wish merely to inspect or analyse the test class, or also explore its behaviour by executing different test sequences, or also validate its behaviour by training a test oracle, which is later saved in an oracle data file:

• inspect modality: performs a static analysis (and optionally a dynamic analysis) of the test class, to reveal the public API, or algebraic structure, or state space;
• explore modality: also exercises the test class by generating and executing test sequences according to the current test strategy, displaying results for the tester to view;
• validate modality: also validates the outcomes of each test sequence, comparing the actual result against the result predicted by a test oracle, which is trained interactively and saved when testing is over.

Depth Settings

The testing algorithms used by JWalk perform bounded exhaustive exploration of the test class's state space. All exploration and validation is carried out to a bounded depth. Furthermore, the initial dynamic analysis, and the comparison of object states, are also affected by custom depth settings.

• testDepth: controls the maximum length of test sequences generated during exploration (and validation) - test sequences of every length up to this limit are exercised, counting initial object construction (or a state-cover sequence) as depth zero;
• probeDepth: controls the maximum length of probing sequences generated during the initial dynamic analysis (in certain strategies) - set this higher to discover missing states and lower to avoid memory exhaustion;
• stateDepth: controls how deeply objects are examined when their states are compared for equality - set this to zero for shallow equality and higher for deeper equality (of object trees), especially if a mutable object structure appears not to change state.

Value Generators

The JWalk strategy algorithms compute what method sequences to execute. Before execution, each constructor or method must be supplied with suitable test inputs. These are supplied by objects known as Generators, which are capable of synthesising values of given Java types. Generators must synthesise values in an exactly repeatable, deterministic order (rather than randomly). This allows the JWalk tools to learn the correct test outcomes, when the same test inputs are presented in subsequent test cycles. There are two kinds of Generator, described by the Java interfaces:

• MasterGenerator: produces a standard series of quasi-unique test values (and objects) in a monotonically increasing sequence;
• CustomGenerator: produces a custom series of test values (and objects) under the explicit control of the tester.

Convention on Inheritance

The JWalk strategy algorithms exercise all public methods of the test class, including those inherited from ancestor classes, all the way up to the root Object class. However, interleaving all of Object's built-in methods is sometimes a nuisance, so these methods may be selectively excluded by convention. There are three conventional settings:

• standard - exclude all of Object's methods
• custom - include some of Object's methods
• complete - include all of Object's methods

Standard Files and Directories

The JWalk tools need to read and write certain data files during their normal operation. These files are few in number, known to the user, and exist in standard locations under the installation directory. The files that JWalk routinely reads and/or writes are the following:

• the JWalk license file, JWalkLicense.txt, which must live in the working directory from which the tools are launched;
• the oracle data file, which is created for each test class when the tools are run in validation mode, and exists either in the test class directory, or in a separate oracle directory specified by the tester;
• the redirected input file, input.txt, which is created for each test class directory, if input is redirected by uploading the custom RedirectInGenerator.

Indiscriminate File Creation

Warning: If the JWalk tools are used in an unguarded way, it is possible to excercise user-defined classes that create or destroy files and directories indiscriminately. If user-defined classes have methods that accept File or Stream parameters, then the JWalk tools will attempt to create instances of these. In principle, exercising an object that creates or deletes files and directories could cause arbitrary havoc to the filesystem, but in practice, this is mitigated in two ways:

• all created directories and files should appear below the current working directory, and have automatically generated names corresponding to the conventional strings emitted by the JWalk tools;
• the tester may take control of how and where files and directories are created, by supplying a CustomGenerator that intercepts the creation of File and Stream types.

The following describes the general approach followed when exploring and testing a prototype class using the JWalk 1.1 Tool Suite. This method applies both to the finished testing tools JWalkTester and JWalkUtility; and to any application built using the JWalker tool kit.

Inspecting the Test Class

After loading the test class, you may choose to inspect it. Depending on the test strategy chosen, you will obtain a different analysis:

• protocol inspection: performs a static analysis of the test class, showing its public constructor and method interface (if the test class is an enumerated type, its constants are displayed);
• algebra inspection: also performs a dynamic analysis of the test class's algebraic structure, classifying its operations into primitive, transformer and observer operations;
• state inspection: instead, performs a dynamic analysis of the test class's high-level state space, using the natural state predicates supplied by the class to help identify states.

To illustrate algebraic analysis, a ReservableBook class provides a number of methods, which are classified. The initial constructor and methods issue() and reserve() are classified as primitive, because they drive the book into all of its different concrete states. The methods discharge() and cancel() are classified as transformers, because they return the book to an earlier state. The methods getBorrower() and isOnLoan() are classified as observers, because they do not modify the book's state.

To illustrate high-level state analysis, if a ReservableBook class provides the two state predicate methods isOnLoan() and isReserved(), then the state detection algorithms will find four high-level states: {Default, OnLoan, Reserved, Reserved&OnLoan}, by seeking to reach concrete states in which each predicate returns true or false. The Default state is the one in which no predicate returns true. The other states are named after the predicates which return true. The shortest test sequences reaching these distinct high-level states are collected in the state cover test set, which forms the starting basis for state-based testing (see below).

Exploring the Test Class

Next, you may wish to explore the behaviour of the test class in some detail. You might do this to check that you haven't overlooked interesting combinations of methods that might yield unusual results. You will start with a low testDepth setting, and increase this as you grow confident that the shorter sequences behave correctly. The results you get depend on the test strategy chosen:

• protocol exploration: executes all constructors followed by all interleaved combinations of methods (including inherited methods), producing some unusual sequences in which the same methods are repeatedly invoked, or are invoked in unexpected orders;
• algebra exploration: executes all constructors followed by only those methods (including inherited methods) that drive the test object into new concrete states, producing sequences that grow the test object and observe its behaviour at the edges;
• state exploration: executes the state cover (see above) extended by all interleaved combinations of methods (including inherited methods), producing a set of test results for each high-level state, explored to each depth.

Algebraic exploration is perhaps the most compact strategy to use at first, when investigating all the expected behaviours of the test class. For example, the prototype ReservableBook is explored in this mode, revealing an interesting combination at testDepth 3:

   ReservableBook target = new ReservableBook();
         target.reserve(Borrower Borrower#1);
         target.issue(Borrower Borrower#2);
         target.getBorrower();
      ==> Borrower#2 : NORMAL  (unchanged)

Protocol exploration is good for identifying boundary cases like null operations and for checking that observer-methods do not unexpectedly modify the object's state. For example, when the ReservableBook is explored, this yields some interesting sequences that include:

   ReservableBook target = new ReservableBook();
         target.isOnLoan();
         target.discharge();
         target.discharge();
      ==> void : NORMAL  (unchanged)

State exploration is good for ensuring that an object behaves robustly in all of its high-level states. For example, a resizable Stack is explored, and the following unexpected exception is raised, when methods are exercised in the Stack's Full state:

   Stack target = new Stack();
         target.push(Object Object#1);
         target.push(Object Object#2);
         target.push(Object Object#3);
         target.push(Object Object#4);
         target.push(Object Object#5);
         target.push(Object Object#6);
      ==> ArrayIndexOutOfBoundsException#1 : EXCEPTION

Validating the Test Class

Finally, you may want to construct a test oracle that learns the correct specification of the test class incrementally. In this modality, the testing tools will load an oracle data file, seeking this in the current oracle directory, which by default is the same as the test class directory (the location from which the test class was loaded). If no data file exists, a new one will be created. The only exception to this is if an oracle data file was created for the superclass of the current test class, in which case this forms the basis for the new test oracle.

When executing in the validation modality, the tools will interact with the tester to ask questions about specific test outcomes. The tester may respond with one of three signals (the input method will depend on the tool):

• yes: to confirm the test outcome, in which case the oracle remembers this as a correct result;
• no: to reject the test outcome, in which case the oracle remembers this as an incorrect result;
• quit: to cancel the current test series, in which case the tools abort, saving the current results.

The best combination of settings to use when constructing a first test oracle is algebraic validation, since this focuses on the most important test cases describing the behaviour of the test class and presents the fewest new cases to the tester for confirmation (this may train a complete oracle, in some cases). An excerpt from the test results after training a test oracle for a Stack to a testDepth of 1 may look like this:

   Stack target = new Stack();
         target.isEmpty();
      ==> true : PASSED  (confirmed)
      
   Stack target = new Stack();
         target.size();
      ==> 0 : PASSED  (confirmed)
      
   Stack target = new Stack();
         target.pop();
      ==> EmptyStackException#1 : PASSED  (confirmed) 
         
   Stack target = new Stack();
         target.push(Object Object#1);
      ==> void : PASSED

If the tester now switches to a different setting, such as protocol validation, the same trained test oracle will be used again, and is able to validate vastly many more test cases, for example:

   Stack target = new Stack();
         target.isEmpty();
         target.isEmpty();
      ==> true : PASSED
      
   Stack target = new Stack();
        target.isEmpty();
        target.size();
      ==> 0 : PASSED 
      
   Stack target = new Stack();
         target.isEmpty();
         target.pop();
      ==> EmptyStackException#1 : PASSED
         
   Stack target = new Stack();
         target.isEmpty();
         target.push(Object Object#1);
      ==> void : PASSED

   Stack.target = new Stack();
         target.push(Object#1);
         target.pop();
         target.push(Object#2);
         target.top();
      ==> Object#2