JWalk software testing tool suite

Lazy systematic unit testing for agile methods

You are here: JWalk Home / User Guide / JWalker Toolkit /
Department of Computer Science

The JWalker Toolkit

JWalkEditor creating and testing a LibraryBook

The JWalk 1.1 Tool Suite provides a kit for building your own testing applications. You build around the JWalker core test engine. All the other featured tools were built from this kit, including this early beta release of the JWalkEditor, a combined Java editor, compiler and testing tool.

Please note that you must first have a valid license to run the test engine, which is obtainable free for most academic and industrial evaluation purposes. For this and the software bundle, please follow the clear instructions given at the JWalk Download Centre.

Unpacking the Toolkit

The following assumes that you have chosen your JWalk working directory, and have installed the software bundle: JWalk11.jar and an accompanying license: JWalkLicense.txt in this directory.

Unpacking the Download Bundle

To use the toolkit, you must first decompress the JAR-file download bundle. Open a command console window and, in this console, change to your chosen JWalk working directory, in which the software bundle was placed. Unpack the archive using the Java console command:

   jar -xf JWalk11.jar
This will create a Java package directory structure under your working directory, containing compiled Java class-files and a number of sub-packages. The packages you see under the newly created subdirectory org should include:
  • org.jwalk - the interface to the toolkit;
  • org.jwalk.core - the core test engine;
  • org.jwalk.gen - a custom generator kit;
  • org.jwalk.out - a test result reporting kit;
  • org.jwalk.test - some example test classes;
  • org.jwalk.tool - JWalkTester and JWalkUtility.
The directory META-INF in your working directory may be safely ignored or deleted (it contains a Manifest file, which tells Java how to launch the compressed archive).

Unpacking the Documentation

Detailed documentation has been created for each of these packages, supplied as a separate download bundle. This is identical to the documentation found under the Look Inside section of this website. However, if you wish to have your own local copy of this, you should download the documentation bundle, a JAR-file archive JWalk11doc.jar, placing this wherever you wish to browse hypertext files. Unpack the archive using the Java console command:

	jar -xf JWalk11doc.jar
This will install a directory structure rooted in the subdirectory html under the current working directory. Point your web browser to the html/ root directory to load the index file.

Key Toolkit Components

The following assumes that you wish to build an application, which incorporates the core JWalker test engine. Your application is expected to provide its own user interface and display facilities. Your testing application will want to load a test class, set various test parameters and specify how to receive call-back communications dispatched by a JWalker, which will cause your application to interact with the tester, or display test results. To do this, your application will create a single instance of the JWalker test engine; and then access its Channels API to set up the communication channels; and access its Settings API to supply the desired test parameters and upload the test class.

The JWalker's Channels API observes Java's event-based model for communication. This allows the core test engine to be decoupled from different kinds of front-end for handling input and output. From version 1.1, the core test engine may also be run in a separate worker thread, in the background. Your application must register suitable event listeners, which are capable of handling the events dispatched by the engine. The two kinds of event are QuestionEvent, encapsulating an interactive query or notification, and ReportEvent, encapsulating a test report. Your application will register objects that satisfy the QuestionListener and ReportListener interfaces with the Channels object, in order to process these events.

The JWalker's Settings API offers fine-grained control over all test settings, ranging from the directories in which files are found and placed, a flexible facility for loading (and re-loading) the test class and custom generators, to complete control over the strategy, modality, convention, test depth, probe depth and state depth parameters. In particular the test class may be re-loaded multiple times into the same Java runtime (after being modified) and will execute in its modified form. The tester may supply one or more generators satisfying the CustomGenerator interface, to take control over how test input values are synthesised.

Examples of Usage

The following examples indicate how your own testing application might interface with the components in the JWalk 1.1 Tool Suite. The application will eventually create a single instance of the JWalker core test engine, set up callbacks, upload test settings and invoke the JWalker's main execute() method. Warning: if you do not have a valid license, the application will terminate. If it was launched from a console, a license violation message will be printed on the console; otherwise it will terminate silently!

Example 1: Protocol Exploration

The following fragment illustrates the style of code necessary to set up the test class, construct a JWalker, register suitable listeners, set test parameters and execute a test series. This example will explore all method protocols of a Stack class to a depth of 4, and will execute as a single-threaded application.

   Class<?> testClass = Stack.class; 

   JWalker walker = new JWalker();
   Channels channels = walker.getChannels();
   Settings settings = walker.getSettings();
   channels.addQuestionListener(new MyQuestionListener());
   channels.addReportListener(new MyReportListener());

The classes MyQuestionListener and MyReportListener are supplied by your application, according to the interface specifications in QuestionListener and ReportListener. These classes may interact with the tester through dialogs, and display the contents of reports in any desired manner. See the documentation for the package org.jwalk.out for further details of kinds of reports emitted by the JWalk 1.1 Tool Suite. Protocol testing to any great depth will soon lead to OutOfMemoryError exceptions, but in single-threaded execution, these are handled safely by the engine.

This first example assumes that your application has supplied a reference to the test class directly - in other words, it was loaded into memory by your application. Note that Java may cache a loaded class and this may prevent you from reloading it, unless you provide a customised ClassLoader, or allow the Settings API to reload the class by name (see below), using its own ClassLoader.

Example 2: Algebraic Exploration

The following fragment illustrates the style of code necessary to initiate the algebraic exploration of a Stack class to a depth of 3, using the JWalker API as before, but this time supplying slightly different test settings, and running the test engine in a separate asynchronous thread. This can be useful if the GUI launching the engine should remain active, while the test engine runs in the background. However, third-party GUI designers must then also ensure that all GUI threads can recover from OutOfMemoryError exceptions, since you cannot then predict in which thread this condition will arise.

	String className = "Stack";   // or by other means
	JWalker walker = new JWalker();
	Channels channels = walker.getChannels();
	Settings settings = walker.getSettings();
	channels.addQuestionListener(new MyQuestionListener());
	channels.addReportListener(new MyReportListener());
	Thread thread = new Thread(walker);
The test engine will run as a parallel worker thread with the same priority as the code that launched the thread. Reducing the priority by one or two will make the worker thread run in the background. All exceptions thrown by the worker thread must be handled within the thread, but will notify the third-party supplied QuestionListener about any exceptions, just like any other notification, such that the third-party system may choose to report the fault. The worker thread eventually sends a silent Notification to signal when it has finished testing. See below, and also the package documentation for org.jwalk.out, for further details of how to process notifications.

This example also illustrates how to set the test class by name, rather than by supplying the class object directly. This allows the test engine to load and reload the test class from a compiled file, even if it has been modified since. The test class will be loaded in the manner specified in setTestClass(String). Java will seek to load the class from the directory specified by getTestClassDirectory(). If the test class is packaged, you should load it using its full package-qualified name from the working directory above the package. See the Settings API for details of how to change the working directory.

Example 3: State-Space Validation

The following fragment illustrates how a custom generator may be supplied via the Settings API; and how the directory in which oracles are saved may be set explicitly (the default is the test class directory). This fragment will perform state-validation on a Vector class to a transition path depth of 2, using a user-supplied custom IndexGenerator as the generator. This example runs in a single thread, and interacts with the human tester on standard input and output.

	Class<?> testClass = Vector.class;       // or by other means
	Class<?> custom = IndexGenerator.class;  // or by other means
	JWalker walker = new JWalker();
	Channels channels = walker.getChannels();
	Settings settings = walker.getSettings();

	Console console = new Console();
	settings.setOracleDirectory(new File("test/oracles/"));
For specific advice on when you should supply your own CustomGenerators, please refer to the section on how to create Custom Generators for JWalk. Custom generator classes may also be supplied by name, in a similar manner to the test class. More than one custom generator may be uploaded, if so desired, but generators may only be loaded once by your testing application during the same Java runtime (this has to do with Java recognising the version of the generator). If you modify the generator, you will have to restart the application.

In this example, the oracle data file will be saved in the file: "test/oracles/Vector.jwk". The oracle data file is valid for retesting using the same generators, but not necessarily valid for other generators, which may synthesise different test inputs in different orders.

This example uses a default Console object to fulfil both the roles of the QuestionListener and the ReportListener interfaces. Console is supplied with the toolkit and it unpacks the various communicated events and pipes them to standard output, expecting responses from the tester on standard input. If the third-party program fails to set any listeners, a Console is then used by default.

Handling JWalk Exceptions

While the JWalk 1.1 Tool Suite recovers automatically from a number of anticipated exceptional situations, such as when methods of the test class fail, or memory is exhausted during testing, or when the tool fails to detect all the expected high-level states, your application will need to recognise and possibly handle the following five kinds of exception. The first two are raised while setting test parameters, while the last three may be raised during test execution.

  • LoaderException - raised if the test class, or a custom generator could not be loaded (wrong pathname given)
  • SettingsException - raised if the third-party software supplies invalid test settings (bad constants, out of range values)
  • PermissionException - raised if the test class refuses permission to be executed (wrong visibility, security permissions)
  • GeneratorException - raised if test inputs could not be synthesised, or a generator fails (unknown types, bad user-supplied generator)
  • ExecutionException - raised if any constructor or method could not be invoked, or behaved randomly (security, nondeterminism)
These are the only conditions which actually prevent a JWalker from executing. In single-threaded execution, these may be caught by the third-party program, which may notify the user. When executing the test engine in a separate worker thread, the latter three exceptions cannot be thrown directly; instead the third-party MyQuestionListener is notified, and may respond appropriately. All of the above exceptions are subclasses of JWalkException, so third-party software may choose to trap just the one exception, or handle them more selectively. Each kind of exception provides access methods to inquire about the kind of exception, and the underlying cause of the exception, which may be used to tailor error messages.

More Detailed Usage

The following gives more details about how to construct listeners that satisfy the QuestionListener and ReportListener interfaces. Further information is also given about the Report and Question objects returned inside events. For full details, please refer to the package documentation for org.jwalk.out.

The Contract for ReportListener

A ReportListener is designed to publish test reports emitted by the JWalker test engine. It must provide the single method void publish(ReportEvent event) which accepts a ReportEvent and publishes the contents of the report that this contains. That is, processing a ReportEvent should cause your application to display the contents of a test report in the desired manner. A simple skeleton implementation might be:

   void publish(ReportEvent event) {
      Report report = event.getReport();
which simply prints the content of the report on standard output. A smarter way of processing reports might interrogate the kind of Report first, and dispatch to specific handlers for each type:
   void publish(ReportEvent event) {
      Report report = event.getReport();
      switch(report.getEdition()) {
         case Edition.PROTOCOL_REPORT :
            handle((ProtocolReport) report); break;
         case Edition.ALGEBRA_REPORT :
            handle((AlgebraReport) report); break;
This uses the enumerated type Edition, which is provided as a convenience so that applications may quickly recover the type of the Report. Each kind of Report has its own API for accessing the contents of the report, which allows your application to display this in a custom way, for example, using icons, or colour to indicate a test outcome in a particular way.

The Contract for QuestionListener

A QuestionListener is designed to interact with the human tester in order to elicit a response to a question; so it must be able to handle both input and output. It must provide the single method: Answer respond(QuestionEvent event) which accepts a QuestionEvent and returns an enumerated value from the Answer type, indicating the user's answer to the question. That is, processing a QuestionEvent should cause your application to ask the tester a question, and accept a reply from the tester, which can be converted into an Answer. A simple skeleton implementation might be:

   Answer respond(QuestionEvent event) {
      Question question = event.getQuestion();
      do {
         /* Get a response from the user */
      while( /* Response not acceptable */ );
      /* Convert response to an Answer */
      return answer;
which prints out the information-content of the question, then repeatedly prompts the user with a query for input, until a valid response is received. The content-part and question-part of a question object may be presented separately, as illustrated here. A smarter way of processing questions might interrogate the kind of Question first, and dispatch to specific handlers, which pop up suitable dialogs for each kind of question:
   Answer respond(QuestionEvent event) {
      Question question = event.getQuestion();
      switch(question.getEdition()) {
         case Edition.CONFIRM_DIALOG :
            return handle((Confirmation) question);
         case Edition.NOTIFY_DIALOG :
            return handle((Notification) question);
This uses the enumerated type Edition, which is provided as a convenience so that applications may quickly recover the type of the Question, and allows a Confirmation to be processed by one kind of dialog, and a Notification to be processed by another. Whereas a Confirmation expects a yes/no/quit answer to a given question, a Notification merely requires an ok response from the tester (a notification may be silent - see below). The dialogs will display the content-part of the question or notification, but offer buttons to accept the tester's response and convert this to an Answer.

The Report and Question Components

All information emitted by the JWalker test engine is some kind of Report. A Question is merely a kind of report that solicits a response from the tester. Both of these produce formatted output strings using getContent(); and questions also produce a formatted query prompt strings using getQuestion(). The various kinds of Report and Question emitted by the tool suite may be classified in a hierarchy:

  • Report - the abstract ancestor of all reports
    • ProtocolReport - reports a class's method protocols
    • AlgebraReport - reports a class's algebraic structure
    • StateReport - reports a class's high-level states
    • StatisticalReport - abstract ancestor of test results
      • CycleReport - gives the results of a single test cycle
      • SummaryReport - summarises a complete test series
    • Question - the abstract ancestor of all questions
      • Confirmation - a request to confirm or reject a test outcome
      • Notification - a notification requiring an acknowledgement
Each of these reporting components also provides specific access methods to access different parts of the reported results; and a number of formatting methods to convert reported elements (objects) into conventional string representations, ready for display. This allows the third-party application to take control, at different levels of granularity, of how the reports or questions are to be displayed, in combination with other graphics or colour. For further details, please refer to the package documentation for org.jwalk.out.

Raising Different kinds of Dialog

The JWalker test engine raises five kinds of exception, and deals with other conditions by dispatching Notification objects to warn the tester. From version 1.1, when the test engine is launched in a separate thread, a SILENT notification is dispatched to signal the end of the thread's execution. This allows a third-party GUI to synchronise itself with the test engine. Also, exceptions raised by the test engine in the spawned thread can only be communicated back to the main thread as Notification objects. Third-party code should distinguish different kinds of Notification and pop up a suitable dialog:

        void handle(Notification notice) {
            switch(notice.getUrgency()) {
                case Urgency.SILENT :
                    // A silent notification, no response needed
                case Urgency.NOTICE :
                    // A notice, trigger an information dialog
                case Urgency.WARNING :
                    // A warning, trigger a warning dialog
                case Urgency.ERROR :
                    // An exception, trigger an error dialog
This uses the enumerated type Urgency, which is provided as a convenience so that applications may detect the severity of the Notification, and pop up a suitable kind of dialog. See the API of Notification for further details.

Regent Court, 211 Portobello, Sheffield S1 4DP, United Kingdom