The JWalk Weblog, 2006

This is the original JWalk weblog! All entries for 2006 are here. It goes back to the first public 0.3 release in May. The earlier 0.1 and 0.2 versions were not released, being experimental in nature. The original JWalk concept was an idea I had over Christmas 2005, after mulling over the reluctance of the agile community to adopt even simple state-based sketches of specifications. You can read about the early evolution of this idea, through several early releases, up to version 0.7. This year ends with a set of mixed reviews on the first JWalk journal paper, which was subsequently revised in major ways.

Monday 4 December, 2006

JWalk 0.7 Beta tentatively went live today. I need to spend more time revising the web-pages, which are not entirely in good shape for the new version, which has many structural changes since the old JWalk 0.6. For new downloaders, the tool is in a package org.jwalk (chosen for its brevity, even though we don't have the domain name yet). This means that the command-line version of the tool has to be invoked differently, with: java org.jwalk.JWalk as the start of the expression. Other changes include re-naming the different testing modes to the slightly longer, but more meaningful set: { inspect, protocol-walk, algebra-walk, state-walk, algebra-test, state-test }. This was in response to one of the referee's comments on the journal paper.

Monday 27 November, 2006

I received six referees' reports on the JWalk paper, submitted to Automated Software Engineering. The editor says: "As you will see from the reviews, the referees' opinions are rather mixed. While some recognise the potential contributions of your work, others feel quite strongly that the paper lacks clarity in terms of contextualising its contributions within related work and validating the approach. The extent of the revisions would suggest that a significantly revised version of your paper is needed before it can be re-considered for publication".

This is quite right - the paper focuses exclusively on the details of JWalk and its implementation; and should have considered a number of other similar and not-so similar tools. Unfortunately, I'm too busy teaching to address this until after the end of semester. I shall upload to the examples section of the website many more of the test-cases that show e.g. oracle acquisition, testing Java kernel classes, testing the behaviour of interleaving during inheritance and testing third-party Java software. Also, I could add some examples of mutation-testing with a fully trained oracle.

Monday 3 September, 2006

The stable release of JWalk is going to stay at version 0.6 for the time being, as I don't have time to rewrite the website for version 0.7 The new academic year is approaching fast and I need to revise two modules and prepare a third. Apparently, I'm now responsible for our Department's input to the Aerospace degree too.

Saturday 26 August, 2006

I have submitted the first full paper on JWalk to the journal Automated Software Engineering. If accepted, this could become the main reference for the JWalk tool and lazy, systematic unit testing. We shall have to wait and see...

Tuesday 18 July, 2006

Chris Thomson has successfully created an integrated version of JWalk 0.7 Alpha with the IBM open source Eclipse IDE. This is quite exciting, as it offers the propsect of using JWalk as a debugger as well as a conformance-testing tool (its main purpose). The IDE runs the editor in the main window and shows JWalk tests in the bottom window. The user can select from a tree-display in the right hand window to view past test runs, in a hierarchical fashion. If the user's tested class throws an exception, the integrated tool can track this back to the place in the source code where the exception was thrown. This can only be used for exceptions thrown by the tested class, and not for properties violated during conformance testing. You can't inject exceptions retrospectively into the code.

Friday 24 June, 2006

Finished the last re-engineering work to provide JWalk in a modular format that can be called from a third-party environment, such as the Eclipse IDE. This involved several things. Firstly, a new top-level API was created, such that third-party software could supply JWalk parameters; and these must be accessible to JWalk through this API. Secondly, rather than communicate on standard input and output, the communication model had to be upgraded to use the Java event model. Thirdly, the whole JWalk package has been encapsulated as a Java package. This version is JWalk 0.7 Alpha - not released yet.

The class hierarchy for the different TestDriver classes was finished, as described below. The new top-level API classes are called JWalker and JTalker. The former has an interface to set JWalk test-parameters, and inherits from the latter, which supports registration of event listeners. Clients using JWalker at the top-level must register classes implementing the QuestionListener and ReportListener interfaces. These have to handle QuestionEvent and ReportEvent events dispatched by the JWalk kernel to the third party environment. The command-line utility JWalk has been rebuilt to use this new mode of communication.

Tuesday 6 June, 2006

In preparation for a modularised version of JWalk that can be called through an API from a third-party environment, such as Eclipse, I have refactored all of the TestDriver classes so that each driver is responsible only for one of the five kinds of testing. The old ProtocolTestDriver has been renamed ProtocolExplorer. The old AlgebraTestDriver has been factored into AlgebraicExplorer and a subclass AlgebraicTester. Similarly, the old StateTestDriver has been factored into StateSpaceExplorer and a subclass StateSpaceTester. The algorithms for computing the next test cycle are better distributed, with more sharing of common code. Oh, and I found the fabulous animated GIF for "Jaywalking Jimbo". I dug him out of a US Agricultural Research Service article aimed at kids, called (honestly) : Attack of the Killer Fungus. Really, he deserves to get out more. I've asked the article publishers for permission to use Jimbo. I think he's been obtained from elsewhere in any case.

Monday 5 June, 2006

Released JWalk 0.6 Beta officially! Incorporated some of the user requests after JWalk 0.5 was trialled. In particular, have improved the capabilities of the Oracle to remember both confirmed and rejected test results, so that the tester need not confirm those tests that are known to have failed in a subsequent validation cycle. Obviously, if the programmer fixes the tested code so that it now returns the correct result, the tester will be presented with this for confirmation. Fixed a minor problem in the way in which array types and array values were being displayed. Formerly, this used Java's own internal short-forms, which have been expanded to more readable forms.

Friday 2 June, 2006

Nearly ready to release JWalk 0.6 Beta. The last things that needed fixing were to upgrade the examples to show the new-style reports and upgrade the Inside JWalk pages to display the javadoc for the new release, which contains a base TestDriver and three subclasses, ProtocolTestDriver, AlgebraTestDriver and StateTestDriver. The javadoc was created using package visibility, so that developers can read more about how JWalk works.

Tuesday 31 May, 2006

The new build for JWalk 0.6 beta is ready. This incorporates the upgrades described below in the last entry. Started work on upgrading the web documentation. This now includes two compiled versions for Java 1.4.2 and Java 1.5.0, to allow execution on older platforms too. The user guide was modified to describe the new way in which JWalk assumes a class has an isDefault state, which covers all other states not specified by the programmer in explicit state predicates. The Generator API was also changed, so have edited the page on customising JWalk to show how to provide custom generators. Still to do: upgrade the examples and upgrade the Inside JWalk pages, which are really for developers.

Friday 26 May, 2006

After a large refactoring, the old Tester has now been replaced by a base TestDriver and three subclasses, ProtocolTestDriver, AlgebraTestDriver and StateTestDriver. The Generator hierarchy has been modified to do without cloning and instead support a reset() method to restart each test sequence. This now does the right thing with respect to the Oracle, so that state cover prefix sequences map to the correct stored oracle values and the same oracles can be used for both algebraic and state-based tests. Had a small problem with predicting the number of pruned paths (instead of actually creating them and then pruning). Increased the generality of the state exploration algorithm.

Saturday 20 May, 2006

Started work on factoring the single Tester class into several test-drivers that perform each kind of testing. This should, in principle, make it easier to understand what is going on in the generation of test sequences, so that we can eliminate the shallow and deep copying issues. If we combine the test creation and the test execution, we can prune failed prefix paths much sooner; and we can then reset a single generator before each sequence is evaluated, which will cut down on memory usage and avoid cloning problems.

Friday 19 May, 2006

Released JWalk 0.5 Beta today. This includes another brilliant idea, to add the target object to the generator so that it is one of the first instances returned by the generator. This allows testing of the equals(Object) method. However, it also necessitated moving the point when argument values are actually synthesized, from when the test cases are first constructed, to just before they are evaluated. Otherwise, the target is never going to be the first object returned. This was going to cause lots of problems... When we also tried to short-circuit the synthesis of arguments (because many test cases are shared and have their arguments created once), we discovered a whole bunch of problems with the cloning of generators for each test sequence. This has the effect that some oracle values are being mis-mapped. This doesn't cause JWalk to fail, but requires more confirmations from the user than we would like. The JWalk website is now complete, with two new pages.

Thursday 18 May, 2006

Clearly, the new oracle encodings could also be used to print out prettier test results for the user. This required a major refactoring of the TestCase hierarchy, so that each of these classes could get a hold of the particular generator that had been used to synthesize their arguments, at the moment they were asked to print themselves. Had some problems with sharing generators that should have been cloned. Eventually, managed to print everything, including all of the char[] and byte[] arrays used in java.lang.String. We got a strange failure during the check mode test-generation, until we realised that no prefix tests were being created for constructors alone, so sequences of constructor-observer pairs were still being evaluated, in which the constructor might fail, causing the observer to be invoked on an exception. Fixed this by admitting constructor prefix tests (I know this isn't algebraic testing, but you can still observe whether the constructor returns a valid object, or an exception!)

Wednesday 17 May, 2006

Started work on trying to find a better way to map objects into oracle values. The first attempt involved the Oracle class mapping object instances to storable forms; but then the obvious realisation came that the generators should be more able to maintain a map from types to created instances. This allows a simple encoding of objects as typename plus index number, representing the rank order in which that object was created on a particular test cycle. Although not infallible, it was a fairly good approximation of pseudo-identity across different test runs. Added the toString(Object) method to the Generator interface.

Tuesday 16 May, 2006

Originally, ObjectGenerator was going to produce just a few kinds of dummy object, compatible with their designated type, but declared serializable and with a simple count field used for equality testing. To synthesize anything else, the user would have to customise JWalk using a custom generator for the new argument type. But then we changed our mind, and decided to synthesize arbitrary object types, using the Class.newInstance() API. This means that object types must provide a default constructor. But, Oh! the pain! Of course, not every kind of object is serializable and, in any case, how do we judge whether two separate non-identical instances, created on separate test-runs, are meant to be treated as equivalent?

Monday 15 May, 2006

Started working on refactoring the generators, so that we could have a hierarchy of them. The catch-all ValueGenerator was divided into ValueGenerator, ArrayGenerator and ObjectGenerator, which each handled the synthesis of progressively more complicated argument types. Each generator now delegates to a single method to generate values of simple, array or object types. This allowed ArrayGenerator to construct arrays of object-types, since element-generation is a template method pattern, intercepted by the subclass. See the online documentation   for how the generators now work.

Friday 12 May, 2006

JWalk 0.3 Beta went live for the first time. This website was partially constructed, so that folks could start JWalking... The 0.3 Beta version could only synthesize values of the simple types, Strings and a single DummyObject type. The various examples on the JWalk Examples page were produced by this version. We then tried it out on java.lang.Object and got a realistic set of results. We tried it on java.lang.String and the generator reported that it couldn't produce char[] and byte[] arrays. Also, the saved oracle files were different for the state and algebra testing modes - why not combine these, so that fewer oracles need to be confirmed by the user?