The JWalk Weblog, 2009

This is the JWalk weblog for 2009! It describes how several experimental projects to build a Java Swing-based GUI for driving the JWalk test engine were evaluated; and this eventually led to the development and release of the JWalk 1.0 toolkit, including the flagship JWalkTester tool (featured here). Research conducted into how to improve the core test engine's path-pruning and test result prediction was finally reported properly at the TAIC PART 2009 conference in Windsor, where JWalk was demonstrated alongside other tools during an afternoon devoted mostly to industrial partners.

Industry and academia partners were invited to sign up as Beta-users for the JWalk toolset and submit any feedback they have to me. I am expecting to get all sorts of comments on things that either work well for them, or don't work the way they would like, and will respond to requests for further enhancements. Over the year, about twenty-five respondents for different organisations have signed up, which is gratifying. We also took JWalk on the road and gave a seminar at Leicester University. Finally, I have agreed to be General Chair for the next software testing conference TAIC PART 2010.

Monday, 21 December 2009

Continuing the saga of the forthcoming TAIC PART 2010 software testing conference, I am pleased to announce that Dr Wolfgang Grieskamp has agreed to be a keynote speaker at the conference. This is great news, and completes a slate of three fantastic guest speakers.

Well, the University term seems to have gone on for far too long this year; and I need to finish off my Christmas shopping. This will be my last blog entry for 2009! Merry Christmas and a Happy New Year to all our readers.

Friday, 18 December 2009

For those following the fortunes of our preparation for TAIC PART 2010, I am pleased to announce that Professor Bertrand Meyer has agreed to be a keynote speaker at the conference. This is good news indeed!

Wednesday, 16 December 2009

I have had a request for a commercial evaluation JWalk license from a company Web Bearings, Inc., based in California. Thanks for signing up! And by all means contact me again with feedback. Please watch this blog for up-to-date information.

It's also the day of the Departmental Christmas Party, so I will be logging off for today at lunchtime and heading down to the Soyo bar for drinks and a buffet with all my colleagues here. Merry Christmas to everyone!

Famous last words - I came back to my desk after the party to find out that the IEEE were being obdurate about using their name in a high-quality software testing conference that has been published by them for the last four years or so. Apparently, they want to enforce a distinction between conferences that are sponsored by them and conferences that are published through their conference publishing arm, Conference Publishing Services (CPS). They are saying that we may only use the name "CPS" in our advertising and web-presence for TAIC PART 2010, and not "IEEE" as before. Since their online submission system is so difficult to use, this may well persuade the steering committee to go with a different publisher.

Tuesday, 15 December 2009

Today Gordon Fraser mailed me to say that we are fortunate to have Professor Sir Tony Hoare as one of our invited speakers for next year's TAIC PART 2010 conference. This is wonderful news - makes a General Chairman very happy!

Monday, 7 December 2009

Today is the deadline for my second-year students to have delivered their project software and final reports. Some teams have produced a nice downloadable applet that demonstrates JWalk in its exploration-mode only (since applets cannot write to the host user's hard drive). Others have produced a web-based system allowing potential Beta-users to sign up for trying out JWalk. Unfortunately, most of these are significantly more complicated than I wanted, with extra burdens on potential users to provide security information (not in my requirements!) and all but two of these systems missed my core requirement to be able to extract the agreed publicity information from the system. It just goes to show how important it is to listen to your customer's requirements, rather than get dazzled by the technical possibilities...

Monday, 30 November 2009

I have had a request for a commercial evaluation JWalk license from a Leicester-based company, ITP Engines UK Ltd. By a strange co-incidence, this company operates on the same site as the former GEC company (chaired by Sir Arnold Weinstock), where Sir Frank Whittle first developed his gas turbine engine. And furthermore, when I was an undergraduate some 30 years ago, I once took a vacation job at GEC on the same site (it was the division called the Mechanical Engineering Laboratory) where we experimented with automating the operation of a crash-gearbox using a microprocessor, as an alternative to the fluid-clutch based automatic gearboxes.

Friday, 20 November 2009

I have had a request for a JWalk license from an interested open source developer, from Delft, Netherlands. Thanks for signing up! And by all means contact me again with feedback.

Friday 13 November 2009

Today I gave a seminar at Leicester University, to Reiko Heckel's group. Here's the abstract:

Lazy Systematic Unit Testing for Java

Testing from state-based specifications is a proven technique for exercising software components, to check for all expected and unwanted behaviours. However, in the Agile Methods community, there is a resistance even to simple state-based specifications and an over-reliance on manually created regression tests, which fail to find all faults. The "lazy systematic unit testing" method was created in response. It is based on the two notions of "lazy specification", the ability to defer finalising the software's apparent specification until a late stage in development; and "systematic testing", the ability to explore the state-space of a software component exhaustively, to bounded depth.

The approach is exemplified in a Java unit testing tool, JWalk. This infers the apparent specification of compiled Java classes, by a combination of static and dynamic analysis, and limited interaction with the programmer. The tool is able to excercise all method protocols, all algebraic constructions and all high-level state transitions, to bounded depth. Once the oracle for the specification has been acquired, testing is fully automatic. The test sets adapt gracefully when changes are made to the code; and re-generated tests perform significantly better than regression tests. In a testing challenge, a JWalk tester was able to test two orders of magnitude more paths than an expert tester writing manual tests for JUnit. The talk will include a demonstration of the JWalk tool.

Reiko and his group took me out afterwards to a fine meal in an Indian restaurant on Granby Street. Thanks for the hospitality!

Thursday, 12 November 2009

The list of known JWalk installations around the world now exceeds twenty! I have added two more known JWalkers to the list of adopters on the JWalk website, including a commercial evaluator from the Test Competency Group, L&T Infotech, Bangalore; and an interested student from the Technische Universiteit Eindhoven, Netherlands. Thanks for signing up! And by all means contact me again with feedback.

Monday, 9 November 2009

I have had another license request for JWalk from an interested student from the Technische Universiteit Eindhoven, Netherlands. Thanks for signing up! And by all means contact me again with feedback.

Wednesday, 4 November 2009

I have had another request for a commercial JWalk license from the Test Competency Group, L&T Infotech, Bangalore. Thanks for signing up! And by all means contact me again with feedback.

Monday, 2 November 2009

Today is the deadline for my second-year student's initial analysis for the JWalk-related projects I set them. Some have gone for the trial version applet, others have gone for the Beta-user sign-up system. As usual, some students have got the idea right, situating the systems in their proper context; whereas others have wasted their analysis diagrams on telling me how the JWalk test engine works, which I knew already.

Friday, 23 October 2009

I decided to re-organise the structure of the JWalk web-site slightly. The stub Examples page was never eventually fleshed out, since the User Guide page already has plenty of examples of usage. I have unhooked this from the menus, and instead added a more prominent Adopters page to show who has installed JWalk around the world (or, at least, who is known to have done so - earlier versions of JWalk could have been downloaded without my being able to track this). If you look at all the countries represented, we only need some from Australia, Africa and South America to have all continents represented!

Friday, 16 October 2009

Had another license request from an Indian company, Alliance Global Services. This will be the third Indian license I have given out. I really think that India is growing as a power in international software development, particularly IT centres such as Bangalore and Chennai. My colleague Guy Brown went there this summer and said the experience was fascinating, particularly the driving-habits (viz. going the wrong way down a stretch of dual carriageway to avoid a traffic jam on the correct side of the road!)

Today I finally agreed (in response to repeated requests from Phil McMinn!) to become General Chair for the next software testing conference TAIC PART 2010. I will be asking Len Botacci and Gordon Fraser to serve as the programme chairs, and Gillian Callaghan to be our local organiser. This is a new venture for me!

Wednesday, 14 October 2009

Today I gave a challenge to my second-year software engineering students. They are working in teams to deliver a software system for which I am the client. They can either choose to develop a free trial version of JWalk that runs as a Java applet (so it cannot write to files on the host user's machine), or they can develop a web-based system for potential Beta-users to sign up and get single-user evaluation license for JWalk. Let the interviews commence!

Thursday, 1 October 2009

It's been a busy month, what with all the master's students finishing up their dissertation projects. In the meantime, I've issued a few more JWalk licenses to France and Germany. No come-back from China yet.

Monday, 7 September 2009

I've had a request from Zhang Xiao-Dan (from Ericsson, China Research and Development) for copies of my slides describing JWalk, after the conclusion of TAIC PART yesterday. If they ask for a license, this will be our first Chinese installation.

Saturday, 5 September 2009

We had an afternoon of tool demonstrations at TAIC PART today. The two industrial sponsors were invited to demonstrate, but one chose to gave a presentation instead. I got the opportunity to show off JWalk's capabilities. We managed to test a lending library system, and discovered some faults that were revealed in five out of 10,000 test cases that we managed to create and run in about three-quarters of an hour. It was quite exhilarating, not knowing what the tool might throw up. The fault was deliberately seeded in the tested code - it didn't show up until you were testing method sequences of length 5. The faults had to do with lending a book, which was subsequently reserved, then returned by the original borrower, then lent to the person who reserved the book. The fault was that issuing a new loan in this context failed simultaneously to cancel the earlier reservation. Nice one!

Friday, 4 September 2009

Today is the start of the premier UK and world software testing conference: Testing in Academia and Industry Conference: Practice and Research Techniques (TAIC PART). This takes place down in Windsor Great Park (on the family estate of HRH Queen Elizabeth) and in a lovely building called Cumberland Lodge, which used to belong to the Queen Mother. It was subsequently given over to an educational charity, with a remit to bring together all the brightest young people in the world to collaborate together to foster world peace and understanding. Well, we hope that building safe software helps towards this goal!

I gave our JWalk paper, Dynamic analysis of algebraic structure to optimize test generation and test case selection, reporting on the latest improvements since we added full algebraic analysis techniques to the JWalk package. The final version of the paper can be accessed through the JWalk Publications Page.

Tuesday, 7 July 2009

Thomas Knauf, from GK Sofware AG, Schöneck, Germany, got in touch with me today to ask for a JWalk license. Thanks to Swen for passing on the news! I hope that they enjoy using the tool, and get the hang of how to customise the Generator framework for their own use.

Friday, 3 July 2009

The oddest things happen. I've been out in Thessaloniki this week, visiting my PhD students at the South-East European Research Centre (SEERC) and on my return flight today, I happened to sit next to a German passenger by the name of Swen Trillitzsch, who works for a point-of-sale equipment company, GK Sofware AG, based in Schöneck. They have a reasonably large Java software development group. We got talking, and it seemed that he would be interested in getting JWalk for their company, for use in their in-house testing. Anyway, I exchanged details with him; we can hope that they get back in touch!

Thursday, 18 June 2009

I sent out the first Polish license for JWalk today! Marek Kuras asked for a copy of the software to try out for personal projects. It's nice to know that the world user-base is growing. I hope they call back later to tell me how they found the tools.

Saturday, 6 June 2009

I heard from Sigrid Eldh today, who showed interest in JWalk at last year's TAIC PART conference. Chris Thomson put us in touch. Sigrid hopes to send a research team to visit me this Autumn. Sigrid works with Ericsson, but is also rather busy trying to complete a PhD at the moment...

Tuesday, 26 May 2009

I've just started working with Rafet Ongun, a capable master's student, who aims to extend the JWalker framework to create a new kind of JWalker that is able to infer the complete state-transition graph of an object. This is a good topic for a dissertation! Rafet has just mailed to say that he installed his user license successfully.

Thursday, 21 May 2009

I've sent out another three or four licenses for JWalk recently. We now have two IT consultancy companies in India, who have asked to trial the software and compare it with their existing unit testing strategy. That means there are known JWalkers from Canada and the USA, across northern and southern Europe and down to the Indian subcontinent! At the moment, you can see brief details about registered users, and their reasons for wanting to use JWalk, on the JWalk Publications page. If this gets any bigger, I shall have to dedicate a special page to the world-wide usage of JWalk, maybe with a map!

Tuesday, 5 May 2009

Our latest JWalk paper has been accepted for the forthcoming Taic Part Conference. The title of the paper is: "Dynamic analysis of algebraic structure to optimize test generation and test case selection", by myself and WenWen Zhao, who worked on an early prototype of the technique to detect revisited states. Thanks for all the help, Wenwen. You can see a draft version of the paper here. This will be revised in accordance with the reviewer's comments before final publication.

Thursday, 23 April 2009

I've sent out another license for JWalk today. The licensing system seems to be working well, in that we are better able to track who is using JWalk and for what reasons. I hope to keep in contact with all users; and am always open to suggestions for improvements. I'll publish any experience reports I receive here on the weblog.

Thursday, 9 April 2009

I've sent out three licenses for JWalk today! People from as far away as Belgium and Sweden are interested in JWalking. This could be the start of something interesting... There are still some other folk who I'm waiting to get back to me with the official registration information I request. I'm off for a week's Easter vacation now, so I'll deal with those requests when I return.

Tuesday, 31 March 2009

The new-look website for the JWalk 1.0 tool suite goes live today! We have been very excited about the latest version of the tool, which is easier to use than ever before. All the details appear at the top of this blog. Here, you see featured the JWalkTester running under Solaris (Sun's implementation of Unix).

Friday, 27 March 2009

Finally, the licensing system works as intended. Once you install a version of JWalk with your license, you cannot transfer this to another machine on the same license. If you try to unpack the toolkit and build your own entry-point into the software, you still can't bypass the license-check; and the software terminates, without a valid license. So I can now track the usage of JWalk a bit better. The earlier "gentleman's agreement" wasn't working that well - only a few folk bothered to let me know they had downloaded the tool. Mind you, those that did were decent folk, from IBM, Accenture and suchlike. Thanks for being well-mannered, guys!

Monday, 23 March 2009

Before I release the JWalk 1.0 tool suite to the curious world, I'm thinking that we should start to protect our IP in this product (which has, after all, taken three years to develop). So, I'm now going to require licenses to be issued with the JWalk tools. I'll not say how I encrypt these, but the plan is to have single-user, single-installation licenses for trial usage, and multi-user site licenses for commercial usage. This will all eventually be described on the new version of the website, which is under development right now, but won't go public until we have a consistent set of web-pages to go with the new version of the tools.

Friday, 13 March 2009

What a relief! I managed to submit the latest JWalk research paper to the Taic Part Conference on time. The title of the paper is: "Dynamic analysis of algebraic structure to optimize test generation and test case selection". I have included Wenwen Zhao as my co-author, because she worked over last summer with me on the first attempt to extend the algebraic analysis technique to determine when tested objects revisited earlier states. She's happy about this! Thanks for all the help, Wenwen. You can see a draft version of the paper here.

Not only that, but I've proof-read another joint paper on empirical software engineering with Chris Thomson, who presented JWalk for me last year, and will ask me to present this, if it gets accepted. Chris is about to leave us; he was the guy who helped me do all of the comparative JWalk versus JUnit testing, as he's an expert JUnit tester. Good luck, Chris! Oh, and did I say that I have also submitted to the European Semantic Web Conference, along with my Greek collaborators Dimitris, Ervin and Dimitrios? Three papers in one week - that's unbelievable. The latter paper has already been accepted; and it's about testing web-services. We may be able to do this with JWalk - watch this space!

Thursday, 12 March 2009

I really want to get out of the office today, to finish off the paper! Chris Dabnor turned up early for supervision today (thanks, Chris!) and so I was able to get back to the version cooking on my laptop at home. The pruning experiments have given great results, as expected. But the test prediction results seem mixed. I'll have to investigate this further, but there isn't much time to alter things for the paper. Finishing at 2 am tomorrow, again. Yawn!

Wendesday, 11 March 2009

Got as far as inserting the screen-shot images into the paper today. I so hate Microsoft Word's inability to frame images properly and anchor the frames to the page that I want. You seem to be able to anchor the images, and choose one of several text flows. I don't want the text to flow round the image, I want a frame around the image, below which the text starts again. You know, a frame, with a certain amount of space for a caption, and then a bit of padding before the 2-column text starts? Microsoft Word keeps on moving the anchor about on the page, whenever I add the caption in a text box (I'm doing this, because the main flow of the paper is in 2-column format; but the image and caption have to go across both columns). I've tried inserting a picture directly into the text (which interrupts the two-column format, sometimes), or inserting a textbox and then adding the image to this. The latter seems to work the best, although sometimes the image is resized when it is captured, such that the DPI goes to hell. I seem to remember that NeXTStep's FrameMaker could do all this kind of thing at a moment's notice.

My father turns 87 today. Happy Birthday, Dad! We all went over to Birmingham last Saturday, with presents and party treats. My sister will be with him this evening.

Tuesday, 10 March 2009

Spent most of the day at home today, working on the Taic Part article. Managed to conduct several series of tests into the pruning-behaviour of the JWalk 1.0 version compared to the old JWalk 0.8 version. Also, am hoping to conduct experiments into the test prediction capabilities of the two tools.

Finishing up after 1.00 am tomorrow. I'm practically falling asleep over the laptop, by the time I realise it's time to stop. At 10.00 am this morning, the cleaning lady came in to whizz round the house with the vacuum cleaner and tackle the stack of clothing in the ironing basket. Had to shut out the noise...

Monday, 9 March 2009

Today I gave a seminar on the JWalkTester flagship tool to our Verifcation and Testing Research Group. Everything went according to plan! Even when folks asked me to test different classes, the results came quickly. The new algebraic analysis facility is really working well. This is the basis for a research paper I'm desperately trying to finish in time for the IEEE Computer Society sponsored Taic Part Testing in Academia and Industry Conference.

It's after midnight, and I'm still working on the paper. I've got the introduction and explanation of JWalking done; I need to do the experiments to show the benefit of the new tools versus the old versions. To do this, I have to keep flicking switches on the new software, to emulate the strategy of the old version.

Friday, 27 February 2009

Mike Stannett and I have submitted a tutorial proposal to the ETAPS conference in York. The idea is that Mike will give the tutorial about our X-Machine based approach to complete state-based testing, aided by Chris Thomson with his eXtreme X-Machine tool, which allows developers to design quickly the X-Machine specification for the interface of an agile software development project. Then, I will come in and teach the attendees how to test in practice using JWalk. We put in the outline before Christmas, but today I finalised the slides, which teach how to use the current stable form of the JWalkTester tool.

Wednesday, 25 February 2009

Having to fit JWalking around various exam boards today! I guess work can be inconvenient, when you're following an idea to conclusion. I'm in the middle of another large scale refactoring, converting the TestCase class into a small hierarchy, with EnumTestCase for enumerated values, ParamTestCase for anything with arguments, CreateTestCase for constructors and InvokeTestCase for methods, as the children of TestCase, which is now an abstract class. This way, I aim to handle the different execution processes for enumerated types, constructors and methods.

Monday, 23 February 2009

Eventually I decided that JWalk must be able to handle any kind of test class, whether this is a regular class, or an enumerated type. Also, JWalk must be able to detect if it has been given something illegal, such as an interface, or an abstract class, which it clearly cannot test directly (because you cannot create instances of anything abstract). I've modified ProtocolWalker to apply a number of static tests to determine whether the test class is abstract. The tools now correctly halt after the inspection phase, and pop up warning dialogs if the tester tries to exercise anything abstract.

Friday, 20 February 2009

The GUI for the JWalk testing tool is now pretty stable and looks good both under Windows and Unix. This week, I have added a new kind of custom generator, called EnumGenerator, that is capable of synthesising every enumerated value from an enumerated type in Java. Initially, this was motivated by the need to synthesise enumerated values for method arguments. It works beautifully, following the CustomGenerator paradigm for handling such weird cases. I'll now have to think about whether the test class could also be an enumerated type. If so, then the basic TestCase will need to be adapted to allow this to contain an initial enumerated constant, instead of just a constructor or method for execution.

Tuesday, 17 February 2009

Finished a large refactoring of the GUI components today. I've made an abstract panel superclass, and an abstract dialog superclass, which take away some of the shared elements out of the different panels and dialogs. The other thing that changed this week was the ability for JWalk to emit a new kind of Question in its event model. Previously we had Confirmation as the only concrete kind of question that requires user interaction. I have now added Notification as another kind of interactive component. The reason for this was to drive the notify, warning and error dialogs that the GUI tool needs to pop up. Having these generated within the core JWalker test engine was much more satisfying than setting flags and having the tool detect these conditions to raise dialogs.

This has paid off in the design of ReportListener and QuestionListener objects, which can now dispatch on the kind of Report or Question unpacked from the events that fly about between the core JWalker and the GUI tool. I've used the names that Mihai-Gabriel's team gave to these concrete listeners in their implementation of the GUI-based tool: ReportPublisher and QuestionMaster. But I've completely re-written the code internally, so that these now dispatch on the different kinds of report, to pop up different dialogs and convert the responses back into something JWalk understands.

Tuesday, 10 February 2009

Still working on the new testing tool. The GUI looks fine now. I'm using the same components and panel-styles for the main test settings interface, and also for the custom settings interface. After trying out a few different combinations, I decided to move all custom settings out to a pair of dialogs. One of these allows you to upload different custom generators (using the new black-box style of architecture). The other allows you to configure the tool's more rarely altered settings, such as the convention on Object's methods, the depth of probing and the new parameter controlling state comparison. Also, the tool supports setting the three different working directories (for the test class, custom generators, and the oracle data file).

Monday, 2 February 2009

The first GUI-based production quality JWalk testing tool is now under development. I've managed to implement the two-stage class selection algorithm using file pathname differences to compute the package-qualified names. Getting the different panels to look right has been harder. I'm grouping Swing elements in the same style as Erik Tittel's team; but I'm using the kind of borders and general look-and-feel from Mihai-Gabriel Glont's team.

What is awkward in Java Swing is that sometimes you only have to insert the input field into a grid position in your panel, and the sizing works fine for the layout manager that you're using. Other times, adding a new component screws up the appearance of the other fields in parallel boxes in your grid. It's infuriating! I want to say "keep this column a fixed width", but cannot. I have to say "set the minimum (or maximum) dimensions of this entire panel to somesuch". Also, if you use a BoxLayout or GridLayout, your components occupy the whole of an equal division of this area. If you want a component to appear centred within such an area, you have to insert an extra dummy panel inside the panel and give the extra panel a FlowLayout. Obvious, huh?

Friday, 30 January 2009

We had to run a whole load of examples for the FACS journal paper for the Z to SAL translator. This has kept me away from JWalking for a bit. The other paper is ready now, so John will send this off to Jonathan Bowen at South Bank, who is in charge of FACS (Formal Aspects of Computer Science).

Tuesday, 20 January 2009

Got a first draft of the Z to SAL translator paper nearly ready. John and Siobhán will want to comment on this. I'll be back to JWalk as soon as this is done...

Wednesday, 14 January 2009

I've had to put off working on the production-quality revision to the JWalk toolset, because of an earlier commitment to write up a journal paper for FACS on our other model-checking work. This is giving new life to the Z specification language. We parse the LaTeX-input of the Z specification, then translate this into the output language for the SAL tools (which include a simulator and a model checker). Then we can check properties of the specification models (look for completeness, reachability of states and so on). This is joint work with John Derrick and Siobhán North.

My part in this is to work out the correct SAL language translation for the Z mathematical toolkit (all the sets, functions, relations and sequence types) and then Siobhán makes her parser-translator produce these encodings for the Z. John supplies lots of Z specifications to use as input to the translator. I then check the translated results, by simulation and by proposing LTL formulae for model checking. Anyhow, we wrote a paper for the ABZ (ASMS, B and Z) conference last Autumn, and the presentation went down so well, that we got asked to write the journal paper version for FACS.

Monday, 5 January 2009

Back again for the new year. Now that the students are all revising hard for examinations in a week or so, I can have a look at the best ideas for a GUI-based front end to the JWalk toolkit. The best student solutions for the stand-alone JWalk testing tool (built as applets, or full applications) were from the groups led by Erik Tittel and Mihai-Gabriel Glont. I think Mihai is quite interested to see what I make of his team's work. One of the things that really confused his team was the fact that you cannot simply use a Java filebrowser to select and upload the test class. This is because the browser knows nothing about Java packages and their naming conventions. You can't select a class from within its package and load it as though it were a simple class in the default package. You really need a two-stage loading process, where you select the working directory, from which you need to load the test class, then you go inside the package levels to pull out the class, but use the difference in file pathnames to compute the package-qualified name of the selected class, and then you can load it using this name. If you don't, Java's ClassLoader tells you that it found the class, but that it had the wrong name (annoyingly, it even knows what name you should have used!)

Back to earlier entries...