The JWalk Weblog, 2010

This is the JWalk weblog for 2010! It describes how we produced an upgrade to the tools after interaction with Beta-users in the aerospace industry, who discovered some problems with class reloading, when they were also using CustomGenerator objects of their own. We successfully pursued this problem and provided the fix. See entries in the blog for 9-12 March for the details!

The year was very busy (no entries after September), due to the author having to prepare a new module, while teaching it! However, the last entries will bring a tear to the eyes of those who remember the magnificent Testing: Academic and Industrial Conference - Practice and Research Techniques, traditionally held at Cumberland Lodge in Windsor Great Park. This was the last year we could afford to go there, due to attendance being sucked away by the new IEEE International Conference on Software Testing. From 2011, TAIC PART will amalgamate with ICST.

Sunday, 5 September 2010

Today was the last day of the 5th Testing: Academic and Industrial Conference - Practice and Research Techniques, held on 3-5 September 2010, Cumberland Lodge, Windsor Great Park. You can see some photographs of the delegates, the keynote speakers and the conference committee members on the conference website, both during the working sessions and taking some fresh air, walking about in the wonderful grounds.

Today we had our final keynote speech by Prof Sir Tony Hoare, on his lifetime of experience trying to improve the quality of software by verification and testing. He described what seemed like two careers, including the second one he had at Microsoft Research after retiring from his academic post. Both worlds contributed different, useful things, which seemed appropriate for the academic and industrial emphasis of TAIC PART. It was wonderful to have, at this last conference in Windsor, both the man who invented pre- and postconditions, and the man who made them a practical reality in a programming language.

Saturday, 4 September 2010

Today I continued as conference chair of the 5th Testing: Academic and Industrial Conference - Practice and Research Techniques, held on 3-5 September 2010, Cumberland Lodge, Windsor Great Park. Today we had another fascinating keynote speech by Dr Woflgang Grieskamp about the use of models for testing the validity of user documentation. At Microsoft, they have a way of specifying abstract models of what user documentation appears to be saying, then mapping from the abstract system execution to the concrete system, to check whether the two are identical. This was quite a novel application of modelling to me, and should be quite useful when determining whether the documentation is accurate, or up to date.

Friday, 3 September 2010

Today was the start of the 5th Testing: Academic and Industrial Conference - Practice and Research Techniques, held on 3-5 September 2010, Cumberland Lodge, Windsor Great Park. TAIC PART has been for many years the premier world conference on software testing research and practice, until the IEEE belatedly launched the 1st International Conference on Software Testing, held on 9-11 April 2008, Lillehammer, Norway. From 2011, TAIC PART will be a joint event with ICST, recognising that we cannot compete against the IEEE!

Today we had a brilliant keynote speech by Prof Bertrand Meyer about the latest tools for testing Eiffel. The approach uses a random test selection method that makes use of Eiffel's own assertions as a way of controlling test case selection, and again for providing the test oracles. Random test cases that satisfy the test object's preconditions are chosen as valid test cases; then cases which violate the test object's postconditions are flagged as indicating errors in the software. This has proved to be surprisingly effective, even when random tests were used. I think partly this is because of the filtering of test cases by preconditions, and the way in which usable test objects are preserved in the pool until their state is corrupted by some test failure. Otherwise, random testing would produce a lot of useless test cases that either failed immediately, or repeatedly tested the same properties.

Thursday, 17 June 2010

While out in Greece, I also gave a seminar on JWalk, presented at SEERC (South East European Research Centre). These slides will soon be appearing on the JWalk website, on the Publications Page. Two academics from City College, Thessaloniki have asked for licenses to use JWalk on their undergraduate programme, to allow beginners learning Java to test their own classes as they go; and also in research as a test-driver for testing Java-based web-services by a different means than the approach involving explicit specifications.

With Dimitris Dranidis and Ervin Ramollari, I managed to test the Java API of a web-service implementation directly (before this was wrapped in WSDL and SOAP). We have also managed in the past to test a Java client-stub that had been hand-written to look like a proxy for the original service. We can't yet use JWalk to test automatically generated thin-client stubs, like those generated using WSDL2Java in Apache Axis. This is due to the impedance mismatch between the generated API and the ideal Java proxy API. On the version of WSDL2Java we used, the generated client required Java SOAP objects (although the Apache Axis website seems to show generation of client-stubs whose API looked much more like regular Java signatures and types - are we using an old version of WSDL2Java?). We may find a way around this in future. Watch this space!

Wednesday, 16 June 2010

I'm in Thessaloniki again this week, supervising my remote PhD students at SEERC (South East European Research Centre). We have some good testing work going on here - developing new ways to test web-services completely from extended finite-state machine (EFSM) specifications. One strand of the work involves testing from specifications prior to deployment; the other strand involves run-time monitoring of deployed services. We have the state specifications and the complete test generation algorithm; the main technical issues involve the mapping between abstract and concrete test sequences (or lifting and grounding, in the technical literature). The SOA team out here is doing excellent work and are part of several EU projects.

Monday, 7 June 2010

JWalk has landed in South America! I've just had my first request from Brazil for a JWalk license. This request was from a student at the Federal University of Piauí (UFPI), who is doing research into software testing and would like to compare JWalk against other unit testing tools. How strange is it that, in the last month, JWalk has landed on three new continents? The only continent not covered yet is Antarctica. Does anyone know someone on the Antarctic Survey?

Monday, 24 May 2010

JWalk has landed in Australasia! I've just had a license request from a student at Swinburne University of Technology, Melbourne, Australia. He is going to compare JWalk against JUnit4, TestNG and JTest, as part of a review for a software testing module. I know SUT quite well, as I spent a month there in 1996 working with Prof. Brian Henderson-Sellers on an alternative to the original UML proposal.

Tuesday, 18 May 2010

JWalk has landed in Asia! I've just had my first request from China for a JWalk license. This request was from a software tester working for the Chengdu Tianfu Software Park Company, in Chengdu, China. I'm quite excited about this! China is becoming a really important country for software development, like the already-famous centres in India. By the way, we already have three JWalk installations in India!

Friday, 30 April 2010

This week was really punishing! On top of a lot of teaching work with my students, I have managed to submit Nastassia's revised paper, suitably formatted in the correct way for an LNCS proceedings, to TAIC PART 2010. For reasons of conflicts of interest, I cannot claim any credit for this, since I am the conference chair!

Friday, 23 April 2010

I liked the student paper that Nastassia Smeets had written so much (see the blog for 17 March) that I've asked if she would like to submit a real paper for TAIC PART 2010. She says she's up for it, so here we go. I will check out all the facts and fill out the references. She says that one area her professor found weak was the conclusions, where he had expected a more systematic point-by-point comparison. So, I guess I'll act as second supervisor for this revision! The image on the right is one of hers - click on this to get an enlarged view of how JWalk looks on an Apple Mac!

Friday, 19 March 2010

I have made a new build of JWalk 1.0 today, with a refactored version of the solution uploaded last week. Now, the org.jwalk package has two explicit class loaders, called GeneratorLoader and TestClassLoader. The JWalk runtime creates a single instance of the former for the duration, and creates a fresh instance of the latter every time the test class is reloaded. Even if the tester resets the test class directory or generator directory, the original GeneratorLoader stays the same, but simply adds the new URLs to its search paths.

I have tested this upgrade both inside and outside of the Eclipse development framework. I have put traces on which class loader classes are used to load every custom generator and every class known to the test class. The behaviour looks right to me: generators that were part of the JWalk distribution are loaded with the AppClassLoader that loaded JWalk. Unseen custom generators are loaded with GeneratorLoader. The test class itself is loaded with TestClassLoader, but any other unseen classes it brings in are loaded with the same GeneratorLoader that loaded the custom generators.

Wednesday, 17 March 2010

Nastassia Smeets, a Masters student at the University of Antwerp, has written back to say she has now finished her student paper that compares Randoop, JWalk and MuJava. This sounded like such a good idea that I downloaded the whole paper. It's pretty good, and needs a little polishing in the conclusions. Maybe we could collaborate on a larger version of this?

Friday, 12 March 2010

I have tested the fix to the multiple ClassLoader problem both under Unix (Solaris) and Windows (XP) and the desired behaviour is now observed when loading CustomGenerator classes that refer to user-defined (external) classes. In these tests, I launched JWalk both by double-clicking the compressed archive file, and by starting Java in the normal manner on the unbundled software. I also tested Mark Tinker's scenario where the test classes were in the launch directory, and when they were in some other subdirectory or super directory. All test scenarios appear to work as intended.

I have therefore now compiled a new build of JWalk1.0 and placed this on the website. You can access this upgraded version in the usual way on the Download Page. I have also regenerated all the javadoc documentation to reflect the changes, which can be accessed from the Look Inside Page, which is the same as the documentation in the download bundle. Finally, I have decided to move the package location of one of the interfaces - MasterGenerator now resides in the package org.jwalk.gen alongside CustomGenerator. Previously, the need to import this type was missing from the online instructions on how to Customise JWalk, so I have explicitly added this import instruction.

Note: all current documentation refers to the upgraded version. If you are using the pre-upgrade version, then MasterGenerator resides in its old location org.jwalk.core. I guess we put the interface there in the first place to be alongside the standard generator classes that implement it. We overlooked the notion that users would need to import this interface in their own custom generators. It makes more sense for all generator imports to come from a single package, from the user's point of view. We hope you appreciate this tidier packaging!

Thursday, 11 March 2010

I've been working round the clock to try and find a solution to the problem isolated yesterday. Information on Java ClassLoaders is available on the web, but appears to be of variable quality. I have tried two published solutions to the problem of selectively reloading classes, only to find that they don't work. Either, the class in question is cached and not reloaded, or all referenced classes on which it depends are also reloaded multiple times, or the published code example is somehow malformed and doesn't execute properly.

Finally, I have settled on the following solution: a single URLClassLoader is created after the custom generator directory has been set. This loader is used to load all user-supplied CustomGenerator classes, which can only be loaded once into each Java runtime (each launch of the tool). Then, a bespoke dynamic subclass of URLClassLoader is created each time the test class is reloaded. This loader statically binds the name of the test class internally, such that when loading, it recognises this name alone as special and reloads the class each time. Otherwise (and this is the important part), it delegates the request to load any other class to the same loader that loaded any custom generators. In this way, further user-defined classes referred to both by the test class and by custom generators will be resolved to the same type by the Java type system.

Today is also my father's 88th birthday! Happy Birthday, Dad!

Wednesday, 10 March 2010

Mark Tinker was right - there is an issue with resolving types when uploading certain user-supplied CustomGenerator classes. The failure scenario is as follows: the user wishes to test a class that has methods containing abstract types in their signatures. So, the user creates a CustomGenerator to synthesise suitable values of a corresponding concrete type instead. The JWalk tools are intended to support this kind of customisation. However, when the user-supplied generator is uploaded, JWalk still seems unable to synthesise concrete values for the abstract arguments.

Looking into this more closely, I have discovered that this fault derives from the behavior of Java ClassLoaders. We previously had problems with these when developing the JWalkEditor (see entries for March, 2008). Then, we discovered that we had to create all-new loaders each time we wanted to reload the test class into the current Java runtime, assuming it had been modified and recompiled in the meantime. This was finessed by having a special loader that did not delegate to the default loader, which caches loaded classes and so prevents reloading.

The new problem is the following: if the test class and a custom generator class both refer to a third user-defined class (this problem does not occur with Java standard library classes, nor with primitive types), then the third class will be loaded twice, by two different ClassLoader objects. This means that the Java runtime will think they are two distinct and mutually-incompatible types. For this reason, the type handled by the user's CustomGenerator is not recognised as the same type for which a value is to be synthesised, in other words, the delegate CustomGenerator fails to intercept requests to handle this type. Therefore, the default ObjectGenerator is used instead and the observed behaviour is as if no custom generator had been loaded.

Tuesday, 9 March 2010

Today I went down to ITP Engines, UK Ltd., to lead an in-house bespoke training day. Their software engineers from the heat transfer group are a great bunch, providing a unique service to calculate the heating profile on spacecraft, based on the spacecraft's configuration and mission. They want to document their software using the latest notations, so I prepared some instruction materials on the latest UML 2.2 notation, particularly focusing on procedural design using the Activity Diagram.

While there, I got to speak to Mark Tinker, who previously requested a JWalk evaluation license. He says he has had difficulty in getting the tool to recognise a bespoke CustomGenerator he uploads, to create specimen concrete values for arguments with abstract types. The tool is supposed to support doing this. After checking through what he had done, I'm convinced he's doing the right thing. I tried and failed to reproduce the same fault on my laptop installation of JWalk, so I will look into this problem again tomorrow.

Wednesday, 3 March 2010

Received another response to the request for TAIC PART sponsors today. Unfortunately, SmartTesting is already committed to ICST and cannot stretch to TAIC PART as well this year.

Friday, 26 February 2010

Some good news and bad news. The company LDRA has agreed once again to offer some soft sponsorship for TAIC PART 2010. LDRA is the company behind the development and licensing of the MISRA-C standard for automotive software. Although I have approached about 20 companies, LDRA is the only positive response so far (thanks!) I have had a polite declining message from IPL Bath, explaining that they are too busy even to attend.

Friday, 19 February 2010

Today we published the call for papers for the forthcoming conference TAIC PART 2010. If you work in the area of software testing, either in academia or in industry, please consider submitting a paper, and coming to the conference in Windsor Great Park, next September.

Wednesday, 17 February 2010

Gordon Fraser has finally closed the deal with Springer Verlag to publish the conference proceedings for the forthcoming software testing conference TAIC PART 2010. This is good news. We also have a deal with Software Practice and Experience to publish a special edition after the main conference, featuring extended versions of the best papers.

Wednesday, 3 February 2010

I have just received another request from a student completing his bachelor's thesis at TH Wildau in Germany, who would like an academic license to use JWalk. Thanks for signing up, and please send me feedback afterwards.

Friday, 29 January 2010

Gordon Fraser has kindly prepared an invitation to send to the journal Software Practice and Experience, which we have selected as a potential journal for publishing the special issue, after the software testing conference TAIC PART 2010. Gordon, Len and I will be the editors for the special edition.

Thursday, 14 January 2010

Today I received a request from a professor in Ottawa for an academic license to try out JWalk as a possible tool for students to use on his software engineering course. Thanks for signing up, and please feel free to send feedback, as this is very useful to our JWalk developers.

Wednesday, 13 January 2010

Here we open the new blog for 2010! Well, the TAIC PART steering committee agreed that we should try a different publisher from CPS, the publishing arm of IEEE. Today I received news from Len Botacci that the ACM will only consider publishing the proceedings for TAIC PART 2010 if the conference was jointly organised by an ACM Special Interest Group. As we have our own steering committee of long standing, it seems pointless to argue this further. Next, we shall approach Springer.

Back to earlier entries...