New EPSRC Grant: Growing Readable Software Tests

Recent studies suggest that 77% of the time that software developers spend with testing is used for reading tests. Tests are read when they are generated, when they are updated, fixed, or refactored, when they serve as API usage examples and specification, or during debugging. Reading and understanding tests can be challenging, and evidence suggests that, despite the popularity of unit testing frameworks and test-driven development, the majority of software developers do not practice testing actively. Automatically generated tests tend to be particularly unreadable, severely inhibiting the widespread use of automated test generation in practice. The effects of insufficient testing can be dramatic, with large economic damage, and the potential to harm people relying on software in safety critical applications. This project aims to improve the effectiveness and efficiency of testing by improving the readability of tests. We will investigate which syntactic and semantic aspects make tests readable, such that we can make readability measurable by modelling it. This, in turn, will allow us to provide techniques that guide manual or automatic improvement of the readability of software tests. This is made possible by a unique combination of machine learning, crowd sourcing, and search-based testing techniques. The GReaTest project will provide tools to developers that help them to identify readability problems, to automatically improve readability, and to automatically generate readability optimised test suites. The importance of readability and the usefulness of readability improvement will be evaluated with a range of empirical studies in conjunction with our industrial collaborators Microsoft, Google, and Barclays, investigating the relation of test readability to fault finding effectiveness, developer productivity, and software quality. (Details)

ASE 2015: ACM SIGSOFT Distinguished Paper Award

ACM SIGSOFT Distinguished Paper Award:

  • S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges,” in Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2015, pp. 201-211.
    [Bibtex]
    @inproceedings{ase15_faults,
      title={Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges},
      author={Shamshiri, Sina and Just, Rene and Rojas, Jos{\'e} Miguel and Fraser, Gordon and McMinn, Phil and Arcuri, Andrea},
      booktitle={Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
      year={2015},
      pages={201--211},
      organization={IEEE}
    }

30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015)

SSBSE 2015: Best Paper with industry-relevant SBSE results

Best Paper with industry-relevant SBSE results:

  • J. M. Rojas, J. Campos, M. Vivanti, G. Fraser, and A. Arcuri, “Combining Multiple Coverage Criteria in Search-Based Unit Test Generation,” in Search-Based Software Engineering, 2015, pp. 93-108.
    [Bibtex]
    @inproceedings{ssbse15_multi,
      title={Combining Multiple Coverage Criteria in Search-Based Unit Test Generation},
      author={Rojas, Jos{\'e} Miguel and Campos, Jos{\'e} and Vivanti, Mattia and Fraser, Gordon and Arcuri, Andrea},
      booktitle={Search-Based Software Engineering},
      pages={93--108},
      year={2015},
      publisher={Springer}
    }

International Symposium on Search-Based Software Engineering (SSBSE 2015)

ESEC/FSE 2015: ACM SIGSOFT Distinguished Paper Award

ACM SIGSOFT Distinguished Paper Award:

  • E. Daka, J. Campos, G. Fraser, J. Dorn, and W. Weimer, “Modeling readability to improve unit tests,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 107-118.
    [Bibtex]
    @inproceedings{daka2015modeling,
      title={Modeling readability to improve unit tests},
      author={Daka, Ermira and Campos, Jos{\'e} and Fraser, Gordon and Dorn, Jonathan and Weimer, Westley},
      booktitle={Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering},
      pages={107--118},
      year={2015},
      organization={ACM}
    }

10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2015)

GECCO 2015: Best paper award in SBSE track

Best Paper Award in the SBSE track:

  • S. Shamshiri, J. M. Rojas, G. Fraser, and P. McMinn, “Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation?,” in Proceedings of the 2015 on Genetic and Evolutionary Computation Conference, 2015, pp. 1367-1374.
    [Bibtex]
    @inproceedings{shamshiri2015random,
      title={Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation?},
      author={Shamshiri, Sina and Rojas, Jos{\'e} Miguel and Fraser, Gordon and McMinn, Phil},
      booktitle={Proceedings of the 2015 on Genetic and Evolutionary Computation Conference},
      pages={1367--1374},
      year={2015},
      organization={ACM}
    }

ACM Genetic and Evolutionary Computing Conference (GECCO 2015)

FSE 2014 Distinguished Paper Award: Are Mutants a Valid Substitute for Real Faults in Software Testing?

ACM Distinguished Paper Award:

  • R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, “Are mutants a valid substitute for real faults in software testing?,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014, pp. 654-665.
    [Bibtex]
    @inproceedings{just2014mutants,
      title={Are mutants a valid substitute for real faults in software testing?},
      author={Just, Ren{\'e} and Jalali, Darioush and Inozemtseva, Laura and Ernst, Michael D and Holmes, Reid and Fraser, Gordon},
      booktitle={Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering},
      pages={654--665},
      year={2014},
      organization={ACM}
    }

ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014)

ASE 2014 Distinguished Paper Award: Automated Unit Test Generation for Classes with Environment Dependencies

  • [PDF] A. Arcuri, G. Fraser, and J. P. Galeotti, “Automated Unit Test Generation for Classes with Environment Dependencies,” in IEEE/ACM Int. Conference on Automated Software Engineering (ASE), New York, NY, USA, 2014.
    [Bibtex]
    @inproceedings{ase14_environment,
        AUTHOR="Andrea Arcuri and Gordon Fraser and Juan Pablo Galeotti",
        TITLE="Automated Unit Test Generation for Classes with Environment Dependencies",
        booktitle={IEEE/ACM Int. Conference on Automated Software Engineering (ASE)},
        pages="",
        year = {2014},
        pages = {79--90},
        numpages = {12},
        publisher = {ACM},
        address = {New York, NY, USA},
    }

Recent EvoSuite papers

  • [PDF] G. Fraser and A. Arcuri, “A Large Scale Evaluation of Automated Unit Test Generation Using EvoSuite,” ACM Transactions on Software Engineering and Methodology, 2014.
    [Bibtex]
    @article{TOSEM_evaluation,
    author = {Gordon Fraser and Andrea Arcuri},
    title = {A Large Scale Evaluation of Automated Unit Test Generation Using EvoSuite},
    journal ={ACM Transactions on Software Engineering and Methodology},
    year={2014}, 
    publisher = {ACM},
    note={To appear}
    }

  • [PDF] A. Arcuri, G. Fraser, and J. P. Galeotti, “Automated Unit Test Generation for Classes with Environment Dependencies,” in IEEE/ACM Int. Conference on Automated Software Engineering (ASE), New York, NY, USA, 2014.
    [Bibtex]
    @inproceedings{ase14_environment,
        AUTHOR="Andrea Arcuri and Gordon Fraser and Juan Pablo Galeotti",
        TITLE="Automated Unit Test Generation for Classes with Environment Dependencies",
        booktitle={IEEE/ACM Int. Conference on Automated Software Engineering (ASE)},
        pages="",
        year = {2014},
        pages = {79--90},
        numpages = {12},
        publisher = {ACM},
        address = {New York, NY, USA},
    }

  • [PDF] J. Campos, A. Arcuri, G. Fraser, and R. Abreu, “Continuous Test Generation: Enhancing Continuous Integration with Automated Test Generation,” in IEEE/ACM Int. Conference on Automated Software Engineering (ASE), New York, NY, USA, 2014, pp. 55-66.
    [Bibtex]
    @inproceedings{ase14_ctg,
        AUTHOR="Jose Campos and Andrea Arcuri and Gordon Fraser and Rui Abreu",
        TITLE="Continuous Test Generation: Enhancing Continuous Integration with Automated Test Generation",
        booktitle={IEEE/ACM Int. Conference on Automated Software Engineering (ASE)},
        year = {2014},
        location = {Vasteras, Sweden},
        pages = {55--66},
        numpages = {12},
        publisher = {ACM},
        address = {New York, NY, USA},
    }

  • [PDF] G. Fraser and A. Arcuri, “EvoSuite at the Second Unit Testing Tool Competition,” in Future Internet Testing, 2014, pp. 95-100.
    [Bibtex]
    @inproceedings{fittest2014,
      year={2014},
      booktitle={Future Internet Testing},
      series={Lecture Notes in Computer Science},
      editor={Vos, Tanja E.J. and Lakhotia, Kiran and Bauersfeld, Sebastian},
      title={EvoSuite at the Second Unit Testing Tool Competition},
      publisher={Springer International Publishing},
      author={Fraser, Gordon and Arcuri, Andrea},
      pages={95-100},
    }

  • [PDF] J. P. Galeotti, G. Fraser, and A. Arcuri, “Extending a Search-Based Test Generator with Adaptive Dynamic Symbolic Execution (Tool paper),” in Proceedings of the 2014 International Symposium on Software Testing and Analysis, New York, NY, USA, 2014.
    [Bibtex]
    @inproceedings{issta14_dse,
     author = {Galeotti, Juan Pablo and Fraser, Gordon and Arcuri, Andrea},
     title = {Extending a Search-Based Test Generator with Adaptive Dynamic Symbolic Execution (Tool paper)},
     booktitle = {Proceedings of the 2014 International Symposium on Software Testing and Analysis},
     series = {ISSTA '14},
     year = {2014},
     numpages = {4},
     publisher = {ACM},
     address = {New York, NY, USA},
     note = {To appear}
    }

(Video available here)

  • [PDF] G. Fraser, A. Arcuri, and P. McMinn, “A Memetic Algorithm for Whole Test Suite Generation,” Journal of Systems and Software, 2014.
    [Bibtex]
    @article{jss14_memetic,
       year={2014},
       journal={Journal of Systems and Software},
       title={A Memetic Algorithm for Whole Test Suite Generation},
       publisher={Elsevier},
       author={Fraser, Gordon and Arcuri, Andrea and McMinn, Phil},
       language={English},
       note = {To appear}
    }

  • [PDF] A. Arcuri and G. Fraser, “On The Effectiveness of Whole Test Suite Generation,” in Proceedings of the Sixth International Conference on Search Based Software Engineering, Berlin, Heidelberg, 2014.
    [Bibtex]
    @inproceedings{ssbse14,
     author = {Arcuri, Andrea and Fraser, Gordon},
     title = {On The Effectiveness of Whole Test Suite Generation},
     booktitle = {Proceedings of the Sixth International Conference on Search Based Software Engineering},
     series = {SSBSE'14},
     year = {2014},
     publisher = {Springer-Verlag},
     address = {Berlin, Heidelberg},
     note = {To appear}
    }

  • [PDF] G. Fraser and A. Arcuri, “Achieving Scalable Mutation-based Generation of Whole Test Suites,” Empirical Software Engineering, 2014.
    [Bibtex]
    @article{emse14_mutation,
       year={2014},
       journal={Empirical Software Engineering},
       title={Achieving Scalable Mutation-based Generation of Whole Test Suites},
       publisher={Springer US},
       author={Fraser, Gordon and Arcuri, Andrea},
       language={English},
       note = {To appear}
    }

  • [PDF] G. Fraser and A. Arcuri, “1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite,” Empirical Software Engineering, 2013.
    [Bibtex]
    @article{emse13_oracle,
       year={2013},
       journal={Empirical Software Engineering},
       title={1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite},
       publisher={Springer US},
       author={Fraser, Gordon and Arcuri, Andrea},
       language={English},
       note = {To appear}
    }