Mutation Testing
Universität des Saarlandes – Informatik
Informatik Campus des Saarlandes
Campus E9 1 (CISPA)
66123 Saarbrücken
E-mail: zeller @ cs.uni-saarland.de
Telefon: +49 681 302-70970
Mutation Testing
How do you know your test suite is "good enough"?One of the best ways to tell is mutation testing. Mutation testing seeds artificial defects (mutations) into a program and checks whether your test suite finds them. If it does not, this means your test suite is not adequate yet.
Despite its effectiveness, mutation testing has two issues. First, it requires large computing resources to re-run the test suite again and again. Second, and this is worse, a mutation to the program can keep the program's semantics unchanged -- and thus cannot be detected by any test. Such equivalent mutants act as false positives; they have to be assessed and isolated manually, which is an extremely tedious task.
Javalanche
The Javalanche framework for mutation testing of Java programs addresses both the problems of efficiency and equivalent mutants. Javalanche is built for efficiency from the ground up, manipulating byte code directly and allowing mutation testing of programs that are several orders of magnitude larger than earlier research subjects. Javalanche addresses the problem of equivalent mutants by assessing the impact of mutations on dynamic invariants: The more invariants impacted by a mutation, the more likely it is to be useful for improving test suites.We have evaluated JAVALANCHE on seven industrial-size programs, confirming its effectiveness. With less than 3% of equivalent mutants, our approach provides a precise and fully automatic measure of the adequacy of a test suite -- making mutation testing, finally, applicable in practice. Get Javalanche from here. Javalanche was presented at ESEC/FSE August 24-28 2009.
MuTest
The MuTest tool uses mutation analysis to drive automated test case generation. The main difference between using structural coverage and mutation analysis to guide test case generation is that a mutation does not only show where to test, but also helps in identifying what should be checked for. MuTest generates test suites that include assertions that are effective at detecting mutants. MuTest has been integrated into the EvoSuite tool, available herePapers
- Gordon Fraser, Andreas Zeller Mutation-driven Generation of Unit Tests and Oracles
- David Schuler, Andreas Zeller (Un-)Covering Equivalent Mutants, Resources: Manual classification of 140 Mutants, Subject programs
- David Schuler, Andreas Zeller Javalanche: Efficient mutation testing for Java
- David Schuler, Valentin Dallmeier, Andreas Zeller Efficient Mutation Testing by Checking Invariant Violations
- Bernhard J.M. Gruen, David Schuler, Andreas Zeller The Impact of Equivalent Mutants
Talks
Watch this presentation as a movie (with narration) (Quicktime, 45 minutes, 50 MB)
Resources
Instrumented Study Subjects
An instrumented version of Jaxen, with invariant checkers added by Javalanche can be downloaded from here and the documentation can be found here.Manual Study of mutations
The results of a manual study comparing invariant violating mutations to non-violating mutations as generated by Javalanche can be found here. The study is described in the paper Efficient Mutation Testing by Checking Invariant Violations
The results for a study of 20 randomly chosen mutations for Jaxen can be found here. The study is described in the paper The Impact of Equivalent Mutants
Contact us
Below you can find links to the people working on mutation testing. Please contact us for questions, rants, and raves.
- David Schuler
- Gordon Fraser
- Valentin Dallmeier
- Andreas Zeller
- Bernhard J.M. Gruen
Impressum ● Datenschutzerklärung
<webmaster@st.cs.uni-saarland.de> · http://www.st.cs.uni-saarland.de/mutation/ · Stand: 2018-04-05 13:41