Mutation Testing

Software Engineering Chair (Prof. Zeller)
Saarland University – Computer Science
Campus E1 1
66123 Saarbrücken, Germany
E-mail: zeller @ cs.uni-saarland.de
Phone: +49 681 302-70970

Deutschsprachige Startseite Page d'acceuil en franšais English home page
  

Mutation Testing

How do you know your test suite is "good enough"?

One of the best ways to tell is mutation testing. Mutation testing seeds artificial defects (mutations) into a program and checks whether your test suite finds them. If it does not, this means your test suite is not adequate yet.

Despite its effectiveness, mutation testing has two issues. First, it requires large computing resources to re-run the test suite again and again. Second, and this is worse, a mutation to the program can keep the program's semantics unchanged -- and thus cannot be detected by any test. Such equivalent mutants act as false positives; they have to be assessed and isolated manually, which is an extremely tedious task.

Javalanche

The Javalanche framework for mutation testing of Java programs addresses both the problems of efficiency and equivalent mutants. Javalanche is built for efficiency from the ground up, manipulating byte code directly and allowing mutation testing of programs that are several orders of magnitude larger than earlier research subjects. Javalanche addresses the problem of equivalent mutants by assessing the impact of mutations on dynamic invariants: The more invariants impacted by a mutation, the more likely it is to be useful for improving test suites.

We have evaluated JAVALANCHE on seven industrial-size programs, confirming its effectiveness. With less than 3% of equivalent mutants, our approach provides a precise and fully automatic measure of the adequacy of a test suite -- making mutation testing, finally, applicable in practice. Get Javalanche from here. Javalanche was presented at ESEC/FSE August 24-28 2009.

MuTest

The MuTest tool uses mutation analysis to drive automated test case generation. The main difference between using structural coverage and mutation analysis to guide test case generation is that a mutation does not only show where to test, but also helps in identifying what should be checked for. MuTest generates test suites that include assertions that are effective at detecting mutants. MuTest has been integrated into the EvoSuite tool, available here

Papers

Talks


Watch this presentation as a movie (with narration) (Quicktime, 45 minutes, 50 MB)

Resources

Instrumented Study Subjects

An instrumented version of Jaxen, with invariant checkers added by Javalanche can be downloaded from here and the documentation can be found here.

Manual Study of mutations

The results of a manual study comparing invariant violating mutations to non-violating mutations as generated by Javalanche can be found here. The study is described in the paper Efficient Mutation Testing by Checking Invariant Violations

The results for a study of 20 randomly chosen mutations for Jaxen can be found here. The study is described in the paper The Impact of Equivalent Mutants

Contact us

Below you can find links to the people working on mutation testing. Please contact us for questions, rants, and raves.

<webmaster@st.cs.uni-saarland.de> · http://www.st.cs.uni-saarland.de/mutation/ · Updated: 2014-03-23 23:44