EXSYST: Exploring Realistic Program Behavior (Technical Report)
by
Florian Gross, Gordon Fraser, Andreas Zeller
March 2012. This is a preliminary version of a paper that is currently under submission to ISSTA 2012.
Abstract
Modern test case generation techniques can automatically achieve high code coverage. If they operate on the unit level, they run the risk of generating nonsensical inputs, which, when causing failures, are painful to identify and eliminate. Running a unit test generator on five open source Java programs, we found that all of the 181 reported failures were false failures -- that is, indicating a problem in the generated test case rather than the program. By generating test cases at the GUI level, our EXSYST prototype can avoid such false failures by construction. In our evaluation, it achieves higher coverage than search-based test generators at the unit level; yet, every failure can be shown to be caused by a real sequence of input events. Whenever a system interface is available, we recommend considering search-based system testing as an alternative to avoid false failures.
BibTeX Entry
@unpublished{fgross-tr-2012,
title = "EXSYST: Exploring Realistic Program Behavior (Technical Report)",
author = "Florian Gross and Gordon Fraser and Andreas Zeller",
year = "2012",
month = mar,
}