|
|
Frequently Asked Questions |
|
|
AskIgor ("Ask Igor") is an experimental automated debugging server that tells you why your program fails.
Igor computes a diagnosis that tells you why your program failed. This diagnosis has the form of a cause-effect chain—first, this happened, therefore, that happened, and therefore the program failed:
- This is what happens in your program when it is invoked as "sample 11 14".
1 Execution reaches line 31 of sample.c in main. Since the program was invoked as "sample 11 14", local variable argc is now 3.
2 Execution reaches line 9 of sample.c in shell_sort. Since argc was 3, variable a[2] at main, the calling function, is now 0.
3 Execution ends. Since a[2] was 0, the output now contains the word "0".
The program fails.In each step ("Execution reaches..."), Igor isolates the failure-inducing variable values—out of up to 70.000 values.
In other words: Igor finds the needle in the haystack that causes the failure.Of course, to truly value a diagnosis like this, you must relate it to the submitted program. Nonetheless, we have found so far that such cause-effect chains give a precise summary of how failures come to be, and we expect them to ease debugging considerably.
You use AskIgor in three steps:This diagram contrasts the traditional debugging process and the AskIgor process:
- Your program fails in some way.
- You submit the executable via the AskIgor web site.
- After a short period of time, Igor computes a diagnosis which you can read on the AskIgor web site.
In future, automated debugging as in AskIgor will become part of development environments, such that submission over the Web is no longer necessary. We are currently realizing appropriate plug-ins for the Eclipse environment.
Igor is the name of the program which does the debugging work behind the scenes. You can access it via AskIgor—the public Web service you're looking at.
Igor is the lab assistant in several Frankenstein movies. He does the dumb and ugly work, while being absolutely loyal to his master. For us, Igor was just the perfect name for an automated debugging gofer. "Igor! Go for bugs!"Actually, Igor (pronounced EE'gore) is a common russian first name, originating from ancient Scandinavian "Ivor", meaning "bow warrior". You might also think of Igor as a hero who hits even the most difficult target.
Briefly spoken, Igor works as follows:For details, see the scientific report Isolating Cause-Effect Chains from Computer Programs or see its presentation.
- We take two program runs, a passing and a failing one.
- The cause of the failure lies in the difference between these two runs.
- Using a number of systematic tests, our Delta Debugging algorithm systematically narrows down the difference between program states until only a few relevant variables remain.
- These variables and their values cause the failure.
Actually, this is what AskIgor is for: To evaluate the effectiveness of our approach. But yes, we have provided concise and precise diagnoses for programs as large as the GNU compiler (GCC). Again, get the scientific report Isolating Cause-Effect Chains from Computer Programs or see its presentation.
In the last years, we have developed a number of approaches that provide very precise diagnoses of program failures. As good scientists, we want to evaluate these approaches. The deal is simple: We get your buggy program, you get our diagnoses, we get your rating.
A command-line version of Igor is available at the download site.Besides the command-line version, we are currently building Eclipse plug-ins that realize delta debugging in the Eclipse programming environment. These plug-ins are licensed under the Eclipse open source license. We're also working on an Eclipse plug-in that realizes AskIgor functionality in Eclipse.
AskIgor was developed by Andreas Zeller and his students at the Software engineering chair at Saarland University, Saarbrücken, Germany. Professor Zeller is perhaps best known to programmers by being co-author and long-time maintainer of GNU DDD, a popular interactive debugger for Linux and Unix systems.
See the Press Center for text and image resources.
Have a look at Zeller's presentation of the subject. Also, see the Press Center for image resources.
Our infrastructure currently only supports Linux i386 programs—that is, i386 machine code executables for Linux. The program submitted must run in batch mode—that is, fully automated—and be invoked from the command line.
Any, as long as the resulting executable can be debugged using the GNU debugger (GDB). So far, best results are obtained for C programs.
Right now, our infrastructure does not provide an X server. Support for X programs is under construction, though.
Not now. Our approach requires testing and executing submitted programs locally, and we do not have the resources to install and maintain a large number of different architectures. Remember that this is a research project, not a commercial service.
Not within AskIgor. However, we are currently building Eclipse plug-ins that realize delta debugging in the Eclipse programming environment. One of these plug-ins implements the AskIgor functionality for Java™ programs.
Not now. We currently do not have the resources for automated debugging of further interpreted languages.
To be effective, Igor must be able to capture and transfer an entire program state from one process to another. Igor can do this with program data. However, interaction with external resources like windows, files, networking, shared memory, must each be supported separately. Right now, only few of these external resources are supported; we are working on it.In C programs, it is a challenge to capture the program's data structures accurately. Igor thus relies on heuristics to determine at how may elements a pointer points to, or to determine the correct member of a union. These heuristics are quite good, but may occasionally fail.
If your program has few interactions with its environment and not-too-weird data structures, though, then neither size nor complexity of programs or data impose a limit. In some experiments, we have automatically determined diagnoses for programs as large and complex as the GNU compiler (GCC). For more details, get the scientific report Isolating Cause-Effect Chains from Computer Programs or see the presentation.
AskIgor has a built-in limit of 15 minutes per debugging task. After this limit, the current debugging task is aborted and the next pending one is processed.Running AskIgor on GCC, for instance, currently requires between 90 minutes and 4 hours. We are currently working on GDB extensions that will speed up AskIgor by a factor of five; if demand is sufficient, we can also increase our local computing power.
We have installed a Debian 3.0 "woody" Linux distribution in all our sandboxes.
We found that Debian offers the most complete distribution on the market; it is also the distribution we use for development and production. So it was a natural choice for our sandboxes.
Unfortunately, we cannot fulfil such requests. In general, your submission must be self-contained. If some library or other resource is missing, feel free to provide it with your program.
The general rule is to redirect the interactive input such that it is taken from a file or from the command line (which you submit together with the program).If your program has a graphical user interface which cannot be separated from the core functionality, be patient—we are currently evaluating various techniques for recording and replaying interaction in a way that is suitable for AskIgor.
As of November, 2003, Igor can compute a diagnosis for only about a third of all submissions; however, only 15% of all problems are actually in our control. This is due to a variety of causes (most frequent first):In all cases, we're trying to provide helpful diagnostics. Let us know if you have further suggestions.
- Your executable does not run on our machine. ("Same_Runs", 44%) This results in error messages like "libfoo.so: not found".
- Your executable was not invoked. ("Not_Invoked" or "Pass_Not_Invoked", 22%) This is typically due to a problem in specifying the invocation.
- Your executable has no or incomplete debugging information. ("No_Symbols", 19%) AskIgor tries to check for debugging information when you submit the program, but at this stage, it can only check for the basics. Be sure to compile your entire program with the "-g" option.
- Internal AskIgor error. ("Internal", 9%) Yes, AskIgor has bugs, too. When such a bug occurs, we typically fix it within 24 hours. And yes, we frequently use Delta Debugging for fixing AskIgor bugs.
- Your program is too complex. ("State_Transfer", 6%) AskIgor is unable to capture the entire state of your program. We're working on this.
Each variable reported in the cause-effect chain is a cause for the failure—that is, altering it to some other value (typically, the value found in the non-failing run) makes the failure disappear. So, all you have to do to "fix" the bug is to change exactly that value.However, such a "fix" would fix only the symptoms, not the cause. It is usually wiser to rewrite to apply a most general fix—in other words, to fix not only the one single failure, but all other related failures, too. Finding such a most general fix is part of writing a correct program—a task still left to you.
Igor currently uses a coverage metric: It examines the program as soon as 50%, 75% and 90% of all functions have been executed at least once. Using the How did this happen feature, you can make Igor focus upon a specific interval.
This is a local variable of the N'th calling function. When Igor examines a program,and so on, until the outermost calling function (typically main) is reached.
- the variables of the currently executing function are local variables
- the local variables of the calling function are in the calling function #1,
- the local variables of the function calling this function are in the calling function #2,
The number N is also known as the number of the current stack frame. In GDB, you can use the frame N command to select the calling function N.
Sorry, no. Actually, we hate debugging—this is why we're building AskIgor.
The symptoms must hold for the passing and the failing run, respectively. This means that Igor has to execute the two runs in order to present a sensible choice to the user. (Igor can only handle symptoms that actually occur in the runs.) And while Igor is reproducing the two runs, it may as well guess failure symptoms and produce a first diagnosis.
Simply have your program issue a last word that distinguishes passing from failing runs. For instance, your program could issueIf that's too much trouble for you, just have Igor guess the symptoms—you can always fix them later.
- "FAIL" if the failure in question occurred as in the failing run,
- "PASS" if all works well as in the passing run,
- "UNRESOLVED" in all other cases.
We execute all submitted programs in sandboxes, where programs can not make persistent changes or contact other sites.
Submissions are processed on a first-come, first-served basis, so it may take some time until Igor provides a diagnosis. The Status page gives a summary about the number of pending submissions.
In future, we anticipate tools that run automated debugging services right on your box. Typically, such tools would be integrated with an automated test suite such as JUnit; whenever a test fails, you will not only be told that it failed, but also why it failed. If all works well, the need for manual debugging and interactive debugging tools will be reduced dramatically.We're currently working on Delta Debugging tools for the Eclipse programming environment that realize simple automated debugging techniques like simplifying input or isolating failure-inducing changes.
Right now, though, we're already happy to have AskIgor up and running in our place, and we definitely want and need more experiences before spreading it to the world.
Yes, you can! AskIgor's automated debugging techniques can easily be built on top of any automated testing technique; whenever a test fails, you will not only be told that it failed, but also why it failed. If all works well, the need for manual debugging and interactive debugging tools will be reduced dramatically.If you are interested in using AskIgor technologies for your tool or programming environment, please contact Prof. Andreas Zeller of Saarland University.
Definitely not! No manager with a sane mind would ever allow programs to leave the company. Right now, we're simply interested in running a big case study on the effectiveness of our techniques. These techniques will hopefully find their way into automated debugging tools for everyone.
As of March 2004, Andreas Zeller (the original inventor) and Passau University (his employer) have let the patent expire. To the best of our knowledge, all of Delta Debugging and AskIgor should now be patent-free; a command-line version is available as open source.
Negative.
Please visit our research web site about Delta Debugging.
| Copyright © 2002-2005 Saarland University |
| www.askigor.org · News · FAQ · Press · Downloads · Usage · Terms of service |
|
