Backstop: A Tool for
Debugging Runtime Errors
Chris Murphy, Eunhee Kim, Gail Kaiser, Adam Cannon
Columbia University
Observation

Novice/CS1 Java programmers have
difficulty dealing with uncaught exceptions
and the resulting stack traces
java.lang.NumberFormatException: For input string: "4.5"
at java.lang.NumberFormatException.forInputString
(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:456)
at java.lang.Integer.parseInt(Integer.java:497)
at Test.main(Test.java:124)
Problems

Students are often confused by runtime
errors in Java

Stack traces were not developed with the
introductory-level CS student in mind!

Few, if any, tools exist to help novice
programmers interpret and debug runtime
errors
Our Solution: Backstop

A command-line tool to produce detailed
and helpful error messages when an
uncaught runtime error (exception) occurs

Also provides debugging support by
displaying the lines of code that are
executed as a program runs, as well as
the values of any variables on those lines
Hypothesis

“Backstop allows students to understand
and debug runtime errors more easily than
just the Java stacktrace”

Backstop is part of Columbia University’s
ongoing efforts to develop tools to help CS1
students work with Java
Overview
Related Work
 Handling of Runtime Errors
 Usability Study
 Debugging Support
 Conclusions and Future Work

Related Work

Programming environments with easier-tounderstand compiler errors
 BlueJ,

ProfessorJ, DrJava, …
Identify most common compiler errors
 Gauntlet,

Identifying logical errors
 InSTEP,

Expresso, HiC, …
DEBUG, …
Debugging
 CMeRun
Handling of Runtime Errors

Designed to catch any uncaught
exceptions in the student’s program

Once the exception (or any other Java
runtime error) is caught, interpret its
meaning and provide a user-friendly
message that also seeks to provide some
enlightenment as to what caused the error
Handling of Runtime Errors

BackstopRuntime “wraps” the student’s
(compiled) Java program
 Uses
reflection to invoke the “main” method
 Catches any uncaught exception and
dispatches it to the appropriate handler

StackTraceElements and source code are
used to produce a friendlier, more helpful
error message that is appropriate for the
type of exception
*********************************************************
************************ ERROR **************************
*********************************************************
ReferAntoerror
“the occurred at line 124 of Test.java in the method
"main". I'm sorry, the program cannot continue.
code”, not
“you”
The line
of code that caused the error is
int myNum = Integer.parseInt(arg);
Reinforce what
is already known
It seems that the code wants to convert a String to an
integer.
However, the String "4.5" appears to be a floating point
value, not an integer. Don’t forget, an integer can’t
contain a decimal point. You may want to use a different
datatype to store the value, like float.
This is called a NumberFormatException.
Make a suggestion
The full error is as follows:
Alsojava.lang.NumberFormatException:
use Java
For input string: "4.5"
at java.lang.NumberFormatException.forInputString
terms
(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:456)
at java.lang.Integer.parseInt(Integer.java:497)
at Test.main(Test.java:124)
Usability Study

Research Question: “Is the information
provided by Backstop helpful for
understanding runtime errors?”
17 students (8 male, 9 female) who had
just completed a CS1 summer course
 Students were asked to find/fix a runtime
error in code they did not write
 After completion, were asked subjective
questions about the experience

Usability Study
1.
2.
3.
4.
5.
Subject was shown the code for a
program that reads a String and displays
the number of occurrences of each letter
Subject was then told how the algorithm
works
It was then suggested that the code
would produce an error for some inputs
Subject was suggested to try a sentence
like “hello world”
Backstop error appeared
Problem Code
// loop through and increment the values for each char
for (int i = 0; i < s.length(); i++)
{
// get the character at index i
char c = s.charAt(i);
// increment the value in the array of occurrences
// figure out its index by subtracting 'a' from it
occur[(int)(c - 'a')]++;
}
****************************************************
********************** ERROR ***********************
****************************************************
An error occurred at line 30 of StringCount.java in the
method "main". I'm sorry, the program cannot continue.
The line of code that caused the error is
occur[(int)(c - 'a')]++;
It seems that the code tried to use an illegal value as
an index in an array.
The code was trying to access an element at index -65
of the array called "occur".
The expression "(int)(c - 'a')" had the value -65 when
the error occurred.
Remember, you cannot have a negative index. Be sure
that the index is always positive.
This error is called an ArrayIndexOutOfBoundsException.
The full error is as follows:
java.lang.ArrayIndexOutOfBoundsException: -65
at StringCount.main(StringCount.java:30)
Questions

Was it more useful than the stacktrace?
 All

said “yes”
Did the output mislead you?
 Two

said “yes”; one admitted he didn’t read it
Describe the tone (vs. stack trace)
 Six
said it was “friendly”
 Four said “too long”
 Two said “too pedantic for a general audience”
 Three did not notice it
Observations

Most common complaint was that the
output is too long
 One
possibility is giving the option to see
more details or user-adjusted verbosity levels

Average time to complete the task with
Backstop was around 5.8 minutes
 Understanding
the cause took 1-2 minutes
General Debugging Support

Display each line of code as it gets executed
 File
name
 Line number
 Source code

Also display the values of any variables
Original source code
sum = x + y;
y = x;
Line number printed before code is
executed, in case of exception
Modified source code
System.out.print("\nFib.java:27: sum = x + y;");
System.out.print(" (x was " + x);System.out.print(" , y was
" + y );System.out.print(")"); sum = x + y;
System.out.print(" (sum is now " + sum + ")");
System.out.print("\nFib.java:29: y = x;");System.out.print("
(x was " + x);System.out.print(")");y = x;
Everything
on one line
System.out.print(" (y is now " + y + ")");
Verb tense indicates
to retain original line
number
Output at runtime
change of variable
values
Fib.java:27: sum = x + y; (x was 2 , y was 1) (sum is now 3)
Fib.java:29: y = x; (x was 2) (y is now 2)
StringCount.java:27: char c = s.charAt(i); (i was 0) (c is now h)
StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now
StringCount.java:27: char c = s.charAt(i); (i was 1) (c is now e)
StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now
StringCount.java:27: char c = s.charAt(i); (i was 2) (c is now l)
StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now
StringCount.java:27: char c = s.charAt(i); (i was 3) (c is now l)
StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now
StringCount.java:27: char c = s.charAt(i); (i was 4) (c is now o)
StringCount.java:30: occur[(int)(c - 'a')]++; (occur[(int)(c - 'a')] is now
StringCount.java:27: char c = s.charAt(i); (i was 5) (c is now !)
StringCount.java:30: occur[(int)(c - 'a')]++;
*************************************************************************
********************************* ERROR *********************************
*************************************************************************
An error occurred at line 30 of StringCount.java in the method "main".
I'm sorry, the program cannot continue.
1)
1)
1)
2)
1)
The line of code that caused the error is
occur[(int)(c - 'a')]++;
It seems that the code tried to use an illegal value as an index to an array.
The code was trying to access an element at index -64 of the array called "occur".
The variable "(int)(c - 'a')" had the value -64 when the error occured.
Remember, you cannot have a negative index. Be sure that the index is always positive.
This error is called an ArrayIndexOutOfBoundsException.
The full error is as follows:
java.lang.ArrayIndexOutOfBoundsException: -64
at StringCount.main(StringCount.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Questions

Was it helpful?
 All

but one said “yes”
Did the output mislead you?
 Five

said “yes”, but four said “only at first”
Describe the appearance
 Five
said more spacing would make it easier to read
 Four said more spacing would make it harder to read
 Three said the spacing would make no difference
Evaluation

Backstop helped students identify and fix
errors in code they did not write

Backstop automates/mimics the way that
many CS1 students debug a Java program
on the command line

Still need a study involving the use of both
parts of Backstop to debug a runtime error
Conclusion and Future Work

Our contribution: Backstop
 Makes
the error messages produced by
uncaught Java exceptions easier to
understand (compared to the Java stacktrace)
 Provides friendlier and more useful
information about how to fix the error
 Automates simple debugging approach

Future work could include integration with
an IDE, and further studies to measure
Backstop’s effectiveness
Backstop: A Tool for
Debugging Runtime Errors
Chris Murphy, Eunhee Kim, Gail Kaiser, Adam Cannon
Columbia University
http://www.cs.columbia.edu/~cmurphy/backstop
cmurphy@cs.columbia.edu