MS thesis/project ideas
for MS students in the Department of Computer Science
at the University of Minnesota Duluth
suggestions by Dr Andrew Brooks, Heller Hall 334
http://www.d.umn.edu/~abrooks/
abrooks@d.umn.edu
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
1
Some presentation material drawn directly from article(s).
background research article
Kanewala, U.; Bieman, J.M.,
"Techniques for testing scientific programs without an oracle,"
Software Engineering for Computational Science and Engineering
(SE-CSE), 2013 5th International Workshop on , pp.48-57, 2013
doi: 10.1109/SECSE.2013.6615099
http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6615099
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
2
Some presentation material drawn directly from article(s).
A gentle introduction to oracles:
A Taxonomy for Test Oracles, Douglas Hoffman,
Software Quality Week (QW’98), 1998, 8pp.
http://www.softwarequalitymethods.com/Papers/OracleTax.pdf
A gentle introduction to assertions:
Chapter 6 Assertion facility, in JDK 1.4 Tutorial, Greg Travis,
Manning Publications Company, 2002.
http://www.oracle.com/technetwork/articles/javase/javapch06.pdf
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
3
the difficulty of testing software
Test case identifier
Test case values
Expected results
Actual results
1
[2,7,0]
1
1
2
[0,7,2]
1
0
How do we calculate the expected results?
(and know that they are correct)
The oracle is the mechanism used to generate
expected results.
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
4
Oracle methods used by Hoffman
• A human calculates the expected results.
• A separate program implementing the same algorithm is used to
calculate the expected results.
• A simulation is used to calculate the expected results.
• A hardware simulator is used to calculate the expected results.
• An earlier version of the software is used to calculate the expected
results.
• The same version of the software on a different hardware platform
is used to calculate the expected results.
• A check is made on the consistency of generated values and end
points.
• A sample of values can be checked against independently generated
expected results from existing commercial or open-source software.
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
5
the difficulty of testing scientific software
“The existence of an oracle is often assumed in
software testing.”
“But in many situations, especially for scientific
programs, oracles do not exist or they are too
hard to implement.”
Kanewala and Bieman, 2013
Scientists often write programs to discover answers that are previously unknown.
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
6
Metamorphic Testing
(a way around the oracle problem)
• A metamorphic relation specifies how a
particular change to the input of the program
would change the output.
failure
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
7
examples of metamorphic relations
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
8
Is it possible to replicate an experiment?
• A report should contain enough detail so that
another researcher can repeat the experiment
and obtain similar results with similar
conclusions.
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
9
MS thesis/project idea 1.
• First begin by replicating in part or in full one
of the previous experiments on metamorphic
testing.
– References to several emprical studies are given
by Kanewala and Bieman.
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
10
MS thesis/project idea 1.
• Then tackle one of the problems described by
Kanewala and Bieman.
– automatic detection of metamorphic relations
• some recent work has been done by Kanewala and
Bieman
– identifying the most effective metamorphic
relations
– identifying the most effective combinations of
metamorphic relations
deliberately introduced mistakes
– understanding why some program mutations
could not be killed using metamorphic testing
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
11
Assertion Testing
(a way around the oracle problem)
• Assertions are embedded into source code
and evaluated when a test case is executed.
An assertion is a predicate placed in a program to
indicate that the developer thinks that the predicate is
always true at that place.
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
12
simple Java example
package asserting;
public class First {
public static void main(String[] args) {
int j = 10;
assert j>10;
}
}
What happens when you run this program?
console
Exception in thread "main"
java.lang.AssertionError
at asserting.First.main(First.java:7)
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
13
Assertion basics
(Greg Travis chapter)
• Preconditions should be true at the start of a
method.
• Postconditions should be true at the end of a
method.
• Invariants should always be true.
– A speed should not be faster than light speed...
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
14
MS thesis/project idea 2.
• First begin by replicating in part or in full one
of the previous experiments on assertion
testing.
– References to several empirical studies are given
by Kanewala and Bieman.
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
15
MS thesis/project idea 2.
• Then tackle one of the problems described by
Kanewala and Bieman.
– automatic invariant detection can yield spurious
invariants
• can ways be found to reduce or eliminate these?
– the effectiveness of assertion checking seems all or
nothing depending on the method
• can we understand why?
– many more experiments are needed
• working with specifications to identify assertions
• rewriting program comments as assertions
10/17/2014
Dr Andrew Brooks - MS thesis/project ideas
16