Coevolutionary Automated Software Correction
Josh Wilkerson
PhD Candidate in Computer Science
Missouri S&T
High Level View of CASC
Page  2
CASC Evolutionary Model
Page  3
CASC Evolutionary Model
Page  4
CASC Evolutionary Model
Page  5
CASC Evolutionary Model
Page  6
Reproduction Phase: Programs
 Randomly select a genetic operation to perform
– Probability of operation selection is configurable and/or adaptive
 Select individual(s) to use
– First select sub-set of individuals (i.e., tournament)
– Then perform fitness proportional selection in sub-set (i.e., roulette)
– Reselection allowed
 Perform operation, generate new program(s)
 Add new individuals to population
 Repeat until specified number of individuals has been created
Page  7
Reproduction Phase: Programs
 Genetic Operations
– Reset
– Copy
– Crossover
• Two individuals are randomly selected based off fitness
• Randomly select and exchange compatible sub-trees
• Generates two new programs
– Mutation
• Off-by-one mutation bias
• Randomly select individual based off fitness
• Randomly select and change mutable node
• Generate a new sub-tree (if necessary)
– Architecture Altering Operations
• Delete a line, add assignment, add flow control
Page  8
Reproduction Phase: Test Cases
 Reproduction employs uniform crossover
 Same selection method as programs
 Each offspring has a chance to mutate
 Genes to mutate are selected random
 Mutated gene is randomly adjusted
– The amount adjusted is selected from a Gaussian distribution
Page  9
CASC Evolutionary Model
Page  10
CASC Evolutionary Model
Page  11
Evaluation Phase
 All programs run against all test cases
– Full population exposure vs. population sampling
– Hash table used to avoid repeat evaluations
 Executions scored based on input and output of the program
– Black box style
– Run-time exceptions and time-outs monitored
 Fitness for program is average of all execution scores
– Test case scores are directly related to this value
Page  12
CASC Evolutionary Model
Page  13
CASC Evolutionary Model
Page  14
CASC Evolutionary Model
Page  15
CASC Implementation Details
Page  16
 Adaptive parameter control
– EAs typically have many control parameters
– Difficult to find optimal settings for these parameters
– In CASC genetic operator probabilities are adaptive parameters
– Rewarded/punished based on performance
•
If one operator is generating improved individuals more than the others
make it more likely to be used
– Allows the system to adapt to the different phases in the search
CASC Implementation Details
Page  17
 Parallel Computation
– Computational complexity is generally a problem for EAs
– CASC typically writes and compiles thousands of programs on a
given run
• Typically executes millions of evaluations (literally)
– To reduce run times executions are done in parallel (NIC cluster)
• All other evolutionary phases are done in serial
– Main node: responsible for generating and writing programs
– Worker nodes: responsible for compiling and executing programs
– Dramatically speeds up execution
CASC Criticisms
 Scalability
– The problem space is infinite for even simple programs
– Must correct software in reasonable time, regardless of program size
 Fitness Function Design
– Each new problem for CASC requires a new fitness function
– Infinite possible fitness functions
– Limited number of high quality fitness functions
– Design of high quality fitness functions is extremely difficult
Page  18
Scalability: ARCD
 Automated Relevant Code Discovery (ARCD) System
– Preprocessor for CASC
– Uses bug localization techniques to remove irrelevant lines of code from
consideration
– Ensemble of analysis methods
•
Each method generates a set of suspect lines of code
•
Results are combined together and a relevant code set is generated
– Voting system
– Confidence levels
•
Employ state of the art bug localization techniques
•
Exploit the availability of fitness function
– Prototype is under development
– Three techniques currently implemented
•
Positive/negative trace comparison
•
Line suspicion based on fitness
•
Fitness run-time plot
Page  19
ARCD: Pos./Neg. Trace Comparison
Page  20
Positive Trace
Negative Trace
1
2
3
4
5
4
5
4
5
10
11
12
1
1
0
0
0
0
0
0
0
0
0
0
0
2
0
2
0
0
0
0
0
0
0
0
0
0
3
0
0
3
0
0
0
0
0
0
0
0
0
4
0
0
0
4
0
1
0
1
0
0
0
0
5
0
0
0
0
5
0
2
0
2
0
0
0
6 7 4 5 6 7 8 9 4 5 8 9 10 11 12
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
1
0
1
0
0
0
0
0
0
0
0
2
0
2
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
1
0
1
0
0
0
0
0
0
0
0
2
0
2
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
0
0
0
3
ARCD: Fitness Plots
Page  21
Fitness Plots
1.2
1
Fitness
0.8
0.6
0.4
0.2
0
0
10
20
30
40
Lines Executed
50
60
70
Incorrect Program
Correct Program
Scalability: CC-CoEA
 Cooperative-Competitve Coevolution (CC-CoEA)
– Multiple program populations
– Cooperative coevolution of program components
– Each sub-population is focused on a specific portion of the program
– Components are selected from each population and a program is
assembled
– Fitness indicates how well each component operated
– Divide the problem space into smaller, more manageable pieces
– Allow CASC to “freeze” sub-populations that are suspected to have
converged
Page  22
Scalability: CC-CoEA
Page  23
Fitness Function Design
Page  24
 Current approach: guide for fitness function generation
– Formalize the thought process for fitness function design
– Incorporate quality measures to assure quality fitness functions
– Incorporate advanced fitness function techniques, mapped to problem
characteristics (indicate when techniques will be useful)
– Extend to be useful for black box search algorithms that use fitness
functions
– Implement as semi-automated tool for fitness function design
 Alternative approach
– Exploit formal specifications
• Information about expected program operation
• Possibly generate new, correct code from scratch
– No evidence this approach will be superior
• Many open problems
• One-to-many relationships
Page  25
Questions?