On the application of GP for
software engineering predictive
modeling: A systematic review
Expert systems with Applications, Vol. 38 no. 9, 2011
Wasif Afzal, Richard Torkar
Blekinge Institute of Technology,
Karlskrona, Sweden.
{waf,rto}@bth.se
Agenda
•
•
•
•
•
•
•
Research question
Symbolic regression
Prediction and estimation in sw engineering
GP for prediction and estimation in sw engineering
Application of GP for sw quality classification
Application of GP for sw cost/effort/size estimation
Application of GP for sw fault prediction and sw reliability
growth modeling
• Future work
• Conclusions
• Recommendations
Our research question
• Is there evidence that:
symbolic regression using GP is an effective
method for:
prediciton and estimation, in comparison with:
regression, machine learning and other models
(including expert opinion and different
improvements over the standard GP algorithm)?
It is about symbolic regression!
• Symbolic regression – One of the many application
areas of GP
– Finds a function, with the outputs having desired
outcomes.
– Makes no assumptions about:
• Structure of the function
• Data distribution
• Relationship between independent and dependent
variables
• Helps in identifying the significant variables in
subsequent modeling attempts
Prediction and estimation in sw
engineering
• Software quality
– Software quality classification
– Software fault prediction
– Software reliability growth modeling
• Software size
• Software development cost/effort
• Maintenance task effort
• Software release timing
GP for prediction and estimation in sw
engineering
• 23 identified primary studies
– Software quality classification (8)
– Software cost/effort/size estimation (7)
– Software fault prediction and software
reliability growth modeling (8)
GP for prediction and estimation in sw
engineering cntd…
Application of GP for sw quality
classification (8 studies)
• Variations of the dependent variable:
– Fault proneness
– Quality ranking of program modules (high risk to low
risk)
• Variations in sampling of training and testing sets:
– Simple hold-out and 10-fold CV.
Application of GP for sw quality
classification cntd…
• Variations in fitness function
– Single objective
• Minimization of root mean square
• Minimization of average cost of misclassification
– Multi-objective
• Minimization of average cost of misclassification +
minimization of tree size
• Maximization of the best percentage of the actual faults
averaged over the percentiles level of interest + controlling
the tree size.
• Balancing the over sampling and under sampling in each
class for a decision tree.
Application of GP for sw quality
classification cntd…
• Variations in comparison groups:
– Neural networks
– k-nearnest neighbour
– Regression (linear, logistic)
– Humans
Application of GP for sw quality
classification cntd…
• Results:
– Majority of the studies (6 out of 8) reported
results in favor of using GP for the classification
task.
• Limitations:
– Increase the comparisons with a more
representative set of techniques.
– Increase the use of publically available data sets
for easier replications.
Application of GP for sw quality
classification cntd…
• Encouraging aspects:
– The datasets used represent real-world
projects.
– Problem dependent objectives represented in
fitness functions perform better than standard
GP.
Application of GP for sw cost/effort/size
(CES) estimation (7 studies)
• Variations of the dependent variable
– Software effort
– Software cost
– Software size
• Variations in fitness function
– Single objective
• Minimization of mean squared error or
MMRE
Application of GP for sw cost/effort/size
(CES) estimation cntd…
• Variations in comparison groups
– ANN, nearest neighbour and different forms
of regression.
• Variations in sampling of training and testing
sets
– Simple hold-out.
Application of GP for sw cost/effort/size
(CES) estimation cntd…
• Results
– No strong evidence of GP performing consistently on
all evaluation measures used.
• Limitations
–
–
–
–
Evaluation measures used are not standardized.
Different hold-out samplings for train and test sets.
Lack of statistical hypothesis testing.
Lack of comparison groups.
Application of GP for sw fault prediciton and
sw reliability growth modeling (8 studies)
• Variations of the dependent variable
– SW fault prediction
– SW reliability growth modeling
• Variations in fitness function
– Single objective:
• Minimization of standard error
Application of GP for sw fault prediciton and
sw reliability growth modeling cntd …
• Variations in comparison groups
– Standard GP, Naive Bayes, traditional
software reliability growth models.
• Variations in sampling of training ad
testing sets
– Hold-out and 10-fold CV
Application of GP for sw fault prediciton and
sw reliability growth modeling cntd …
• Results:
– 7 out of 8 studies favor the use of GP.
• Limitations:
– Poor representation of comparison groups
– Absence of a baseline to compare to.
Promising future work to undertake
• Multi-objective fitness evaluation (e.g.
Minimization of standard error and maximization
of correlation coefficient)
• Simplification of GP solutions to help
interpretation of relationships between variables.
• Evaluation of techniques to minimize overfitting
of GP solutions.
Conclusions
• A total of 23 studies apply GP for predictive studies in sw
engineering:
– sw quality classification (8)
– sw cost/effort/size estimation (7)
– sw fault prediciton and sw reliability growth modeling (8)
• There is evidence in support of using GP for:
– sw quality classifiaction
– sw fault prediction and SW reliability growth modeling
• but not for:
– sw cost/effort/size estimation.
Recommendations
• Use public data sets wherever possible.
• Apply commonly used sampling strategies.
• Use techniques to avoid overfitting in GP
solutions.
• Report the settings of GP parameters.
• Compare the performances against a commonly
used baseline.
• Use statistical experimental designs.