Software Engineering
Experimentation
An Experimental Framework
Jeff Offutt
http://www.cs.gmu.edu/~offutt/
Experimental Framework
•
Victor Basili, Richard Selby and David Hutchens,
“Experimentation in Software Engineering”, IEEE
Transactions on Software Engineering, SE-12(7):733–743,
July 1986
•
•
A little old but still relevant and helpful
Three objectives
1. Framework for performing experiments
2. Classify and discuss experiments
3. Problem areas and lessons learned
© Jeff Offutt, 2005-2012
2
Framework
Four categories
1.
2.
3.
4.
Definition
Planning
Operation
Interpretation
– Problem definition and hypothesis
– Design
– Conduct test
– Hypothesis test, disseminate
© Jeff Offutt, 2005-2012
3
Framework – 1. Definition
Motivation
(why)
Object
(what)
Purpose
(what)
Perspective
(who)
Domain
(which)
Understand
Assess
Manage
Engineer
Learn
Improve
Validate
Assure
Product
Process
Model
Metric
Theory
Characterize
Evaluate
Predict
Motivate
Developer
Modifier
Maintainer
Project
manager
Corp. manager
Customer
User
Researcher
Programmer
Program /
project
© Jeff Offutt, 2005-2012
Scope
(which)
Single project
Multi-project
Replicated project
Blocked subjectproject
4
Framework – 2. Planning
Design
(samples, control)
Criteria
(what to measure)
Measurement
(how to measure)
Experimental designs
Incomplete block
Completely
random
Randomized block
Fractional factorial
Multivariate analysis
Correlation
Factor analysis
Regression
Statistical models
Non-parametric
Sampling
Direct reflections of cost / quality
Cost
Errors
Changes
Reliability
Correctness
Indirect reflections of cost / quality
Data coupling
Information visibility
Programmer comprehension
Execution coverage
Size
Complexity
Metric definition
Goal-question-metric
Factor-criteria-metric
Metric validation
Data collection
Automatability
Form design and test
Objective vs. subjective
Level of measurement
Nominal / classificatory
Ordinal / ranking
Interval
Ratio
© Jeff Offutt, 2005-2012
5
Framework – 3. Operation
Preparation
Pilot study
Execution
Data collection
Data validation
© Jeff Offutt, 2005-2012
Analysis
Quantitative vs. qualitative
Preliminary data analysis
Plots and histograms
Model assumptions
Primary data analysis
Model application
6
Framework – 4. Interpretation
Interpretation
context
Statistical framework
Study purpose
Field of research
Extrapolation
Impact
Sample representativeness Visibility
Replication
Application
© Jeff Offutt, 2005-2012
7
Framework Example Definition
Definition Element
Example
Motivation
To improve unit testing
Object
characterize and evaluate
Purpose
the process of functional & structural testing
Perspective
from the perspective of the developer
Domain: programmer
Domain: program
as they are applied by experienced
programmers
to unit-size software
Scope
in a blocked subject-project study
© Jeff Offutt, 2005-2012
8