University of Southern California
Center for Systems and Software Engineering
A Constrained Regression
Technique for COCOMO
Calibration
Presented by
Vu Nguyen
On behalf of
Vu Nguyen, Bert Steece, Barry Boehm
{nguyenvu, berts, boehm}@usc.edu
© USC-CSSE
1
University of Southern California
Center for Systems and Software Engineering
Outline
• Introduction
– Multiple Linear Regression
– OLS, Stepwise, Lasso, Ridge
• Constrained Linear Regression
• Validation and Comparison
– COCOMO overview
– Cross validation
• Conclusions
• Limitations
• Future Work
© USC-CSSE
2
University of Southern California
Center for Systems and Software Engineering
Introduction
• Building software estimation models is a
search problem to find the best possible
parameters that
– generate high prediction accuracy
– satisfy predefined constraints
© USC-CSSE
3
University of Southern California
Center for Systems and Software Engineering
Multiple Linear Regression
• Multiple linear regression is presented as
yi = 0 + 1xi1 +…+ kxik + i , i = 1,2,…, n
• Where,
– 0, 1,…, k are the coefficients
– n is the number of observations
– k is the number of variables
– xij is the value of the variable jth for the ith observation
– yi is the response of the ith observation
© USC-CSSE
4
University of Southern California
Center for Systems and Software Engineering
Ordinary Least Squares
• OLS is the most common method to
estimate coefficients 0, 1,…, k
• OLS estimates coefficients by minimizing
the sum of squared errors (SSE)
– Minimize
n
2
ˆ
(
y

y
)
 i i
i 1
–
ŷi is the estimate of ith observation
© USC-CSSE
5
University of Southern California
Center for Systems and Software Engineering
Some Limitations of OLS
• Highly sensitive to outliers
• Low bias but high variance
(e.g., caused by
collinearity or overfitting)
• Unable to constrain the
estimates of coefficients
Expected values
OLS estimates
– Estimated coefficients may
be counter-intuitive
– Example, OLS coefficient
estimate for RUSE is
negative, e.g., increase
RUSE rating results in a
decrease in effort
Develop for Reuse (RUSE)
© USC-CSSE
6
University of Southern California
Center for Systems and Software Engineering
Some Other Approaches
• Stepwise (forward selection)
• Ridge
– Minimize SSE and impose a penalty
on sum of squared coefficients
N
Minimize
(y
i 1
 yˆ i ) subject to
2
i
p

s
2
j
High bias
Low variance
Low bias
High variance
Prediction error
– Start with no variable and gradually
add variables until “optimal” solution
is achieved
Testing sample
Training sample
j
• Lasso
Number of variables
– Minimize SSE and impose a penalty
on sum of absolute coefficients
N
Minimize
(y
i 1
 yˆ i ) subject to
2
i
p

j
s
j
© USC-CSSE
7
University of Southern California
Center for Systems and Software Engineering
Outline
• Introduction
– Multiple Linear Regression
– OLS, Stepwise, Lasso, Ridge
• Constrained Linear Regression
• Validation
– COCOMO overview
– Cross validation
• Conclusions
• Limitations
• Future Work
© USC-CSSE
8
University of Southern California
Center for Systems and Software Engineering
Constrained Regression
• Principles
– Use optimization paradigm: optimizing
objective function with constraint
Minimize f(y, X) subject to cf(z)
– Impose constraints on coefficients and relative
error
– Expect to reduce variance by reducing the
number of variables (variance and bias
tradeoff)
© USC-CSSE
9
University of Southern California
Center for Systems and Software Engineering
Constrained Regression (cont)
• General form
Minimize f ( y , X ) subject to
•
MRE i  c and ˆ j  0
Constrained Minimum Sum of Squared Errors (CMSE)
N
f ( y, X )   ( y i  yˆ i ) 2
i 1
•
Constrained Minimum Sum of Absolute Errors (CMAE)
N
f ( y, X )   | y i  yˆ i |
i 1
•
Constrained Minimum Sum of Relative Errors (CMRE)
N
f ( y, X )  
i 1
yi  yˆ i
yi
© USC-CSSE
10
University of Southern California
Center for Systems and Software Engineering
Solve the Equations
• Solving the equations is an optimization problem
– CMSE: quadratic programming
– CMRE and CMAE: transformed to the form of linear
programming
– We used lpsolve and quadprog packages in R
0.7
PRED(0.3)
• Determine parameter c
using cross-validation
0.8
CMSE
0.6
CMAE
0.5
CMRE
0.4
0.4
© USC-CSSE
0.6
0.8
1
values c
1.2
1.4
11
University of Southern California
Center for Systems and Software Engineering
Outline
• Introduction
– Multiple Linear Regression
– OLS, Stepwise, Lasso, Ridge
• Constrained Linear Regression
• Validation and comparison
– COCOMO overview
– Cross validation
• Conclusions
• Limitations
• Future Work
© USC-CSSE
12
University of Southern California
Center for Systems and Software Engineering
Validation and Comparison
• Two COCOMO datasets
– COCOMO 2000: 161 projects
– COCOMO 81: 63 projects
• Comparing with popular model building approaches
– OLS
– Stepwise
– Lasso
– Ridge
• Cross-validation
– 10-fold cross validation
© USC-CSSE
13
University of Southern California
Center for Systems and Software Engineering
COCOMO
• Cost Constructive Model (COCOMO) first
published in 1981
– Calibrated using 63 projects (COCOMO 81 dataset)
– Uses SLOC as a size measure and 15 cost drivers
• COCOMO II published in 2000
– Reflects changes in technologies and practices
– Uses 22 cost drivers plus size measure
– Introduces 5 scale factors
– Calibrated using 161 data points (COCOMO II
dataset)
© USC-CSSE
14
University of Southern California
Center for Systems and Software Engineering
COCOMO Overview (cont)
• COCOMO Effort Equation, non-linear
p
PM  A * Size B *  EM i
i
• Linearize the model using log-transformation
– COCOMO 81
log(PM) = 0 + 1 log(Size) + 2 log(EM1) + … + 16 log(EM15)
– COCOMO II
log(PM) = 0 + 1 log(Size) + i SFi log(Size) + j log(EMj)
 Estimate coefficients using a linear regression method
© USC-CSSE
15
University of Southern California
Center for Systems and Software Engineering
Model Accuracy Measures
• Magnitude of relative errors (MRE)
yi  yˆi
MREi 
yi
• Mean of MRE (MMRE)
1
MMRE 
N
N
 MRE
i 1
i
• Prediction Level: PRED(l) = k/N
– Where, k is the number of estimates with
MRE ≤ l
© USC-CSSE
16
University of Southern California
Center for Systems and Software Engineering
Cross Validation
• 10-fold cross validation was used
• Step 1. Randomly split the dataset into K=10
subsets
• Step 2. For each i = 1 ... 10
– Remove the subset i th and build the model
– i th subset is used as testing set to calculate MMREi
and PRED(l)I
• Step 3. Repeat 1 and 2 for r=15 times
© USC-CSSE
17
University of Southern California
Center for Systems and Software Engineering
Non-cross validation results
COCOMO II
dataset
CMSE
Max. MRE (c)
MMRE
CMAE
PRED*
MMRE
CMRE
PRED*
MMRE
PRED*
+ infinity
0.23
0.78
0.22
0.81
0.21
0.78
OLS:
1.2
0.23
0.78
0.21
0.81
0.21
0.80
Max MRE=1.23
PRED=0.78
1.0
0.23
0.76
0.21
0.77
0.21
0.77
0.8
0.23
0.72
0.22
0.73
0.21
0.75
0.6
0.24
0.68
0.25
0.65
0.23
0.70
(N = 161)
COCOMO 81
dataset
(N = 63)
* PRED(0.3)
CMSE
Max. MRE (c)
MMRE
CMAE
PRED*
MMRE
CMRE
PRED*
MMRE
PRED*
+ infinity
0.30
0.62
0.29
0.65
0.30
0.68
1.2
0.30
0.65
0.28
0.65
0.28
0.62
1.0
0.30
0.62
0.29
0.62
0.29
0.59
0.8
0.30
0.58
0.29
0.60
0.30
0.59
© USC-CSSE
18
University of Southern California
Center for Systems and Software Engineering
Cross-validation Results
COCOMO II dataset
COCOMO 81 dataset
© USC-CSSE
19
University of Southern California
Center for Systems and Software Engineering
Statistical Significance
• Results of statistical significance tests on MMRE
(0.05 confidence level used)
– Mann-Whitney U hypothesis test
COCOMOII.2000
COCOMO 81
There are statistical differences among
Lasso, Ridge, OLS, Stepwise
p > 0.11
p > 0.15
CMSE outperforms Ridge, OLS
p > 0.10
p > 0.10
CMSE outperforms Lasso, Stepwise
p < 0.02
p > 0. 05
CMAE outperforms Lasso, Ridge, OLS
Stepwise
p < 10-3
p < 0. 02
CMRE outperforms Lasso, Ridge, OLS
Stepwise
p < 10-4
p < 10-4
© USC-CSSE
20
University of Southern California
Center for Systems and Software Engineering
Comparing With Published Results
• Some best published results in for COCOMO
datasets
– Bayesian analysis (Boehm et al., 2000)
– Chen et al., 2006
• Best cross-validated mean PRED(.30):
Dataset
Bayesian
Chen et al
CMRE
COCOMO II
70
NA
75
COCOMO 81
NA
51
56
© USC-CSSE
21
University of Southern California
Center for Systems and Software Engineering
Productivity Range
FLEX
RUSE
TEA M
FLEX
RUSE
TEAM
COCOMO II.2000
P REC
A P EX
A = 2.94
P LEX
A = 2.27
P LEX
SCED
DA TA
B = 0.91
LTEX
SCED
PM AT
RELY
STOR
P CON
TOOL
P REC
P VOL
P VOL
A P EX
RESL
DOCU
TIM E
SITE
STOR
RELY
TOOL
P CON
SITE
TIM E
P CA P
P CA P
DOCU
A CA P
A CA P
CP LX
CP LX
1.20
1.40
1.60
1.80
2.00
B = 0.98
PM AT
DA TA
1.00
CMRE
LTEX
RESL
2.20
2.40
2.60
© USC-CSSE
1.00
1.50
2.00
2.50
3.00
22
University of Southern California
Center for Systems and Software Engineering
Outline
• Introduction
– Multiple Linear Regression
– OLS, Stepwise, Lasso, Ridge
• Constrained Linear Regression
• Validation and comparison
– COCOMO overview
– Cross validation
• Conclusions
• Limitations
• Future Work
© USC-CSSE
23
University of Southern California
Center for Systems and Software Engineering
Conclusions
• Technique imposes constraints on the estimates of
coefficients and the magnitude of errors term
– Directly resolving the unexpected estimates of coefficients
determined by data
– Estimation accuracies are favorable
• CMRE and CMAE outperform OLS, Stepwise, Ridge,
Lasso, and CMSE
– MRE and MAE are favorable objective functions
• Technique can be applied in not only COCOMO-like
models but also other linear models
 An alternative for researchers and practitioners to build
models
© USC-CSSE
24
University of Southern California
Center for Systems and Software Engineering
Limitations
• As the technique deals with the optimization,
sub-optimal solution is returned instead of
global-optimal one
• Multiple solutions exist for the estimates of
coefficients
• There are only two datasets investigated, the
technique might not work well on other datasets
© USC-CSSE
25
University of Southern California
Center for Systems and Software Engineering
Future Work
• Validate the technique using other datasets
(e.g., NASA datasets)
• Compare results from the technique with others
such as neutral networks, generic programming
• Apply and compare with other objective
functions
– MdMRE (median of MRE)
– Z measure (z=estimate/actual)
© USC-CSSE
26
University of Southern California
Center for Systems and Software Engineering
References
•
Boehm et al., 2000. B. Boehm, E. Horowitz, R. Madachy, D. Reifer, B. K. Clark, B.
Steece, A. W. Brown, S. Chulani, and C. Abts, Software Cost Estimation with
COCOMO II. Prentice Hall, 2000.
•
Chen et al., 2000, Z. Chen, T. Menzies, D. Port, and B. Boehm. Finding the right data
for software cost modeling. IEEE Software, Nov 2005.
© USC-CSSE
27
University of Southern California
Center for Systems and Software Engineering
Thank You
Q&A
© USC-CSSE
28