University of Southern California
Center for Systems and Software Engineering
Domain-Based Phase Effort Distribution
Analysis
Annual Research Review
Thomas Tan
March 2012
University of Southern California
Center for Systems and Software Engineering
Table of Contents
•
•
•
•
Introduction
Research Approach
Current Results
Summary and Next Steps
2
University of Southern California
Center for Systems and Software Engineering
Introduction
• This research is aimed at finding alternative phase effort
distribution guideline for the current COCOMO II model.
– Current COCOMO II model use an one-size-fit-all guideline which may lead to
unrealistic schedule.
• Studying many mainstream cost estimation models, it seems
domain information can be a good candidate:
– Available early.
– Easier to tell by all stakeholders.
• So, if we are to use domain information to define an alternative
guideline, then, we must prove the following:
– Projects from different domains have different phase effort
distribution patterns.
– Projects from different domains does not always follow the given
COCOMO II model’s effort distribution guideline.
3
University of Southern California
Center for Systems and Software Engineering
Table of Contents
•
•
•
•
Introduction
Research Approach
Current Results
Summary and Next Steps
4
University of Southern California
Center for Systems and Software Engineering
Research Approach
•
Establish domain breakdown definitions.
– Two possible breakdowns:
– Conventional Application Domains using application types, such as communication,
simulation, sensor control and process, etc.
– Innovative way using productivity rates to group traditional domains, resulting
Productivity Types.
– The following slide will provide mapping between the two breakdowns.
•
•
Select and normalize subject data set.
Analyze effort distribution patterns and prove that differences between
domains exist.
– Calculate effort distribution percentages and find distribution patterns.
– Use ANOVA and T-Test for proof.
– Study personnel ratings and system size to observe additional effects.
– This analysis is performed on data set that is categorized by either domain
breakdowns, namely, Application Domains and Productivity Types.
5
University of Southern California
Center for Systems and Software Engineering
Domain Breakdowns
Productivity Types
Sensor Control and Signal Processing (SCP)
Vehicle Control (VC)
Real Time Embedded (RTE)
Vehicle Payload (VP)
Mission Processing (MP)
Command & Control (C&C)
System Software (SYS)
Telecommunications (TEL)
Process Control (PC)
Scientific Systems (SCI)
Training (TRN)
Test Software (TST)
Software Tools (TUL)
Business Systems (BIS)
Application Domains
Sensor Control and Processing
Executive
Spacecraft Bus
Communication
Controls and Displays
Mission Planning
Weapons Delivery and Control
Spacecraft Payload
Mission Management
Mission Planning
Command & Control
Infrastructure or Middleware
Information Assurance
Maintenance & Diagnostics
Communication
Infrastructure or Middleware
Process Control
Scientific Systems
Simulation and Modeling
Training
Test and Evaluation
Tools and Tool Systems
Business
Internet
Mapping between Application Domains and Productivity Types
6
University of Southern California
Center for Systems and Software Engineering
Select and Normalize Data
• Project data (about 530 records) is extracted from the standard
Software Research Data Report (SRDR).
• Normalization includes:
– Evaluate and eliminate records with missing important fields or weird
patterns.
– Backfill those with missing limited phase effort data.
– Calculate system size, personnel ratings, and other necessary
parameters.
• Data processing results:
– After evaluation and elimination, we have 345 total records.
– Within these 345 total records:
• 135 are “perfect” records (all effort fields are filled with original
values).
• 257 are “missing 2” records which we backfilled 2 of the 5 phase
effort data.
7
University of Southern California
Center for Systems and Software Engineering
Calculate Average Percentages
Calculate percentages
for each records
Calculate average percentages
for each domain
8
University of Southern California
Center for Systems and Software Engineering
ANOVA Test
• Test 1: show if there is difference between application domains in
term of effort distribution percentages.
• Test 1: Use simple ANOVA to test the following:
– H0: effort distributions are same.
– Ha: effort distributions are not all the same between domains.
• Test input is the list of effort percentages grouped by application
domains.
• Test uses 90% confidence level to determine the significance of
the results
9
University of Southern California
Center for Systems and Software Engineering
T-Test
•
•
Test 2: show if there is difference between application domains averages
and the COCOMO II effort distribution averages.
Test 2: Use independent one-sample t-test to test the following:
– H0: domain average is the same as COCOMO average.
– Ha: domain average is not the same as COCOMO average.
– Tests run for every domain on every activity group.
•
Use the following formula to calculate T value in order to determine the
result of the t-test:
– where s is the standard deviation, n is the sample size, and µ0 is the COCOMO
average we used to compare against.
•
Also uses 90% confidence level to determine the significance of the
results.
10
University of Southern California
Center for Systems and Software Engineering
Study on Personnel Ratings and Size
• Goal: to find if changes in personnel ratings and/or size will
generate changes in phase effort distribution patterns.
• Procedure:
– Each record is supplied with personnel ratings and size.
– Note: personnel rating is calculated as the following:
– Simple plot of these values vs. effort percentages for each activity
group for each domain/productivity type.
– Observe trends from the plot, using statistical analysis if necessary.
• Results of the study may indicate that we can use more precise
distribution pattern when size or personnel rating is given.
11
University of Southern California
Center for Systems and Software Engineering
Table of Contents
•
•
•
•
Introduction
Research Approach
Current Results
Summary and Next Steps
12
University of Southern California
Center for Systems and Software Engineering
Data Processing Results
•
Records by application domains:
Research Data Records Count
Application Domains
Missing 2
Missing 1
Business Systems
Command & Control
Communications
Controls & Displays
Executive
Information Assurance
Infrastructure or Middleware
Maintenance & Diagnostics
Mission Management
Mission Planning
Process Control
Scientific Systems
Sensor Control and Processing
Simulation & Modeling
Spacecraft BUS
Spacecraft Payload
Test & Evaluation
Tool & Tool Systems
Training
Weapons Delivery and Control
Total
Perfect Set
6
31
51
10
3
1
8
3
28
14
4
3
27
19
9
2
2
7
1
28
6
25
47
10
3
1
3
1
26
13
4
3
22
18
9
2
1
7
1
19
5
15
32
5
3
1
1
1
19
10
0
3
10
11
5
0
1
2
0
10
257
221
135
13
University of Southern California
Center for Systems and Software Engineering
Results – Application Domains
Average Effort Percentages – Perfect Set
Domain
Requirement Arch&Design Code&Unit Test Integration & QT
Business
Biz
20.98%
22.55%
24.96%
31.51%
Command & Control
CC
21.04%
22.56%
33.73%
22.66%
Communications
Comm
14.95%
30.88%
28.54%
25.62%
Control & Display
CD
14.72%
34.80%
24.39%
26.09%
Mission Management
MM
15.40%
17.78%
28.63%
38.20%
Mission Planning
MP
17.63%
12.45%
44.32%
25.60%
Sensors Control and Processing Sen
7.78%
45.74%
22.29%
24.19%
Simulation
Sim
10.71%
39.11%
30.80%
19.38%
Spacecraft Bus
SpBus
33.04%
20.66%
30.00%
16.30%
Weapons Delivery and Control
Weapons
11.50%
17.39%
29.82%
41.29%
Average Effort Percentages Distribution
50.00%
45.00%
Biz
Percentages
40.00%
CC
35.00%
Comm
30.00%
CD
25.00%
MM
20.00%
MP
Sen
15.00%
Sim
10.00%
SpBus
5.00%
Weapons
0.00%
Requirement
Arch&Design
Code&Unit Test
Integration & QT
University of Southern California
Center for Systems and Software Engineering
Results – Application Domains
Average Effort Percentages – Missing 2
Domain
Requirement Arch&Design Code&Unit Test Integration & QT
Business
Biz
18.51%
23.64%
27.85%
30.00%
Command & Control
CC
19.41%
23.70%
34.59%
22.31%
Communications
Comm
14.97%
27.85%
27.89%
29.28%
Control & Display
CD
14.67%
26.66%
27.72%
30.95%
Mission Management
MM
16.59%
17.60%
25.74%
40.07%
Mission Planning
MP
14.41%
16.42%
43.47%
25.69%
Sensors Control and Processing Sen
7.75%
31.84%
25.32%
35.09%
Simulation
Sim
13.73%
29.09%
30.33%
26.85%
Spacecraft Bus
SpBus
36.38%
17.23%
27.90%
18.50%
Weapons Delivery and Control
Weapons
11.73%
17.79%
28.97%
41.51%
Average Effort Percentages Distribution
50.00%
45.00%
Biz
Percentages
40.00%
CC
35.00%
Comm
30.00%
CD
25.00%
MM
20.00%
MP
Sen
15.00%
Sim
10.00%
SpBus
5.00%
Weapons
0.00%
Requirement
Arch&Design
Code&Unit Test
Integration & QT
University of Southern California
Center for Systems and Software Engineering
Application Domains ANOVA Results
Activity Group
“Missing 2” Data Set
“Perfect” Data Set
F
P-Value
Results
F
P-Value
Results
Plan & Requirements
2.9461
0.0035
Reject
7.2908
0.0000
Reject
Architecture & Design
4.5656
0.0000
Reject
3.5347
0.0004
Reject
Code & Unit Testing
2.1018
0.0350
Reject
3.3470
0.0007
Reject
Integration and
Qualification Testing
3.7787
0.0003
Reject
7.2467
0.0000
Reject
ANOVA Test: Reject means that for the given activity group (development
phase), there is significant differences between the domains analyzed.
16
University of Southern California
Center for Systems and Software Engineering
Application Domains T-Test Results
Activity Group
COCOMO
Averages
“Perfect” Data Set
“Missing 2” Data Set
6.5%
All domains reject except Control
and Display, Sensor Control, and
Simulation domains.
All domains reject except Sensor
Control.
39.3%
All domains reject except Control
and Display, Sensor Control, and
Simulation domains.
All domains reject except Control
and Display.
30.8%
Only Mission Planning rejects.
Communication, Mission
Management, Mission Planning,
and Sensor Control reject, other
six domains do not.
23.4%
Only Mission Management,
Spacecraft Bus, and Weapon
Delivery domain reject.
Communications, Mission
Management, Sensor Control,
Spacecraft Bus, and Weapons
Delivery domains reject; other five
domains do not.
Plan & Requirements
Architecture & Design
Code & Unit Testing
Integration and
Qualification Testing
T-Test: Reject means that the given domain has significantly different average
percentages from the COCOMO II model.
NOTE: Because the sum of all COCOMO II averages are 107%, we have divided them
by 1.07 to make sure all measurements are at the same level for comparison purpose.
17
University of Southern California
Center for Systems and Software Engineering
Data Processing Results
•
Records by productivity types:
Research Data Records Count
Productivity Types
Missing 2
Missing 1
C&C
ISM
MP
PC
PLN
RTE
SCI
SCP
SYS
TEL
TRN
TST
TUL
VC
VP
Total
Perfect Set
8
14
28
7
10
57
22
35
27
4
4
2
4
26
9
3
14
23
6
9
49
21
28
23
4
4
2
3
25
7
0
6
14
3
7
33
16
12
15
4
3
1
2
15
4
257
221
135
18
University of Southern California
Center for Systems and Software Engineering
Results – Productivity Types
Average Effort Percentages – Perfect Set
Productivity Type Requirement Arch&Design Code&Unit Test Integration & QT
ISM
11.56%
27.82%
35.63%
24.99%
MP
20.56%
15.75%
28.89%
34.80%
PLN
16.22%
12.27%
50.78%
20.73%
RTE
15.47%
26.65%
26.71%
31.17%
SCI
7.38%
39.90%
32.05%
20.67%
SCP
10.80%
45.20%
20.34%
23.66%
SYS
17.61%
21.10%
28.75%
32.54%
VC
18.47%
23.60%
31.32%
26.61%
Average Effort Percentages Distribution
60.00%
50.00%
ISM
MP
Percentages
40.00%
PLN
RTE
30.00%
SCI
SCP
20.00%
SYS
VC
10.00%
0.00%
Requirement
Arch&Design
Code&Unit Test
Integration & QT
University of Southern California
Center for Systems and Software Engineering
Results – Productivity Types
Average Effort Percentages – Missing 2
Productivity Type Requirement Arch&Design Code&Unit Test Integration & QT
ISM
12.34%
25.32%
32.81%
29.53%
MP
22.43%
15.06%
26.11%
36.40%
PLN
14.99%
14.15%
49.23%
21.62%
RTE
15.43%
24.02%
28.84%
31.70%
SCI
7.40%
34.98%
30.47%
27.15%
SCP
12.50%
29.07%
24.11%
34.32%
SYS
15.74%
20.87%
32.61%
30.79%
VC
18.02%
21.36%
30.02%
30.59%
Average Effort Percentages Distribution
60.00%
50.00%
ISM
MP
Percentages
40.00%
PLN
RTE
30.00%
SCI
SCP
20.00%
SYS
VC
10.00%
0.00%
Requirement
Arch&Design
Code&Unit Test
Integration & QT
University of Southern California
Center for Systems and Software Engineering
Productivity Types ANOVA Results
Activity Group
“Missing 2” Data Set
“Perfect” Data Set
F
P-Value
Results
F
P-Value
Results
Plan & Requirements
1.9431
0.0694
Reject
4.0141
0.0003
Reject
Architecture & Design
4.9696
0.0000
Reject
4.7831
0.0000
Reject
Code & Unit Testing
3.8851
0.0008
Reject
5.2205
0.0000
Reject
Integration and
Qualification Testing
1.9848
0.0634
Reject
1.8849
0.0733
Reject
ANOVA Test: Reject means that for the given activity group (development
phase), there is significant differences between the productivity types analyzed.
21
University of Southern California
Center for Systems and Software Engineering
Productivity Types T-Test Results
Activity Group
Plan & Requirements
Architecture & Design
Code & Unit Testing
Integration and
Qualification Testing
COCOMO
Averages
“Perfect” Data Set
“Missing 2” Data Set
6.5%
All types reject except ISM, SCI,
and SCP.
All types reject except SCI.
39.3%
All types reject except SCI and
SCP.
All types reject except SCI.
30.8%
Only PLN, RTE, and SCP reject.
Only MP, PLN, and SCP reject.
23.4%
Only MP, RTE, and SYS reject.
All types reject except PLN and
SCI.
T-Test: Reject means that the given domain has significantly different average
percentages from the COCOMO II model.
NOTE: Because the sum of all COCOMO II averages are 107%, we have divided them
by 1.07 to make sure all measurements are at the same level for comparison purpose.
22
University of Southern California
Center for Systems and Software Engineering
Results on Personnel Ratings and Size
• For both Application Domains and Productivity Types:
– Personnel ratings and system size are ineffectively to indicate any
changes of effort distribution patterns in most cases.
– Few cases observed distinguishable trends but later proved
statistically insignificant.
– Grouping sizes is an alternative for analyzing size, however, it is
extremely difficult to apply on all domains or productivity types:
– No appropriate way to divide group sizes to fit all domains or productivity
types.
• Conclusion:
– Personnel ratings can be dropped from the effort distribution pattern
analysis.
– Can spend a little more time playing with size groups, but initial
results favors dropping size as well.
23
University of Southern California
Center for Systems and Software Engineering
Table of Contents
•
•
•
•
Introduction
Research Approach
Current Results
Summary and Next Steps
24
University of Southern California
Center for Systems and Software Engineering
Summary and Next Step
• Established research goal and plan.
• Defined domain breakdowns.
• Normalized the subject data collection: prepared all the needed
data sets for analysis.
• Finished the analyzing effort distribution patterns for both
Application Domains and Productivity Types:
•
– Both breakdowns show proven differences in effort distribution
patterns.
– Both show proven differences against COCOMO II model’s average
percentages.
– None of them show significant trends adding in personnel ratings or
size.
The next major thing is to determine which breakdown is better to provide
an alternative effort distribution guideline for the COCOMO II model.
25
University of Southern California
Center for Systems and Software Engineering
For more information, contact:
Thomas Tan
thomast@usc.edu
626-617-1128
Questions?
26
University of Southern California
Center for Systems and Software Engineering
References (1/2)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Blom, G. Statistical estimates and transformed beta variables. John Wiley and Sons. New York. 1958.
Boehm, B., et al. Software Cost Estimation with COCOMO II. Prentice Hall, NY. 2000.
Boehm, B. Software Engineering Economics. Prentice Hall, New Jersey. 1981.
Borysowich, C. “Observations from a Tech Architect: Enterprise Implementation Issues & Solutions – Effort
Distribution Across the Software Lifecycle”. Enterprise Architecture and EAI Blog.
http://it.toolbox.com/blogs/enterprise-solutions/effort-distribution-across-the-software-lifecycle-6304. October
2005.
Defense Cost and Resource Center. “The DoD Software Resource Data Report – An Update.” Practical
Software Measurement (PSM) Users’ Group Conference Proceedings. July 2005.
Department of Defense Handbook. “Work Breakdown Structure for Defense Material Items: MIL-HDBK-881A.”
July 30, 2005.
Digital Equipment. VAX PWS Software Source Book. Digital Equipment Corp., Maynard, Mass., 1991.
Heijstek, W., Chaudron, M.R.V. “Evaluating RUP Software Development Process Through Visualization of
Effort Distribution”. EUROMICRO Conference Software Engineering and Advanced Application Proceedings.
2008. Page 266.
IBM Corporation. Industry Applications and Abstracts. IBM. White Plains, N.Y., 1988.
Kruchten, P. The Rational Unified Process: An Introduction. Addison-Wesley Longman Publishing Co., Inc.
Boston. 2003.
Kultur, Y., Kocaguneli, E., Bener, A.B. “Domain Specific Phase By Phase Effort Estimation in Software
Projects”. International Symposium on Computer and Information Sciences. September 2009. Page 498.
McConnell, S. Software Estimation Demystifying the Black Art, Microsoft Press, 2006, page 62.
Milicic, D., Wholin, C. “Distribution patterns of Effort Estimation”. EUROMICRO Conference Proceedings.
September 2004. Page 422.
Norden, P.V. “Curve Fitting for a Model of Applied Research and Development Scheduling”. IBM J. Research
and Development. 1958. Vol. 3, No. 2, Page 232-248.
North American Industry Classification System, http://www.census.gov/eos/www/naics/, 2007.
O'Connor, J. Robertson, E. "Student's t-test", MacTutor History of Mathematics archive, University of St
Andrews, http://www-history.mcs.st-andrews.ac.uk/Biographies/Gosset.html.
27
University of Southern California
Center for Systems and Software Engineering
References (2/2)
•
•
•
•
•
•
•
•
•
•
•
Pearson, K. "On the criterion that a given system of deviations from the probable in the case of a correlated
system of variables is such that it can be reasonably supposed to have arisen from random sampling".
Philosophical Magazine, Series 5 50 (302), 1901. Page 157–175.
Putnam, L.H. “A Macro-Estimating Methodology for Software Development”. IEEE COMPCON 76
Proceedings. September 1976. Page 138-143.
Putnam, L. and Myers. W. Measures for Excellence. Yourdon Press Computing Series. 1992.
Reifer Consultants. Software Productivity and Quality Survey Report. El Segundo, Calif., 1990.
SEER-SEM. http://www.galorath.com.
Shapiro, S. S.; Wilk, M. B. “An analysis of variance test for normality (complete samples).” Biometrika 52 (3-4),
1965: page 591–611.
Stephens, M. A. "EDF Statistics for Goodness of Fit and Some Comparisons". Journal of the American
Statistical Association. Vol. 69, No. 347 (Sep., 1974). Page 730-737.
Tan, T. Clark, B. “Technical Report of a New Taxonomy for Software Application Domains and Operating
Environments.” USC CSSE Technical Reports. 2011.
Upton, G., Cook, I. Understanding Statistics. Oxford University Press. Page 55. 1996.
US Air Force. Software Develoopment Cost Estimating Handbook, Software Technology Support Center, Vol
1, Sep 2008.
Yang, Y., et al. “Phase Distribution of Software Development Effort”. Empirical Software Engineering and
Measurement. October 2008. Page 61.
28