Da Yang, Qing Wang, Mingshu Li, Ye Yang, Kai Ye, and Jing Du
Institute of Software, Chinese Academy of Sciences (ISCAS)
COCOMO Forum’08
October 28, 2008

4/14/2020
• Background and Research Questions
• Research Methods
• Survey Results and Discussions
• Threat to Validity
• Future Works
2

• What to improve and how?
– Important to know the current situation of software cost estimation in the industry
• Performance, methods used, causes for the low use of cost estimation models, etc.
• Factors motivated this survey research
– Still no survey on software cost estimation in the Chinese software industry
– Most of the previous surveys were conducted in the 1980s or early 1990s.
• Business environment, development technology, and process are changing
– Lacked research on the factors which influence the adoptions of software cost estimation methods and the factors lead to the low use of model-based software cost estimation
4/14/2020 3

• Current Performance
– RQ#1: What is the accuracy of effort and schedule estimation?
– RQ#2: Does project size affect effort and schedule estimation accuracy?
• Current Practice
– RQ#3: To what extent are different estimation methods used in the industry?
– RQ#4: For what purposes are cost estimations used?
– RQ#7: When do organizations usually make cost estimations?
• Current stakeholders
– RQ#5: How important do people think estimation is, in comparison with other aspects of development?
– RQ#6: How satisfied are people with the current software cost estimation?
• Factors for current situation
– RQ#8: What are the causes of inaccurate estimations?
– RQ#9: What are the barriers and difficulties in the application of software cost estimation models?
4/14/2020 4

• Investigate performance of estimation
– Needs large amount of detailed project data
• The CSBSG data set
– Collected by the Chinese Software Benchmarking
Standards Group to reflect the status and best practices of the Chinese software industry.
– 112 projects have recorded complete information of planned and actual values of project development effort and schedule.
4/14/2020 5

• Project distribution across business areas

Summary of the 112 projects
Size (SLOC)
Mean Median Min Max
123788.1
Effort (Man-Hours) 7883.6
Schedule (Days ）
4/14/2020
175.1
46116.5
4102
150
1480
160
10
2339728
115816
851
6

• Conducting questionnaire survey
– Followed SEI Guideline of Survey Design
• 1) Identify the research objectives
• 2) Identify and characterize the target audience
• 3) Design the sampling plan
– "2007 Chinese Systems and Software Process Improvement Conference“
• 4) Design and prepare the questionnaire
• 5) Pilot test the questionnaire
– Reduce ambiguities, remove difficult questions
• 6) Distribute the questionnaire
– 400 questionnaires. Got 171 respondents from 116 organizations.
• 7) Analyze the results
4/14/2020 7

• Size of the 116 respondent organizations
Organization size (Persons)
< 51
51-100
101-200
201-500
501-1000
> 1000
# of organizations
9
11
26
32
19
19

Information about software process improvement
Software Process Improvement Standards
CMM
CMMI
ISO-9000
CMM&CMMI
CMM&ISO-9000
CMMI&ISO-9000
CMM, CMMI, ISO-9000
No SPI assessment
# of organizations
5
43
15
1
8
25
4
15
116 8

4/14/2020
• Background and Research Questions
• Research Methods
• Survey Results and Discussions
• Threat to Validity
• Future Works
9

• To present the survey results, we needs to
– Organize our various observations
• estimation performance, technology usage, barriers of technology transfer, potential improvement, etc.
– Provide a holistic view of the current situation of software cost estimation
– The UTAUT (Unified Theory of Acceptance and Use of Technology) model is used
4/14/2020
RQ#1: What is the accuracy of effort and schedule estimation?
RQ#2: Does project size affect effort and schedule estimation accuracy?
RQ#8: What are the causes of inaccurate estimations?
10

Sources
Year
[6]
1984
Cost overrun 34% median
61%
[7] [8] [9] [10]
1988 1989 1991 1992
33% mean
[11]
2003
33% mean 18% mean
70% 63% 59%
[3]
2004
21% median
41% mean
76% Effort
Act. > Est.
Effort
Act. < Est.
Schedule overrun
Schedule
Act. > Est.
Schedule
Act. < Est.
4/14/2020
10%
22% mean
65%
4%
80%
14% 15%
23% mean
35%
3%
19%
9% median
25% mean
62%
7% median
17% mean
63%
2%
New
2007
5% median
12% mean
68%
29%
21%
11

30%
50%
40%
20%
22% projects
Overrun > 20% 30%
29% projects
Overrun > 20%
20%
10%
10%
0%
0.00
0.50
1.00
REE of Effort Estimation
0.00
1.00
2.00
REE of Duration Estimation
• Observations on research question #1:
– More than half of the software projects suffered from effort or cost overruns.
– 22% projects overran effort greater than 20%,
– 29% projects overran schedule greater than 20%.
12

REE of
Estimation
Groups Mean Median
Std.
Dev.
Levene's Test for
Equality of t-test for
Equality of
Variances (p) Means (p)
Effort
Schedule
SMALL 0.07
0.04
0.20
LARGE 0.16
0.08
0.33
SMALL 0.06
0.00
0.27
LARGE 0.28
0.15
0.49
0.004
( ** p < 0.01)
0.033
( * p < 0.05)
0.057
( + p < 0.01)
0.003
( ** p < 0.01)
• Observations on research question # 2: Large software projects vs. small ones
– had lower effort and schedule estimation accuracy
– were more prone to effort and schedule overruns
– had higher variance of estimation accuracy
4/14/2020 13

Causes of inaccurate estimations
Rate on the extent of responsibility 1 (min) – 5 (max)
1 Requirements are volatile
2 Requirements are unclear
3
Pressure from senior manager and client to set or change the estimation results
4 Not enough resource for estimation
5 Not efficient historical projects
6
Lack of appropriate software cost estimation methods and process
7 Lack of stakeholder collaboration
8 Lack of risk assessment and management
9 Lack of cost control according to plan
10 Lack of estimation tools
11 Hard to assess the ability of developers
12 Lack of product risk assessment
13 Estimation lack involvement of developers
Other causes proposed by respondents
14) The project bidding requirements predefined the project cost
15) The survival pressure and business pattern of company
Rating
(1-5 scale)
3.82
3.70
3.22
3.17
3.13
3.10
3.07
3.06
3.01
2.97
2.90
2.87
2.72
14

4/14/2020
RQ#3: To what extent are different estimation methods used in the industry?
15

Sources
Year
[14]
1987
[8]
1989
Percentage used each methods (more than one method possible)
Expert consultation 26% 86% 100% 70%
Intuition and experience 85% 62%
[4]
1995
[3]
2004
New
2007
Analogy
Software cost models
Price-to-win
13%
Capacity related
Top-down
1) existing literature suggests that the capacity-related and price-to-win
12%
61%
14%
8%
21%
9%
65%
26%
16%
11%
51%
28%
70%
15%
53% like capacity or price' is one of the top three rated causes for inaccurate cost estimations.
16

4/14/2020
RQ#9: What are the barriers and difficulties in the application of software cost estimation models?
17

Barriers or difficulties of applying cost models data, configure parameters, calibrate models, etc
Persons (n)
89
%
58%
69 45% software cost estimation benefit
4) Lack corresponding tools which are easy to use
5) Haven't found appropriate software cost estimation models or tools
6) Software cost estimation models are hard to learn and use
Total
Other causes proposed by respondents
7) Client didn't require using software cost estimation models
8) Estimation models were not accurate and effective
61
56
52
34
153
40%
37%
34%
22%
100%
10) Senior managers only looked at the result and didn't care for the estimation process
11) Schedule pressure was very high, and the estimation could not help
12) Estimation models haven't well defined what kind of data need to be collected
4/14/2020 18

4/14/2020
RQ#5: How important do people think estimation is, in comparison with other aspects of development?
RQ#6: How satisfied are people with the current software cost estimation?
19


The importance of software cost estimation
Importance
Most unimportant
Very unimportant
Unimportant
Neutral
Important
Very important
Most important
Total
Persons (n) Percentage
1 1%
2 1%
3
15
63
77
5
166
2%
9%
38%
46%
3%
100%

The satisfaction level of respondents
Satisfaction Level
Very unsatisfactory
Unsatisfactory
Neutral
Satisfactory
Very Satisfactory
Persons (n) Percentage
2 1%
34
100
21%
62%
26
0
162
16%
0%
100%
20

• Based on the UTAUT model, the findings suggest:
– Only providing accurate cost estimation is not enough to guaranty the acceptance and usage of a cost estimation technology
– To introduce new cost estimation technology and change the current situation, we should also pay attention to the
"performance expectancy", "effort expectancy", "social influence", and "facilitating conditions“ of the technology.
4/14/2020 21

4/14/2020
RQ#4: For what purposes are cost estimations used?
RQ#7: When do organizations usually make cost estimations?
22

Where are estimates used?
Project proposal evaluation
Contract negotiation
Making budget
Project-level planning and control, e.g. effort or schedule distribution among development phases
Short-term planning and control, e.g. weekly or monthly team work plan
Software process improvement, e.g. assess new process, improve productivity
Total
Organizations
(n)
52
53
85
97
58
43
115
4/14/2020
Percentage
45%
46%
74%
84%
50%
37%
100%
23

Software Development Phases
Initial project proposal stage
Feasibility study
Requirement
Design
Implementation
Integration and testing
Transition
Total
Organizations (n) Percentage
64
76
84
41
31
20
13
113
57%
67%
74%
36%
27%
18%
12%
100%

OB8 and OB9 indicate two needs of software cost estimation
 better support of various estimation goals

4/14/2020 better support of early lifecycle cost estimation (when uncertainty is large)
24

• Projects Data
– may be recorded by organizations with higher maturity levels
• Questionnaire samples
– may be biased to organizations above average size and process maturity level of the industry
• Generalizability of the results
– Cultural issues that reduce the generalizability of the results
4/14/2020 25

• Conduct surveys and experiments in more detail
– See the variance and changes in the performance of software cost estimations
– Identify related factors and potential improvements
• Design measures
– To evaluate cost estimation methods based on 1)performance expectancy,
2)effort expectancy, 3)social influence, and 4)facilitating conditions.
• Developing new estimation processes, methods, and tools to address those considerations
4/14/2020 26

4/14/2020
Contact info: ye@itechs.iscas.ac.cn
27