Phase Distribution of Software Development Effort
Ye Yang1, Mei He1,2, Mingshu Li1, Q ing Wang1, Barry Boehm3
1Institute
2Graduate
of Software, Chinese Academy of Sciences, China.
University of Chinese Academy of Sciences, China.
3University
of Southern California, USA.
{ ye, hemei, mingshu, wq}@itechs.iscas.ac.cn, boehm@csse.usc.edu
ABSTRACT
Effort distribution by phase or activity is an important but often
overlooked aspect compared to other steps in the cost estimation
process. Poor effort allocation is among the major root causes of
rework due to insufficiently resourced early activities. This paper
provides results of an empirical study on phase effort distribution
data of 75 industry projects, from the China Software
Benchmarking Standard Group (CSBSG) database. The phase
effort distribution patterns and variation sources are presented,
and analysis results show some consistency in effects of software
size and team size on code and test phase distribution variations,
and some considerable deviations in requirements, design, and
transition phases, compared with recommendations in the
COCOMO model. Finally, this paper discusses the major findings
and threats to validity and presents general guidelines in directing
effort allocation. Empirical findings from this study are beneficial
for stimulating discussions and debates to improve cost estimation
and benchmarking practices.
Categories and Subject Descriptors
D.2.8 [Metrics]: process metrics and product metrics.
D.2.9 [Software Engineering]: Management–Cost estimation
General Terms
Management, Measurement.
Keywords
Cost Estimation, Effort Distribution, Phase Distribution,
Development Type, Estimation Accuracy, Effort Allocation.
1. INTRODUCTION
Cost estimation methods/techniques have been intensively studied
and evolved over the past few decades. Researchers employ
different methods, such as parametric modeling, knowledge-based
modeling, fuzzy logic, dynamic modeling, neural networks, or
case-based reasoning [1, 2], to increase general estimation
accuracy and improve estimation capability with respect to
emergent development paradigms, such as COTS or open sourcePermission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
ESEM’08, October 9-10, 2008, Kaiserslautern, Germany.
Copyright 2008 ACM 978-1-59593-971-5/08/10...$5.00.
based development [3, 4]. However, despite the increasing trend
of amount and varieties of estimation methods, it remains a great
challenge for software projects to be “successfully completed”
according to the 29% success rate of year 2004 reported by
Standish group [5].
In examining the assumptions and construction of typical cost
estimation methods [6-11, 12], two limitations are often the major
impediment for software project to meet the success criteria, i.e.
on time, within budget, and with originally-specified
functionalities. On one hand, limited progress has been made to
fully understand characteristics of emergent development
paradigm, as well as its implications to cost estimation. This
consequently adds lots of complication factors in making processoriented decisions in early project phases. On the other hand, even
with the classic waterfall development process, there is very
limited research effort towards analyzing and understanding root
causes of variations in effort distribution patterns.
Effort distribution by phase or activity is an important but often
overlooked aspect compared to other steps in the cost estimation
process. Poor effort allocation is among the major root causes of
rework due to insufficiently resourced early activities.
Distribution of effort in software engineering process has been the
basis for facilitating more reasonable software project planning,
and is provided as one of the major functionalities in most
estimating or planning methods. A general form of such basis is a
breakdown structure of the total estimate over different life cycle
phases or activities, with corresponding distribution percentage
numbers for each phase/activity.
As the rapidly increasing number of estimation methods and tools
becoming available, practitioners are often left with even greater
difficulty and confusion in making cost estimation and
distribution given large variation in the lifecycle phases/activities
covered in each method. Many software practitioners are facing
insufficient planning support when handling a particular project
conditions, and solely relying on Rule of Thumb or expert
judgment to handle project phase effort planning as this is only
feasible approach available to them [13]. Compared with a total
estimate, it is more important to develop an accurate phase-based
estimation for software development process in order to facilitate
appropriate and strategic resource planning.
The paper presents empirical analysis results on phased effort data
of 75 industry projects, from the China Software Benchmarking
Standard Group (CSBSG) [26] database. The analysis investigates
the general effort distribution profiles of different development
type, software size, business area, and team size. The study also
concludes guidelines on how to take the most significantly
influential factors into consideration when making effort
distribution decisions. Such results are beneficial to guide
software engineers in the effort estimations process and help
project managers in better allocating of resources which could
meet specific project needs.
providing more relevant effort distribution guidance to meet
specific project needs compared to the original COCOMO 81
model.
The reminder of the paper is structured as follows: section 2
introduces an overview of related work; section 3 describes the
objective, subject, and approach of the study; section 4 presents
major results of phase effort distribution profiles on the 75
industry projects; section 5 discusses the results, and section 6
concluded the paper with directions for future work.
One major cause of the lack of phase distribution data is because
of large variation in the lifecycle phases/activities defined in many
leading estimation methods. This complicates the lifecycle
concept synthesizing, model usage, data collection and analysis,
as well as cost model calibration. Table 1 summarizes the
comparison among phase coverage of several leading estimation
models.
2. OVERVIEW OF RELATED WORK
Table 1. Comparison of phase definitions in different model
Empirical assessment of industrial software development project
data, as an integral part of many effort distribution studies, has led
to the production of several well-known and widely adopted bases
to guide software estimation practices. In his seminal work to cost
estimation field, Norden [14] observed the Rayleigh Curve to be a
good staffing level approximation to most hardware development
projects. Though it has been criticized for its inapplicability in
software development projects due to the slow early build-up and
the long-tail effects, Rayleigh distribution has influenced on the
effort distribution mechanisms in many later models, such as the
COCOMO model [6, 11] and the SLIM model [15]. Based on
further empirical studies on effort distribution data, Boehm
presented the waterfall phase/activity breakdown quantities in the
COCOMO model [6, 11], and Krutchen concluded the lifecycle
effort/schedule distribution structure used by the Rational Unified
Process (RUP) [12].
The COCOMO 81 model’s [11] way of handling phase
distribution is to provide a distribution scheme consisting of ratios
for every development phase (i.e. product design, code,
integration and test, for the waterfall processes), approximating
the Rayleigh curve. Furthermore, it defines three development
modes, five levels of project scale, phase-sensitive effort
multipliers to help derive more accurate allocation decision with
respect to specific project settings. For example, COCOMO
model suggests that more development focus should be placed on
integration and test phase as the software size grows (i.e. 16% for
small projects and 25% for large projects) [6], and less focus to be
given to the code phase as the software size increases (i.e. 68% for
small projects and 59% for large projects). Meanwhile, the phasesensitive effort multipliers enable the estimation to be performed
on a more accurate, phase-wise basis, by tracking down the
influences of different effort multiplier ratings on phase
distribution variation. This offers an alternative to adapt the basic
distribution scheme to a better-fitted one according to a particular
project situation. However, the application of the detailed
COCOMO model is a rather complex procedure to produce the
forms and tables following strict steps, which requires intensive
project knowledge upfront.
The COCOMO II [6] model unifies the phase-sensitive effort
multipliers to be the same for all development phases for several
considerations. Simplification of model usage and lack of detailed
COCOMO calibration data are the two major reasons. The
COCOMO II extends the COCOMO 81 model to include the
“plans and requirements” and “transition” phases into its waterfall
lifecycle effort distribution scheme, and a MBASE/RUP
phase/activity distribution scheme adopted from [12]. One
drawback of this simplification is the lack of flexibility in
Model
The phases (effort distribution percentages)
covered in different models
COCOMO
81 [11]
Plan and requirement, preliminary design,
detailed design (25%), code (33%), Integration
& Test (25)
COCOMO
II [6]
Waterfall distribution scheme: Plan and
requirement (7%), preliminary design (17%),
detailed design (25%), code (33%), Integration
& Test (25), deployment & maintenance (12%);
MBASE/RUP
distribution
scheme:
Inception(6%),
Elaboration(24%),
Construction(76%), Transition (12%)
RUP [12]
Inception(5%),
Elaboration(20%),
Construction(65%), Transition(10%)
COCOTS
[6]
Assessment, Tailoring, Gluecode&Integration.
No unified distribution guideline
SLIM [15]
Concept Definition, Requirement & Design,
Construct & Test, Perfective Maintenance
(distribution percentage not available)
SEERSEM [17]
Early specification, design, development,
delivery & maintenance (distribution percentage
not available)
One the other hand, while traditional methods are more focused
on the improvement of general estimation accuracy [6, 18], a
number of recent studies start to investigate different aspects of
effort distribution patterns. Heijstek and Chaudron [10] reported
empirical data on effort distribution from projects employed
model-based development practices, and confirmed the similarity
between the reported effort distribution and the original RUP
hump-chart [12]. Milicic and Wholin [13] studied characteristics
affecting estimating accuracy and proposed a lean approach to
improve estimation based on distribution patterns of estimation
errors. Yiftachel et. al. [18] developed an economic model for
optimal allocation of resources among development phases. Other
researchers also studied on determining optimal effort distribution
across lifecycle phases based on defect-introduction and defectslippage considerations [19, 20]. However, few of these studies
have performed a comprehensive analysis of phase effort
distribution patterns and its possible causes. Consequently,
notable inconsistency exists in results from different researchers
regarding phase distribution patterns even for the same
development life cycle, RUP, as concluded in [10].
Meanwhile, communications and discussions with estimation
experts and practitioners at the COCOMO community indicates
an increasing interests in the needs for more phase-sensitive
distribution facility than what are available now from COCOMO
II default schemes. All this confirms our motivations of this study.
3. OBJECTIVE, SUBJECT AND
APPROACH OF THE STUDY
3.1 Objective of the study
The objective of this study is to perform extensive data analysis
towards developing a more in-depth understanding on what
factors impact on the degree of intensity of different development
phases, and eventually facilitating strategic planning through more
reasonable phase effort prediction. Concluded from previous
studies on ISBSG[21, 22], we focused on four project factors
including development life cycle model, development type, size
of the software, and team size, trying to derive preliminary
answers to our objective. Hence, the central research questions are
formulated as:
1)
What does the overall phase distribution profile of
development effort look like?
2)
Is there a causal relation between difference development
life cycles and phase effort distribution?
3)
Is there a causal relation between phase distribution pattern
and development type, e.g. new development or
enhancement?
4)
Is there a causal relation between phase distribution pattern
and actual software size?
5)
Is there a causal relation between phase distribution pattern
and maximum development team size?
Code
Code, Unit Test, Integration
Test
System Test
Transition
Installation, Transition,
Training, Support
Data collection. Firstly, the CSBSG staff sent questionnaire to
organizations. The electronic questionnaire has the capability to
check some error automatically, such as spelling error and data
inconsistency. Then, after the organization sent back the data to
CSBSG, a special expert group checks the data quality again and
confirms it is eligible to put into the CSBSG database. During this
procedure, there is some specialized focal contact personnel to
connect with the organization and the expert group, and the
information of the organization is hidden in the data submitted to
the expert group; the contact person also reacts when some data
problem is founded.
Data Cleaning. Due to the tight schedule of the data collection
process, there are many phase-level attributes unreported, such as
predicted and actual effort of each development phase. Therefore,
for the purpose of our study, we only make use of a subset of the
CSBSG database with data on our focused attributes. The
attributes and data cleaning steps are summarized in Table 3. Next
we will briefly introduce the data cleaning steps.
Table 3a Summary of the minimum set of attributes
considered in the study
Metric
Size
Effort
Plan phase effort
Empirical data used in this study is from the China Software
Benchmarking Standard Group (CSBSG) [16]. The CSBSG was
established in Jan. 2006 under the support of Chinese government
and software industry association, to advocate and establish
domestic benchmarking standards for system and software process
improvement in Chinese software industry. The establishment of
CSBSG database is consistent with the ISBSG database [21], and
each data point is defined as over 1000 product and process
metrics. The CSBSG dataset we used contains 1012 software
development project data points from 141 organization and 15
geographical regions.
Reqt’s phase effort
Table 2 Activities defined in the CSBSG phases
Phase
Activities Included
Plan
Plan, Preliminary Requirement Analysis
Requirement
Requirement Analysis
Design
Product Design, Detailed Design
User
3.3 Data Collection and Cleaning
3.2 Subject of the study
Development phases defined in CSBSG database roughly follow
the waterfall model, with a comprehensive list of guidelines for
individual data submittal organization to match their own process
and transform data appropriately. The following table lists the
CSBSG development phases and major activities included in each
phase.
Acceptance Test,
Design phase effort
Code phase effort
Test phase effort
Transition phase
effort
Development
life
cycle
Team Size
Development Type
Unit
SLOC
PersonHour
PersonHour
PersonHour
PersonHour
PersonHour
PersonHour
PersonHour
Nominal
Person
Nominal
Description
Total Lines of Code
Summary Work Effort
Work Effort of plan phase
Work Effort of reqt’s phase
Work Effort of design phase
Work Effort of code phase
Work Effort of test phase
Work Effort of transition phase
Waterfall, iterative, rapid
prototyping
Maximum size of the development
team
New development, Enhancement,
Re-development
Table 3b The process of data selection
Step
ID
1
# of proj.
excluded
2
14
# of proj.
remained
115
101
Reason for exclusion
Do not contain phased-related
effort records
there are two or more phases’
effort data missing
3
1
the sum of its 5 recorded phased
effort is greater than the total effort
if only 1 of the 5 phase effort value is missing, it is filled up by
subtracting the other 4 phases from the total effort
25
75
one or more than one phase’s effort
data recorded as zero, and they are
excluded and considered to not
cover the whole life-cycle phases
4
5
100
At first, 115 data points containing phased-related effort records
were extracted out of the whole CSBSG dataset. Among these, 14
data points were excluded because there are two or more phases’
effort data missing. Furthermore, 1 project is identified to be bad
data and excluded, because the sum of its 5 recorded phased effort
is greater than the total effort. Additionally, there are 25 projects
with one or more than one phase’s effort data recorded as zero,
and they are excluded sine the records are considered to not cover
the whole life-cycle phases. Finally, 75 projects with complete
phased effort values are included in this research.
4. RESULTS
The 75 projects came from 46 software organizations distributed
in 12 regions across China, and the largest number of projects
from the same organization is 8. Table 4a at below summarizes
the project data on size and effort of our dataset. Note the
significant difference between the mean and median of the size
metric is mainly due to 1 very large project whose size is greater
than 1000KSLOC.
4.1 Overall Phase Distribution Pattern
Table 4b summarizes the median, mean, and standard deviation of
effort distribution percentage for each of the five CSBSG
development phases after the combining of the original plans and
requirements phases.
Table 4b Overall phase distribution profile
Phase Plan&Req.
1.82%
Min
35%
Max
15.94%
Median
16.14%
Mean
8.62%
Stdev
Design
0.62%
50.35%
14.21%
14.88%
8.91%
Code
6.99%
92.84%
36.36%
40.36%
16.82%
Test
4.24%
50.54%
19.88%
21.57%
11.04%
Trans.
0.06%
36.45%
4.51%
7.06%
7.06%
Though it is generally believed that “major variations in both
waterfall and RUP phase distribution quantities come in the
phases outside the core development phases” [6], the CSBSG data
shows that code and test are the two development phases with
greater variation than other phases. For example, the project with
the least design phase effort distribution quantity, i.e. 0.62%, is
also the same project with the greatest percentage numbers in
code phase, i.e. 92.84%.
To examine the differences of individual phase distribution
between CSBSG dataset and COCOMO II recommendations, we
compare the mean distribution profile of CSBSG dataset with the
COCOMO II waterfall distribution quantities.
Table 4a Summary of the project data
Median
Min
Max
Size (KSLOC)
136.4
45.7
0.77
2340
Effort (Person-Hours)
8969
4076
568
134840
45%
40%
35%
Effort %
Mean
To illustrate the stability of the variation associated with the
dataset, scatter-plot of the log-transformed size and effort is given
in Figure 2. It indicates a significant linear relationship between
them (significance level is based on p-value <0.01), and the Rsquare of the linear regression line in the chart is 0.5231.
30%
25%
20%
15%
10%
5%
0%
Plan&Req.
Design
Code
T est
T rans.
Phase
14
R2 = 0.5231
12
COCOMO II
CSBSG
Ln(Effort)
10
Figure 3. Comparison of overall waterfall distribution
quantities between CSBSG and COCOMO II datasets
8
6
4
As shown in Figure 3, distribution similarities appear only in the
Test and Transition phases. Some significant differences between
the two datasets include that:
2
0
-2
0
2
4
6
8
10
1)
a greater emphasis on Plans and Requirements phase in
CSBSG dataset: 16.14% of overall development effort,
compared to 6% in COCOMO II;
2)
the design phase is dramatically less resource-allocated in
CSBSG projects (i.e. 14.88%) than COCOMO II projects
(35% in average); and
Ln(Size)
Figure 2. Scatterplot of total development effort against
software size after log-transformation.
3)
the average distribution percentage for Code phase (40.36%)
in CSBSG projects is much higher than that (27.8% in
average) in COCOMO II projects.
One possible explanation is that the great distribution percentage
for code phase in CSBSG database may have resulted from some
projects’ insufficiently resourced design phase, which might cause
frequent rework in code phase.
4.2 Development life cycle
Excluding 1 project with bad data on development life cycle with
vague description, our dataset consists of projects employed three
different development life cycles: waterfall (57 projects), iterative
(15 projects), and rapid prototyping (2 projects). Due to the small
sample size of rapid prototyping projects, we exclude it from
further analysis. Figure 4 illustrates the comparison of phase
distribution profiles of waterfall and iterative models using
median value.
Effort%
A further examination is also performed to check the average
software size of projects employing each of these three
development life cycles. The results show that the average
software size for these three groups is: 103KSLOC, 259KSLOC,
and 85KSLOC for waterfall, iterative, and rapid prototyping
respectively. It indicates that iterative is the mostly deployed life
cycle model for ultra large projects.
4.3 Development Type
Development type is another factor that might affect phased effort
distributions. Three software development types are pre-defined
by the CSBSG database, namely new development (New), redevelopment (ReDev), and enhancement (Enhance), according to
the following definitions:

New development: where a software product was developed
and introduced for the first time to satisfy specific customer
needs;

Re-development: where new technologies was incorporated
in replacing or upgrading an existing software product in
use;

Enhancement: where new functionality requirements were
designed and implemented to modify or extend an existing
software system.
Effort%
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
since the concept of phase in iterative or rapid prototyping
development is not as clear as that in waterfall processes, data
collection process is likely to involve greater participant bias in
matching and transforming data from their own process
breakdown structure to CSBSG process structure.
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Plan&Req.
Design
Code
Test
Trans.
Phase
Plan&Req.
Design
Code
Test
Re-Dev
Trans.
Phase
Waterfall
New Dev
Enhance
Figure 5 Comparison among difference development types
Iterative
Figure 4 Comparison among different process models
Table 5 Summary of phase distribution differences from Mean
Effort% for whole dataset by development type
LifeCycle
Model
N
Plan &
Req.
Design
Code
Waterfall
Iterative
57
15
0.00%
-2.38%
1.85%
-1.45%
-0.88%
7.42%
Test
Each of the 75 data points in our study contain a development
category identified by the project team based on the above
definition. The median effort ratios of each phase for three
development type projects are illustrated in Figure 5, and the
phase distribution deviation from the total average is listed in
Table 6. The numbers of projects for the three groups are: 6
(ReDev), 53 (New), and 16 (Enhance).
Trans.
0.17% -0.99%
-1.60% 2.67%
The phase distribution deviation from the total average is listed in
Table 5. Iterative processes, most suitable for complex large scale
projects in CSBSG dataset, employ less effort percentage in plan
and requirement, design, and test phases. It is actually due to
iterative testing and integration effect on the basis of multiple
iterations. This can also explain the highest distribution
percentage of iterative development in code phase, because the
iterative integration effort is included into code phase, according
to CSBSG phase definition discussed in section 3.2. However,
Table 6 Summary of phase distribution differences from Mean
Effort% for whole dataset by development type
Dev type
N
Plan &
Design
Req.
ReDev
New Dev
Enhance
6
53
16
-3.64%
-0.44%
2.82%
-3.96%
-0.12%
1.88%
Code
Test
Trans.
8.95%
2.20%
-10.67%
-4.96%
-1.07%
5.40%
3.59%
-0.58%
0.56%
New development and enhancement projects seem to have similar
distributions in most of the development phases, except that new
development projects spend more effort (roughly 13% more) in
Code phase, while enhancement projects have greater focus on
A further examination on the estimation accuracy of phase
effort distribution in the three development types shows that the
Transition phase has been overestimated in all projects as shown
in Figure 6. The accuracy measure used is the relative error (RE)
percentage, which is calculated by the equation of RE =
(Estimated phase effort – Actual phase effort)/Actual phase effort,
for each individual development phase. Further investigation is
needed to explain the overestimation phenomena in transition
phase in CSBSG dataset.
28
30
24
25
# of Projects
Test phase (about 6.5% more). This is because enhancement
development involves through system testing even though the
modifications happen on a smaller code base. At the meantime,
re-development type has the greatest distribution emphasis on
Code phase. A possible reason is that re-development projects
involve the migrating of existing system to a different technology,
but not requirements or design by its definition.
20
15
12
10
6
4
5
1
0
<2
2~8
8~32
32~128
128~512
>512
KSLOC
Figure 7 Distribution of software size
50%
40%
Phase Error
30%
20%
10%
0%
-10%
Plan &
Req.
Design
Code
T est
T rans.
-20%
Phase
ReDev
New
Enhance
Figure 6. Comparison of estimation accuracy
The data also indicates that new development projects have much
lower estimation error and hence more easier to estimate than the
other two types for almost all phases, except the underestimation
for the test phase. Moreover, design and code phases have been
underestimated for the re-development projects.
First of all, all the projects are divided into six groups: Small,
Intermediate, Medium, Large, Ultra-Large, and Super-Ultra-Large
according to the ranges adapted from COCOMO model, as shown
in Figure 7. About 69% of projects in our dataset are medium to
large scale projects, with 24% of projects are ultra-large scale or
even bigger, while the small project ratio is relative as low as 7%.
The data indicates that software size data in CSBSG dataset
roughly follows a normal distribution after taking logtransformation, consistent with the distribution reported in the
ISBSG dataset [21].
Figure 8a depicts the average phase distribution of each size
group. The CSBSG data does not show a clearly consistent
decreasing trend in design and code phase, or an increasing trend
in integration and test phase, as software size increase from
smaller to larger scales. However, such trends are only observed
between ultra-large group and super-ultra-large group. One
possible reason might be that projects in CSBSG dataset were
mostly completed in the past 3 years supported by advanced
programming technologies, and it is likely for the size data to take
the automatically generated code into account. However, this may
again challenge on the validity of using LOC as the unit for
effective software size.
4.4 Software Size
It is noted that size metrics used in our study is Lines of Code. In
this dataset, only 1 project recorded its size in Function Point. At
the same time, another 3 projects from the same organization as
this project are found, and they also use the same primary
programming language, Java; moreover, these 3 projects have
size records in both FP and LOC, and the ratio of LOCs per FP
are all 53, which is consistent with the transformation ratio
reported in SPR documentation [23]. In that case,
transforming the size of this project into LOC metrics is
considered to be reasonable.
60%
50%
40%
30%
20%
10%
0%
P&R
2 KLOC
8 KLOC
Design
32 KLOC
Code
Test
128 KLOC
512 KLOC
Trans.
>512 KLOC
Figure 8a Comparison among difference software sizes scale in
LOC
4.5 Team Size
60%
Maximum team size is another metric collected in CSBSG dataset.
It is also reported in [24] that the maximal number of members
50%
involved in the whole project life-cycle is easier to measure
than average team size. An analysis of its influence on phase
distribution is performed to explore possible causality. Since one
project contains missing data on max team size, the other 74
projects are included in this analysis. The average phase
distribution ratio results are depicted in Figure 9. And the phase
distribution differences from the total average value in each group
are listed in Table 8.
40%
30%
20%
10%
45%
0%
40%
Plan & Req.
Design
XXS
M1
Code
M2
L
Test
XL
XXL
Trans.
30%
XXXL
Figure 8b Comparison among difference software sizes in FP
scales
We then considered to use ofFunction Point (FP) as alternative
software size metric. Following backfiring ratios in [23] and FP
sizing scales in [25], we re-group all 75 projects into 7 groups,
from “Extra-extra-small” to “Extra-extra-extra-large”. The effort
distribution comparison of the 7 groups is illustrated in Figure 8b.
According to this categorization, the phase distribution variations
in each group are listed in Table 7.
Table 7 Summary of phase distribution variation by software
size in FP scales
Size
(KLOC)
XXS
M1
M2
L
XL
XXL
XXXL
Size
(KLOC)
XXS
M1
N
M2
L
XL
XXL
XXXL
1
Plan &
Req.
7.72%
-1.18%
2.40%
1.47%
-5.90%
-5.32%
0.16%
Plan &
Req.
7.72%
11
-1.18%
27
2.40%
17
1.47%
9
-5.90%
6
-5.32%
4
0.16%
1
11
27
17
9
6
4
N
Design
Code
9.42%
-0.66%
0.74%
-0.01%
-0.55%
-3.66%
1.21%
-18.88%
-3.29%
-4.25%
-0.08%
12.83%
15.74%
-9.69%
35%
Test
Trans.
-2.56%
6.55%
2.74%
-2.64%
-5.69%
-4.87%
-4.56%
4.30%
-1.42%
-1.63%
1.27%
-0.69%
-1.89%
12.87%
Design
Code
Test
Trans.
9.42%
0.66%
0.74%
0.01%
0.55%
3.66%
1.21%
-18.88%
-2.56%
4.30%
-3.29%
6.55%
-1.42%
-4.25%
2.74%
-1.63%
-0.08%
-2.64%
1.27%
12.83%
-5.69%
-0.69%
15.74%
-4.87%
-1.89%
-9.69%
-4.56%
12.87%
The data shows that as the software size grows from small to
medium, more development effort should be allocated to code and
test phases; as the software size grows from medium to large,
more development effort should go to plan and requirement and
code phases; when the software size is extra large, more attention
should be paid to design, code, and test phases.
25%
20%
15%
10%
5%
0%
P&R
Design
5
Code
10
15
Test
Trans.
>15
Figure 9 Average phase distribution for different team size
Table 8 Summary of phase distribution differences from Mean
Effort% for whole dataset by team size
Team
size
N
5
10
15
>15
31
27
13
3
Plan &
Req.
2.22%
-1.45%
-2.26%
-2.76%
Design
Code
Test
Trans.
-0.90% 0.16%
0.23%
0.51%
0.66% -0.15%
1.27%
0.64%
-1.85%
0.30%
3.16%
3.52%
0.36%
0.40%
-1.42%
-2.67%
The data indicates a consistent relationship between maximum
team size and phase distribution pattern. It shows a
distinguishable emphasis on Test phase with the growth of team
size..
5. ANALYSIS
The discussions in previous section contain some indications to
the research questions. However, to further elaborate more
statistical proofs about factors influencing phased distribution and
their relation to the phase distribution patterns, we performed
more statistic analysis on our dataset.
5.1 Factors Influencing Phased Distribution
ANOVA analysis is used to examine to what a degree the variance
of each phase effort distribution percentage is explained by class
variables, i.e. development type, software size, and team size. The
variable of development life cycle was excluded from this study
because possibly biased data from projects that can not easily
mix-match their non-waterfall-based data into CSBSG structure,
as well as its correlation with software size, as discussed in
section 4.2.
The data summarized in Table 9 shows the degree of variance
explained by an influencing factor for the variation in individual
phase effort distribution. For example, it shows that FP software
size is the major metric to be considered when making
distribution decisions for all development phase with the highest
values in all phases. It also indicates that FP is a stronger
predictor in measuring effective software size and predicting
development effort. This confirms the software-scale-sensitive
distribution scheme in COCOMO model. Development type is
another major factor to be considered in determining appropriate
adjustment for code and test phases. However, team size factor is
a less significant factor to influence effort distribution on all
phases.
Table 9 Summary of ANOVA results
Phase
Distribution
DevType
Software size
FP scale
LOC scale
5)
CSBSG dataset shows that for enhancement type of projects,
percentage of development effort in code phase decreases by
10.67%, and that for test phase increases by 5.4%.
6)
CSBSG dataset indicates that as software size dramatically
grows, distribution has an intensive focus on both code and
test phases.
7)
CSBSG dataset indicates that team size is not a significant
factor which may cause phase effort distribution variation.
However, it shows a distinguishable emphasis on Test phase
with the growth of team size
5.3 Threats to Validity
Two major threats to external validity of our study are:
Team
size
P&R%
3.96%
14.80%
13.65%
5.50%
Design%
2.57%
3.67%
3.36%
0.62%
Code%
12.22%
7.93%
20.56%
0.02%
Test%
7.47%
7.16%
14.59%
3.05%
Trans.%
2.72%
9.36%
22.45%
1.52%
Through such analysis, parametric model can be developed in
order to sophisticatedly consider these influencing factors, and
predict appropriate amount of adjustment to be made against the
nominal quantities offered by current methods for estimation and
benchmarking purposes.
5.2 Guidelines for Phase Allocation
Though it is infeasible to deploy a unified distribution scheme
over different types of projects, analyzing empirical effort data
helps to develop understanding to variations and possible causes
of phase effort distribution, identify most frequently presented
effort distribution patterns, and conclude insightful suggestions to
lead to improved project planning practices. Concluded from the
observations and analysis results of this study, the following
general guidelines are provided for further discussion, adoption,
and validation by software estimation and management
practitioners, esp. in Chinese software industry:
1)
Analysis on determining reasonable phase effort distribution
quantities should be performed in the cost estimation
process.
2)
CSBSG data analysis shows a waterfall-based phase
distribution scheme as: 16.14% for plans and requirements
phase, 14.88% for design phase, 40.36% for code phase,
21.57% for test phase, and the other 7.06% for transition
phase.
3)
CSBSG data shows that iterative development projects have
allocation intensity on code (+7.42%) and transition
(+2.67%) phases compared to median values of the overall
dataset.
4)
FP-based software size and development type are two major
factors to be considered when adjusting effort allocation.
1)
Data representativeness. The dataset used in our study was
pre-collected by CSBSG through its member organizations.
Hence, the data may represent software organizations with
above average performance, because otherwise they are
rarely willing to submit data even if they have collected such
historical records.
2)
Data quality. On the other hand, the data collection process
is likely to bring in individual biases or participant error
which may involve mis-reporting or double counting issues.
Even though we have gone through strict data clean and
exclusion criteria, this still remains a big risk to the
trustworthiness of this study.
The major threat to internal validity is the generalization of the
empirical findings. Given that all projects are from domestic
China mainland, in which many external and internal conditions
might differ greatly from oversea projects, the findings will be
more reliable and appropriate for projects from a similar setting..
In addition, for the issue of sizing metrics, one may question the
way how each organization count the lines of code. In fact, we
cannot figure out that in detail, and it really leaves us a question
expected to be further explore in the future work. However, as we
know, since the scare resource of expert in Function Point Method,
most organizations still choose the relative direct and original
counting method to measure the software size.
Another issue has to do with the interpretations following the
analysis on each influencing factor. Most interpretations are
compared with COCOMO findings and only with respect to
individual factors, i.e. development life cycle, development type,
software size, and team size. Possible correlations among multiple
factors are not studied and may lead to counter-intuitive findings.
Moreover, analysis and exclusion of potential outliers from the
subject is not included in the current study. This could also reduce
the validity of this study.
6. CONCLUSION AND FUTURE WORK
Poor effort estimation and allocation comes from lack of
recognition to process variations and lack of understanding to
process effort distribution patterns. However, limited amount of
studies are found to date to develop in-depth exploration what
factors cause the variations in effort distribution.
The work performed within the study aimed to examine the effort
distribution profile using 75 project data from a certain number of
Chinese software organizations to gain additional understanding
about the variations and possible causes of waterfall-based phase
effort distribution. The phase effort distribution patterns were
compared with that from the COCOMO model, and further
analysis examines the relationships between the distribution
patterns and some significant factors including development life
cycle, development type, software size, and team size. It identifies
distinguishable phase effort distribution similarities and
differences, and provided guidelines for improving future
estimation and allocation practices. Such empirically-based
information offers various beneficial outcomes in developing
better understanding to development phase focuses and predicting
cost estimation and phase allocation.
Future work includes investigation and extension of current study
to address the following issues:
[4] Feller, J. and Fitzgerald, B.: A Framework Analysis of The
Open Source Software Development Paradigm.
Proceedings of the 21st International Conference in
Information Systems (ICIS 2000). (2000), 58-69.
[5] The Standish Group. 2004 the 3rd Quarter Research Report,
(2004). http://www.standishgroup.com
[6] Boehm, B.W., et al.: Software Cost Estimation with
COCOMO II. Prentice Hall, NY(2000)
[7] Reifer, D.: Industry software cost, quality and productivity
benchmarks, software. Tech News 7(2) (July 2004)
[8] Kroll, P.: Planning and estimating a RUP project using
IBM rational SUMMIT ascendant. Technical report, IBM
Developerworks (May 2004)

Continuing on similar analysis to search for other significant
influencing factors, possible candidate including
requirement volatility and business area;

[9] QSM Inc.: The QSM Software Almanac: Application
Development Series (IT Metrics Edition) Application
Development Data and Research for the Agile Enterprise.
Quantitative Software Management Inc., McLean,
Virginia, USA (2006)
Strengthening the findings through more thorough statistical
analysis including interrelationship modeling among
multiple factors and proper outlier identification and
handling;
[10] Heijstek, W. and Chaudron, M. R. V.: Effort distribution in
model-based development. 2nd Workshop on Model Size
Metrics (2007)

Using linear regression to construct a parametric model of a
consolidated set of predicting factors to calculate phase
sensitive effort distribution quantities for a given project
setting;

Extending the scope of current analysis to cover phase
schedule distribution analysis.
7. ACKNOWLEDGMENTS
We would like to thank the Chinese Systems and Software
Process Improvement Association for their data support of this
study and the anonymous reviewers for their constructive
suggestions.
This work is supported by the National Natural Science
Foundation of China under grant Nos. 60573082，90718042; the
National Hi-Tech Research and Development Plan of China under
Grant No. 2006AA01Z182， 2007AA010303; the National Key
Technologies R&D Program under Grant No. 2007CB310802.
8. REFERENCES
[1] Boehm, B. and C. Abts, Software Development Cost
Estimation Approaches-A Survey. Annals of Software
Engineering, 2000. 10(1-4): p. 177-205.
[2] Jorgensen, M. and M. Shepperd, A System Review of
Software Development Cost Estimation Studies. IEEE
Transaction on Software Engineering, 2007. 33(1): p. 3353.
[3] Basili, V.R.: Software development: a paradigm for the
future. Proceedings of the 13th Annual International
Computer Software and Application Conference, (1989)
471-485.
[11] Boehm, B.W.: Software Engineering Economics. Prentice
Hall, (1981)
[12] Kruchten, P.: The Rational Unified Process: An
Introduction. Addison-Wesley Longman Publishing Co.,
Inc., Boston, MA, USA (2003)
[13] Wohlin, C.: Distribution Patterns of Effort estimation.
EUROMICRO Conference (2004)
[14] Norden P.V.: Curve Fitting for a Model of Applied
Research and Development Scheduling. IBM J. Research
and Development, Vol. 3, No. 2, (1958), 232-248.
[15] Putnam, L. and Myers, W. (1992), Measures for
Excellence, Yourdon Press Computing Series.
[16] He, M., et al.: An Investigation of Software Development
Productivity in China. ICSP 2008, (2008) 381-394
[17] SEER-SEM. http://www.galorath.com/index.php
[18] Peleg Yiftachel, et al.: Resource Allocation among
Development Phases: An Economic Approach. EDSER’06,
May 27, 2006, Shanghai, China, (2006) 43-48
[19] Huang, L., Boehm, B.: Determining how much software
assurance is enough?: a value-based approach. In: EDSER
’05: Proceedings of the seventh international workshop on
Economics-driven software engineering research, NY,
USA, ACM Press (2005) 1–5
[20] Yiftachel, P., Peled, D., Hadar, I., Goldwasser, D.:
Resource allocation among development phases: an
economic approach. In: EDSER ’06: Proceedings of the
2006 international workshop on Economics driven
software engineering research, New York, NY, USA, ACM
Press (2006) 43–48
[21] Jiang, Z., Naudé, P. and Comstock, C.: An investigation on
the variation of software development productivity.
International Journal of Computer, Information, and
Systems Sciences, and Engineering, Vol. 1, No. 2 (2007)
72-81
[22] ISBSG Benchmark Release 8. http://www.isbsg.org
[23] SPR Programming Languages Table 2003, Software
Productivity Research. ttp://www.spr.com
[24] Agrawal, M., Chari, K.: Software Effort, Quality and Cycle
Time: A Study of CMM Level 5 Projects. IEEE
Transactions on Software Engineering, Vol. 33, No. 3
(2007) 145-156
[25] Software Measurement Services Ltd. ‘Small project’,
‘medium-size project’ and ‘large project’: what do these
terms mean?”
http://www.measuresw.com/library/Papers/Rule/RulesRelat
iveSizeScale%20v1b.pdf
[26] http://www.csbsg.org