Risk management and Process
Improvement of Off-The-Shelf Based
Development
Jingyue Li (jingyue@idi.ntnu.no),
Reidar Conradi, Odd Petter N. Slyngstad,
Norwegian University of Science and Technology
Marco Torchiano, Maurizio Morisio,
Dip.Automatica e Informatica, Politecnico di Torino
Christian Bunse
Fraunhofer IESE
CBSE Seminar -4 Feb 2005- OSLO
1
Agenda

Research design
• Background
• Research questions
• Sample selection

Results
• Selected samples
• Answers to research questions


Discussions
Conclusions and future work
CBSE Seminar -4 Feb 2005- OSLO
2
Research design – Motivation

Pre-study background
• This study followings a pre-study with 16 structured
interviews in Norway, from Oct. 2003 to Feb. 2004.
• Focused on SPI in COTS-based development
• Respondents shared a lot of experiences on risk
management in COTS-based development
• Limitations of the pre-study
 Small sample size
 Sample selected on convenience

Motivation of this main study
• State-of-the-practice survey
• Randomly selected much larger samples to validate
conclusions of the pre-study
• Also included Open Source Component
CBSE Seminar -4 Feb 2005- OSLO
3
Research design – research
questions




RQ1 - How to improve the development
process in projects using OTS
components.
RQ2 - How to predict possible risks
(problems) in projects using OTS
components?
RQ3 - What are the effective methods to
mitigate risks in projects using OTS
components?
RQ4 - What are the similarities and
differences between projects using COTS
and OSS components?
CBSE Seminar -4 Feb 2005- OSLO
4
Research design – sample
selection



Norway
Germany
Italy
(Sample selection reported in later
presentation)
CBSE Seminar -4 Feb 2005- OSLO
5
Research results – selected
samples

Current data
• Total 86 projects
• Norway



46 projects from 38 companies
One company filled in 4, one filled in 3, and one filled in 2.
In other companies, we selected only one project each
company
• Germany

29 projects from 29 companies
• Italy


11 projects from 11 companies
Data collection is still on-going in Germany and
Italy
CBSE Seminar -4 Feb 2005- OSLO
6
Research results – selected
companies
Company's main business
3%
12 %
Software house
43 %
IT consulting
IT department of a
traditional industry
Telecom. Industry
42 %
CBSE Seminar -4 Feb 2005- OSLO
7
Research results – selected
companies (cont’)
Company size
Small
29 %
30 %
Medium
Large
41 %
Small (0-19) Medium (20-99) Large (more than 100)
CBSE Seminar -4 Feb 2005- OSLO
8
Research results – selected
projects
Application domain of the integrated system
Traditional industry
23 %
26 %
Bank
Other private
services
Public sector
20 %
ICT sector
19 %
12 %
CBSE Seminar -4 Feb 2005- OSLO
9
Research results – selected
respondents
Role of respondents
14 %
24 %
IT Manager
Project manager
20 %
Software architect
Developer
42 %
CBSE Seminar -4 Feb 2005- OSLO
10
Research results – selected
respondents (cont’)


85% respondents have more than 3
years experience on OTS-based
development
Most respondents have the Bachelor
degree in informatics, 10% have
Ph.D degree.
CBSE Seminar -4 Feb 2005- OSLO
11
Research question RQ1
 How to improve the development
process in projects using OTS
components?
• Overall development process
Do I need to change my main development
process dramatically in projects using OTS?
 What activities and roles should be added?

• OTS selection process
Formal decision making process?
 Familiar with component process?

CBSE Seminar -4 Feb 2005- OSLO
12
RQ1: Do I need to change my main
development process dramatically?


More than 80% projects members
decided their main development
process (Waterfall, incremental, etc.)
before they started to think about
using OTS.
It actually worked.
CBSE Seminar -4 Feb 2005- OSLO
13
RQ1: What should be added?

Activities
• ”Acquire” vs. ”build” decision
• OTS component selection
• Learning OTS component
• Build glueware and/or addware

A new role (OTS knowledge keeper)
• Germany (100%)
• Norway (37%)
• Italy (9%)
CBSE Seminar -4 Feb 2005- OSLO
14
RQ1: What is the proper OTS
selection process?

Formal decision making process (by 15% used)
• Selecting evaluation criteria (factors)
• Collecting and assigning values to these criteria
• Applying formal decision making algorithms such as
MAUT or MCDA etc.

Familiar with component process (by 85% used)
• Search internet
• Limited to 2-3 components
• Download demo version and try it, then decide
Or
• Recommended from internal/external experts
CBSE Seminar -4 Feb 2005- OSLO
15
Research question RQ2

How to predict possible risks in
projects using OTS components?
• What were the most frequent risks
(problems) in practice?
• Was there any relationship between
those risks (problems) and the project
profile?
CBSE Seminar -4 Feb 2005- OSLO
16
RQ2: Typical risks
Phase
Risks
Project plan
The project was delivered long after schedule
Effort to select OTS components was not satisfactorily estimated
Effort to integrate OTS components was not satisfactorily
estimated
Requirement
Requirements were changed a lot
OTS components could not be sufficiently adapted to changing
requirements
It is not possible to (re) negotiate requirements with the customer,
if OTS components could not satisfy all requirements
Component
integration
OTS components negatively affected system reliability
OTS components negatively affected system security
OTS components negatively affected system performance
OTS components were not satisfactorily compatible with the
production environment when the system was deployed
CBSE Seminar -4 Feb 2005- OSLO
17
RQ2: Typical risks (cont’)
Phase
Risks
Maintenance
and evolution
It was difficult to identify whether defects were inside or
outside the OTS components
It was difficult to plan system maintenance, e.g. because
different OTS components had asynchronous release cycles
It was difficult to update the system with the last OTS
component version
Provider
Relationship
Provider did not provide enough technical support/ training
Information on the reputation and technical support ability of
provider were inadequate
CBSE Seminar -4 Feb 2005- OSLO
18
RQ2: Frequency of typical risks (problems)
in OTS based development
6
5
4
3
2
1
rm
fo
in
r ort
de p
viup
ro s
pck lemte
ckLa ob da
La pr up
yS
lo T nce
ep O a
D w en
l o nt
Fo ai cts
m e e
n ef g e
la d an g
P fy ch an
i h e
nt e c g
e i at u. n
Id ot eq ha
c
eg r t
Nw n
l o me
Fo ire
u
eq
R
ce
an
rm
fo
rt
er
P
fo
tyf
rie
oun ty
teic ili
Sa b rt
gr l ia fo
te e ef
In R on
i le
ct u
le ed
h
sc
se
er
ft
19
CBSE Seminar -4 Feb 2005- OSLO
74
75
83
79
80
84
78
85
85
78
67
79
83
N =
85
82
A
0
RQ2: Frequency of typical risks in OTS
based development (cont’)

Most frequent risks
• Effort to integrate OTS components was not
satisfactorily estimated
• Keep up with requirements evolution
• Identify defects inside or outside OTS
component

Least frequent risks
•
•
•
•
Negative reliability effect
Negative security effect
Negative performance effect
Lack provider information
CBSE Seminar -4 Feb 2005- OSLO
20
RQ2: Relationship between typical risks
(problems) and project context

The more different OTS-components
used in the project, the more
frequent the following risks:
• Identify whether defects were inside or
outside the OTS components
• It was difficult to update the system
with the last version OTS components
• Provider did not provide enough
technical support/training
CBSE Seminar -4 Feb 2005- OSLO
21
RQ2: Relationship between typical risks
(problems) and project context (cont’)

The higher the general experience on
OTS-based development in projects,
the less frequent the following risks:
• Effort to integrate OTS c components
was not satisfactorily estimated
• It was difficult to identify whether
defects were inside or outside the OTS
components
CBSE Seminar -4 Feb 2005- OSLO
22
RQ2: Relationship between typical risks
(problems) and project context (cont’)

The project with an OTS knowledge
keeper had less frequency on the
following risks than project without
OTS knowledge keeper:
• Difficult ot identify risks inside or
outside OTS components
• Lack the information of the vendors’
reputation and support ability
CBSE Seminar -4 Feb 2005- OSLO
23
Research question RQ3

What are the effective methods to
mitigate risks in projects using OTS
components?
• Which strategies had been frequently
used in practice?
• What were the effective strategies?
CBSE Seminar -4 Feb 2005- OSLO
24
RQ3: Proposed risk management
strategies





Customer had been actively involved in “acquire”
vs. “build” decision
Customer had been actively involved in OTS
component selection
OTS components were selected mainly based on
architecture and standards compliance, instead of
expected functionality
OTS components qualities (reliability, security
etc.) were seriously considered during selection
Effort in learning OTS component was seriously
considered in effort estimation
CBSE Seminar -4 Feb 2005- OSLO
25
RQ3: Proposed risk management strategies
(cont’)






Effort in black-box testing of OTS components
was seriously considered in effort estimation
Unfamiliar OTS components were integrated first
Did integration testing incrementally (after each
OTS component was integrated)
Local OTS-experts actively followed updates of
OTS components and possible consequences
Maintained a continual watch on the market and
looked for possible substitute components
Maintained a continual watch on provider support
ability and reputation
CBSE Seminar -4 Feb 2005- OSLO
26
RQ3: Frequency of using proposed risk
management strategies in practice
6
5
4
3
2
1
r
de
vi
o
pr
ch
et
at
rk
W
a t
m ra
ch e g
at int t
W l r
ta xpe
en e
em TS
cr O
st
In
fir
r
il a t
m r
a fo
nf ef
u st t
r
te fo al
t f v
ni e e
u ng ity
i l
rn a
a qu
Leful
st
fir
re
re t
ca
tu lec
ec e
it s on
ch in i si
ar er e c
d
om n
st r i
cu e
om
st
cu
0
27
CBSE Seminar -4 Feb 2005- OSLO
80
85
84
82
76
81
83
83
84
84
85
N =
RQ3: Frequency of using proposed risk
management strategies in practice (cont’)


The most frequently used risk management
strategies:
• OTS components qualities were seriously
considered in the selection process
• Unfamiliar OTS components were integrated first
• Did integration testing incrementally
• Local OTS-experts actively followed updates of OTS
components and possible consequences
The least frequently used risk management
strategies:
• Involve customers in the “acquire” vs. “build”
decision
• Invove customers in OTS selection
CBSE Seminar -4 Feb 2005- OSLO
28
RQ3: What were effective risk management
strategies ?
Risks
Effective risk management method
Estimate selection
effort
OTS components qualities (reliability, security etc.)
were seriously considered in the selection process
Estimate integration
effort
OTS components qualities (reliability, security etc.)
were seriously considered in the selection process
Follow requirement
changes
Maintained a continual watch on the market and
looked for possible substitute components
Plan maintenance
OTS components qualities (reliability, security etc.)
were seriously considered in the selection process
Lack provider support
Maintained a continual watch on provider support
ability and reputation
CBSE Seminar -4 Feb 2005- OSLO
29
RQ3: Risk management
recommendations in OTS-based
projects

Avoid risk
• Do not use too many different OTS
components in one project

Manage risk
• Manage the knowledge of OTS properly (Have
a OTS expert and share OTS experience
regularly)
• Spend enough time on OTS quality evaluation.
Hand-on trial is necessary
• Do not marry specific OTS. Be ready for
possible replacement
• Maintain a continual watch on provider support
ability and reputation
CBSE Seminar -4 Feb 2005- OSLO
30
Research question RQ4

What are the similarities and
differences between projects using
COTS and OSS components?
• Are there any similarities and
differences in:
Company, project, system profile ?
 Motivation of using them ?
 Frequency of risks (problems) ?

CBSE Seminar -4 Feb 2005- OSLO
31
RQ4: Selected samples – COTS
projects vs. OSS projects



56 projects used only COTS
25 projects used only OSS
5 projects used both COTS and OSS
(not considered in data analysis)
CBSE Seminar -4 Feb 2005- OSLO
32
RQ4: Are there any similarities and
differences in company profile ?
Company size
0,5
0,45
0,4
OSS
0,35
0,3
0,25
COTS
0,2
0,15
0,1
0,05
0
Small
Medium
CBSE Seminar -4 Feb 2005- OSLO
Large
33
RQ4: Are there any similarities and
differences in company profile ?
(cont’)
Company's main business
0,6
0,5
0,4
OSS
COTS
0,3
0,2
0,1
0
Software
house
IT consulting
IT department
of a traditional
industry
CBSE Seminar -4 Feb 2005- OSLO
Telecom.
Industry
34
RQ4: Are there any similarities and
differences in project profile ?
System application domain
0,35
0,3
0,25
0,2
OSS
COTS
0,15
0,1
0,05
0
Traditional industry
Bank
Other private services
Public sector
CBSE Seminar -4 Feb 2005- OSLO
ICT sector
35
RQ4: Are there any similarities and
differences in system profile ?
6
6
5
5
4
4
3
3
2
2
1
1
ov
pr
Im
ed
nc
fu
.
nc
fu y
t
ew ili
N i nb
a
nt
ai
M
ce
an
m
or
rf
Pe
.
.
t
ke
nc
fu
r
rta
fom
Etof
y
rit
cu
Se y
li t
bi
l ia
Re
e
ed
36
CBSE Seminar -4 Feb 2005- OSLO
m
Ti
e
r
rta
fom
Etof
y
rit
cu
Se y
li t
bi
l ia
Re
m
Ti
t
ke
.
nc
fu l ity
i
ew b
N i na
a
nt
ai
M
ce
an
m
or
rf
Pe
52
54
55
56
54
55
55
54
N=
OSS projects
COTS projects
23
24
25
25
24
25
25
25
N=
ov
pr
Im
0
0
RQ4: Are there any similarities and
differences in company, project,
and system profile ?

Our conclusion
• There is no difference in company,
project and system profile between
projects using COTS and OSS.
CBSE Seminar -4 Feb 2005- OSLO
37
RQ4: Are there any similarities and
differences in
motivation of using COTS vs. OSS ?

Commonalities
• Shorter time-to-market
• Less development and maintenance effort
• Higher reliability

Differences
• COTS



Follow the market trend
Paid software will give good reliability
Good support
• OSS



New technology
Free source code
Avoid the risk in OSS evolution
CBSE Seminar -4 Feb 2005- OSLO
38
RQ4: Are there any similarities and
differences in frequency of risks
(problems) ?

Commonalities
• Requirement changed a lot and it was
difficult to keep up with these changes

Differences
• COTS: higher risk on following evolution
of both requirements and COTS
component
• OSS: higher risk on getting good
support
CBSE Seminar -4 Feb 2005- OSLO
39
Questions ?
CBSE Seminar -4 Feb 2005- OSLO
40