-1:08 AM 08.03.2016
Survey on Risk Management in OTS-based Development in IT
Industries in Several Countries
(Draft Design Version 1.15, 24 May 2004)
Reidar Conradi1,2, Jingyue Li1, Odd Petter N. Slyngstad1,
Vigdis By Kampenes2, Dag Sjøberg2, Maurizio Morisio3, Macro
Torchiano3, and Christian Bunse4
1
Norwegian University of Science and Technology (NTNU)
NO-7491 Trondheim, Norway
Tel +47 73.593444(rc), Fax +47 73.594466,
{conradi, jingyue, oslyngst}@idi.ntnu.no
2
Simula Research Laboratory
P.O.Box 134, 1325 Lysaker, Norway
3
Politecnico Di Torino
4
Fraunhofer-Institute for Experimental Software Engineering (IESE)
Preliminary abstract
This report describes an initial version of the empirical design on risk management in
COTS-based development in IT industry in several countries (Norway, Germany, and
Italy). The initial work of this research was prompted by recent papers e.g. by Morisio et
al. that seem to indicate new trends in using e.g. COTS components. We designed several
hypotheses and did a prestudy in 16 projects in Norway from Oct. 2003 to Feb. 2004 by
structured interview. The results of the prestudy showed that some of the hypotheses can
be tested and some of them are not easy to test. The results of the open questions in the
questionnaire also showed some new research questions and theses. We discovered that
all the hypotheses used in the prestudy and the new theses found from the prestudy are
relevant to risk-management in COTS-based development. We therefore change the
research topic to risk management issues and combine all the research questions in this
framework. After a literature survey on risk management in COTS-based development,
we also found some other interesting research questions. This empirical study is designed
to investigate all these research questions. We plan to design a questionnaire and use the
SESE web tool of the Simula Research Lab to collect the data.
Change Log:
Version 1.1: Added the research question relevant to build vs. buy decision;
Merged RQ2 into RQ3 and RQ4;
Made a clear fate between the prestudy and this design
Version 1.2: Revised the project level and OTS component level risks
Completed risk reduction questions
Added analysis methods
Version 1.3: Added software evolution questions to the questionnaire
Version 1.15 24-May-2004
1
-1:08 AM 08.03.2016
(also see appendix D)
English language checks & corrections (needs crosscheck)
Version 1.4: Integrated software evolution material from Appendix D in Ver1.3 into the
document, removed Appendix D.
Version 1.5: Add OSS relevant questions based on discussion with Macro.
Merge the questions of evolution and moved them into a separate part.
Version 1.6: Change the definition in the first page of the questionnaire.
Change the direction of the survey from COTS to OTS.
Version 1.7: Simplified questions on evolution and maintenance in section 6 of the
Questionnaire. - Reidar: Added v1.7 and 31.3 in line three.
Version 1.8: Reidar: new file name as cbse-survey-v1.8-2apr04.doc,
reverted many OTS terms to COTS: Have taken abstract and chs. 1-2 from
V1.5, chs. 3-4 from V1.7, References from V1.5, Questionnaire and
Appendices from V1.7 where I also have added hypotheses resume from
3 Feb.04 meeting. Revised questionnaire (two new pages in start, plus lots
of smaller changes) and Appendix B.
Version 1.9, 5apr04: Reidar: Lots of minor changes!
Please recheck questionnaire text vs. design
Version 1.10, 13apr04: Section 4.3 revised, the hypotheses vs. questionnaire questions
have been coordinated
Question 2.2 has been added to answer some OSS research question (see
chapter 3.2.1)
Question 4.2 is dropped, alternative 4.1.a and 4.1.b are added
Question 5.1 is divided into question 5.1, 5.2 and 5.3
Invitation letter is added in the first page of the questionnaire
Evolution question 6.1 – 6.4 have been revised
Version 1.11, 19apr04:
Change the definition of OTS component because Open Source Component
may not be free.
Added some examples of OSS components
Question 1.2.6, 1.2.7 were deleted. Question 5.1 was extended so that
respondents can fill in all components used in their project and their relative
importance.
Question 2.3 was added to ask the motivation of using COTS component
Question 5.3 was modified so that the OTS component type follows the
categories in ComponentSource (The biggest component market in the
world).
Question 5.7 was revised to make the alternative a) and b) more flexible.
Version 1.12, 26apr04:
Based on the comments we got from Tore Dybå and Christian Bunse, we
changed the version 1.11 a lot. The main purpose was to increase the
construct validity of the questionnaire. We current changed only the data
analysis chapter and questionnaire. The rest of this design will be changed
later.
Version 1.15 24-May-2004
2
-1:08 AM 08.03.2016
Question 1.2.5: Alternative “New functionality” was changed to easy to add
new functionality. Alternative “Improved functionality” was changed to
maintainability.
Question 2.1: Alternative “Better overall system quality” was divided into
three separate alternative to ask for “overall system reliability”, “overall
system security”, and “overall system performance”.
Question 2.3: Alternative “Reduction of the risk of security because source
code is not open” was added.
Question 3.4: deleted because the same question was asked in question 5.7
Question 4.1: changed dramatically. Several new alternatives were added.
The alternatives were clustered into different categories of risks (Need
future discussion).
Question 4.2: changed dramatically. Some new risk reduction methods were
added and they were clustered into 6 categories.
Question 5.1: The version of the OTS component was asked. We asked the
respond to fill in one of the most prominent component.
Question 5.2: It was separated into 5.2, 5.3, and 5.4. The scale was changed
into five-scale instead of Yes/NO. The number of the component was
reduced into one.
Question 5.3: changed into question 5.5. The classification was the same as
ComponentSource.
Question 5.4: changed to 5.6. The number of the component was reduced
into one.
Question 5.5: changed to 5.7. The number of the component was reduced
into one. The alternative Prestudy/Requirements/Analysis phase was divided
into two.
Question 5.6: changed to 5.8. The number of the component was reduced
into one. The scale was changed to five-scale.
Question 5.7: changed to 5.9. The number of the component was reduced
into one. The scale was changed to five-scale. The alternative b) and c) were
merged into d). alternatives g) h) i) were moved to question 4.2.
Question 5.10 was added to investigate the effect of the OTS component
Question 6.1: changed from pre-defined scale to the format that the
respondent can fill in the exact number.
Question 6.5: changed from pre-defined scale to the format that the
respondent can fill in the exact number.
Question 7.1.6 was added. Previous 7.1.6 moved to 7.1.7. Previous 7.1.7
moved to 7.1.8
Data analysis methods were revised:
Hypothesis New Hx2, H10, and possible PHx5 were revised. The variable
will be measured by using several items. The reliability and construct
validity testing methods were proposed.
Version 1.13 Deleted section 6 (on maintenance/evolution)
Deleted all meta-questions
Deleted all risk relevant hits in question 4.1,4.2,5.8.
Question 1.2.3:Added full description of terms in
Version 1.15 24-May-2004
3
-1:08 AM 08.03.2016
Question 2.2 Added alternative f)
Question 2.3 Added alternative e)
Question 3.2: Changed to multiple-selection
Question 4.1: One alternative was added, i.e., the risk of moving from
development environment to production environment
Alternative d) e) f) were revised so that they are relevant to
OTS components
Alternative a,b, c, k, and l were revised to use negative words
Question 4.2: Alternative a) change words to make it clear
Alternative k) was deleted
Question 5.2: Changed format
Question 5.3,5.4: Changed alternatives “hardly ever” to “never”.
Question 5.8, 5.9: Alternatives have been changed from 5-scales to Yes/No
Deleted the risk related words in the question
Other changes after discussion with Reidar Friday morning (21May)
Definition:
- OTS component definition was changed based on comments from
Macro
Component definition was changed
- Changed “executable unit” to “program unit”
- Changed “built in house” to “built from scratch”
- Platform definition was changed based on comments from Macro
Question 1.2.2: changed to less categories
Question 2.2: added alternatives “It was decided by customer” and “political
reasons”
Question 2.2 and 2.3: changed all the term “vendor” to “provider”
Question 4.3: changed the “background” to “role and OTS component
relevant experience”.
Question 5.1: slim so that only 5 component information was required
Open issues
Delete section 6 or not? Arguments:
- most information of the company can be acquired before the interview
start (by company list and internet)
- The personal information of the respond can be acquired in the
contacting procedure by calling them.
- Some respondents don’t want to give personal information by filling in
them into the questionnaire
Version 1.14 Revised invitation letter and term definition (Reidar)
The design and hypotheses have been cross checked with the questions
(Possible PHx1 was deleted because it was not possible to be tested and not
question can match it.)
Question 1.2.6 was moved to 1.1.2
1.1.2 to 1.1.3
1.1.3 to 1.1.4
Version 1.15 24-May-2004
4
-1:08 AM 08.03.2016
1.1.4 to 1.1.5
1.2.2 was changed to 1.2.3 (Include completed list again because it is
relevant to possible hypotheses PHx2 and PHx3)
1.2.3 to 1.2.4
1.2.4 to 1.2.5
1.2.5 to 1.2.2 (format reason)
2.1 – 2.3 question revised (past tense )
4.1 alternatives i,o,q were deleted to slim the question
4.2 slim the alternatives
4.1 alternative s and 4.2 alternative l were deleted because it is not
possible to answer others based on the question.
5.1 was added to know the whole number of different OTS components
5.1 to 5.2 deleted the last column, only importance of comp. 1 was asked
(see 5.3)
5.2 to 5.4
5.4 to 5.6
5.5 slimmed and changed to 5.7
5.6 to 5.8
5.7 to 5.9
5.8 to 5.10
5.9 to 5.11
5.10 to 5.12
Section 6.2 was deleted because all information can be gathered from
other resource
Open issues: 2.2 – 2.3 what is the meaning of political reasons?
V1.15, 24.5: Reidar - Polished initial definitions a bit more.
Fixed table-of-contents, reinserted 6.2 for the time being.
Version 1.15 24-May-2004
5
-1:08 AM 08.03.2016
Table of Contents
1. Research rationale ....................................................................................................... 7
1.1 Previous studies .................................................................................................... 7
1.2 Research motivation.............................................................................................. 8
1.3 Literature review ................................................................................................... 9
2. Results of the prestudy .............................................................................................. 10
3. Research Design........................................................................................................ 11
3.1 General design .................................................................................................... 11
3.2 Detailed design.................................................................................................... 12
3.2.1 RQ1: Motivation of using COTS/OSS components instead of building inhouse. (Exploratory study) ....................................................................................... 12
3.2.2 RQ2: Actual OTS-based process (Descriptive study) ..................................... 13
3.2.3 RQ3: Project level risks (Descriptive study) ................................................... 13
3.2.4 RQ4: OTS component-level risks (Descriptive study) .................................... 16
4. Questionnaire design ................................................................................................. 19
4.1 General questionnaire design .............................................................................. 19
4.1.1
The project background ............................................................................ 19
4.1.2
Motivation of using OTS components ...................................................... 19
4.1.3
The project process ................................................................................... 19
4.1.4
Project level risks ...................................................................................... 19
4.1.5
Project level risks management ................................................................ 20
4.1.6
The OTS components background............................................................ 21
4.1.7
OTS component-level risks....................................................................... 21
4.1.8
OTS component-level risk reduction methods ......................................... 22
4.1.9
The background information of the respondent ........................................ 22
4.2 Data analysis methods......................................................................................... 22
References: .................................................................................................................... 27
Appendix A. Final questionnaire .................................................................................. 29
Appendix B. Summary of hypotheses and theses in the prestudy ................................ 44
Version 1.15 24-May-2004
6
-1:08 AM 08.03.2016
1. Research rationale
1.1 Previous studies
The prestudy was prompted by recent papers e.g. by Morisio et al. [Torchiano04] that
seem to indicate new trends in using e.g. COTS components. We have proposed 10
hypotheses based on the theses proposed in that paper. We designed a questionnaire
which includes both closed and open questions. We did personal interviews based on this
questionnaire in 16 projects in 13 companies in Norway.
The results of the closed questions show that some of the hypotheses are possible to be
tested quantitatively by web-survey, but some of the others are not easy to test
quantitatively. The results are summarized in the internal report of the prestudy
[Conradi04].
One result of the open questions we discovered is a revised COTS-based development
process model in the following graph. From these open questions we also found some
new theses and research questions.
Non-COTS process
P1
Decide the whole
development process
(When and how? RQ1)
Make vs. Buy decision
(Why? RQ2)
P2
N
Start project level risk
identification and
evaluation (RQ3)
Decide to use COTS ?
Y
Start project level risk
mitigation (RQ3)
COTS selection
(When and how? RQ1)
P3
Start COTS level risk
management (RQ4)
N
Have found the proper COTS ?
Y
COTS integration
P4
COTS maintenance and evolution
Version 1.15 24-May-2004
7
-1:08 AM 08.03.2016
The revised common COTS-based development process has four phases, P1-P4:
 P1: Decide the whole development process (waterfall, prototyping, etc.) first.
 P2: In one phase of the development process (requirement, design, or
implementation), developers will decide whether they should buy a COTS component
from a third-party company or should build the component themselves. When to
decide this issue is different based on the whole development process and other
factors [Li04b]. In this phase, the main issue is to identify and evaluate the possible
risks between build and buy.
 P3: After the developers decided to buy COTS components from the market, they
should start to perform some risk reduction (mitigation) based on the risks they
identified. The next step is to search for and evaluate the COTS components. When to
select the COTS components is also different based on the developers experience
with various COTS components [Li04b].
 P4: After the COTS components have been selected. The next step is to integrate
them into the system.
This process is different from the COTS-based development process proposed by Boehm
et al. [Boehm99]. Their spiral model proposed to discover the risk first and decide the
waterfall or prototyping process later based on the possible risks [Boehm88]. Our process
proposes that the main processes are not decided by the COTS-related risks. They may be
decided by other possible risks first. After the project manager has decided to use COTS
components, the process may start to be COTS-related risk driven.
This process is similar to that proposed by Morisio et al. [Morisio00]. The difference is
that our results showed several possible variances [Li04b]. Morisio et al. pointed at some
possible risks in COTS-based development. Our results show that there are two levels of
risks in COTS-based development and each level may need a different kind of risk
reduction strategies. Before the developers decided to buy COTS components instead of
building them in-house, they must identify and compare the possible risks between the
two alternatives based on their project context. For example, they need to check if they
have such prior experience, and if their requirements are flexible enough etc. After they
have decided to buy COTS components, some project level risk reduction should be done
immediately, such as re-organizing the group to include experienced personnel. In the
COTS component selection procedure, they also need to do some risk management based
on the COTS component characteristics. For example, if they have no previous
experience with a specific COTS component, they may need to spend a lot of time on
understanding and testing it. On the other hand, they may not need much effort on this
issue if they have used the COTS components several times before.
1.2 Research motivation
The purpose of this study includes two parts:
- Further investigation of the hypotheses we tested in the prestudy
Version 1.15 24-May-2004
8
-1:08 AM 08.03.2016
- To study the above process model and related risks management issues we discovered
from the prestudy.
From the comments of we got from colleagues, we found that the main limitation of our
current study is that those hypotheses we tested in the prestudy are very independent, and
some of them have context validity problems because they look very obvious. However,
after a careful analysis of these hypotheses and the new findings we gathered from the
results of the open questions, we found that all of them are relevant to one common issue:
The risk management in the revised COTS-based process model we mentioned above.
We therefore decided to merge the two research purposes under the framework of risk
management in the COTS-based development process.
1.3 Literature review
We did a literature review of the previous studies on the risk management issues in
COTS-based development processes.
- Most previous studies are case studies that gathered a lot of different risks [Rose03],
[Kotonya01]. These studies proposed risks and strategies on how to resolve them.
 The first limitation is that most of them mixed COTS component level and project
level risks together.
 The second limitation is that there is no systematic summary on COTS component
characteristics and project contexts of these risks and risk management methods.
Their risk reduction strategies are case by case. If the developer just follows their
suggestions, it may not be a cost-effective way, for example, to use a very formal
evaluation process to evaluate a very familiar and unimportant COTS component.
 Another limitation is that there are too many proposed COTS component related
risks and the inter-relationship between these risks is not clear. That two or more
risks have an inter-relationship means that the occurrence of one risk might
trigger the occurrence of one or several other risks, which might not otherwise
occur. It is therefore hard for the developers to catch the real risks and make a
proper risk reduction plan. There is no factor analysis to see if some of them have
a real inter-relationship.
- The most influential study in this field was made by Boehm in the COCOTS project
[Abts00] [COCOTS00]:
 The first limitation of their research is that they assumed that every process using
COTS components should start with risk analysis. However, our previous result
concluded that the risk-driven process starts only after the project members have
decided to use a COTS component instead of building the component themselves.
 The second limitation is that they mixed the COTS component level and project
level risks together and calculated the possible cost by mixing them together. It
does not look reasonable from our prestudy results, because project level risks
may have more influence, since they are more general than COTS component
level risks, which are more specific.
Version 1.15 24-May-2004
9
-1:08 AM 08.03.2016


Another limitation is that they didn’t touch the risk reduction issues or analyze the
reduction strategies to see if they really worked.
Lastly, the empirical base mostly cover large companies in aerospace and defense,
- A major inspiration for this survey is the study by Torchiano and Morisio
[Torchiano04]. They interviewed seven small and medium sized software companies
in Norway and Italy. Their results are summarized in six theses, and challenge many
“assumed” truths about COTS-based development:
T1: Open source software is often used as closed source.
T2: Integration problems result from lack of compliance with standards;
architectural mismatches constitute a secondary issue.
T3: Custom code mainly provides additional functionalities (i.e. addware, not
glueware).
T4: Developers seldom use formal selection procedures. Familiarity with either the
product or the generic architecture is the leading factor in selection.
T5: Architecture is more important than requirements for product selection.
T6: Integrators tend to influence the vendor on product evolution whenever
possible.
However, the study covers few respondents, and there is a need for a more
representative study.
2. Results of the prestudy
We proposed a total of 10 hypotheses (See Appendix B) before, and tested them in the
prestudy [Conradi04] [Li04a]. Although we planned to delete 6 and continue 4 of them
initially, we decided to revise some of them, and change the delete/continue decision
based on the intention of our research. The revised hypotheses and theses are listed in
Appendix B. Hypotheses/theses relevant to risk management will be kept in this study, so
that they can be tested in the future.
The possible hypotheses that will be kept are (The same sequence number as Appendix
B): H01, H02, H04, and H10. New ones are Hx1, New Hx2.
Version 1.15 24-May-2004
10
-1:08 AM 08.03.2016
3. Research Design
3.1 General design
Based on our research purpose and the limitation of current studies, we generalized
several research questions as follows:
- RQ1: What are the main motivations of using OTS components instead of
building in-house?
There are several previous studies which mention the motivation of using OTS
components, such as shorter time-to-market, better quality. Our previous studies
pointed out that some projects are using OTS components because they just want to
keep up with new technology or follow some industry standard. So, it is important to
know the key motivation of using OTS components. The motivation may affect the
possible risks they may meet.
- RQ2: What are the real common OTS-based process models, and what are the
possible variances?
The rationale of this research question is to check whether the new OTS-based
process model [Li04b] we discovered in our prestudy is correct or not.
- RQ3: What project-level risks have been identified and their relationships with
the project contexts? Which risk reduction methods have been performed from
the project level and the results?
There are two levels of risks which need to be identified and mitigated in OTSbased projects: project level risks and OTS component-level risks. Project level risks
are high-level risks that are relevant to all components of the system. OTS
component-level risks are low-level risks that are relevant to only one specific OTS
component.
When the project managers decide to use OTS components instead of building inhouse, they need to start to identifying and evaluating the project-level risks in the
OTS-based projects. The possible risks are:
1. System requirement always change, the OTS component cannot keep up with
the requirement changes
2. Lack of skilled personnel in OTS component evaluation and integration
3. …
The second step in risk management is a risk reduction plan. Many project level
risk reduction methods have been proposed in the OTS-based development. For
example: Involving all stakeholders in the OTS component selection; Starting the
integration testing as early as possible, etc. There are few empirical studies on
Version 1.15 24-May-2004
11
-1:08 AM 08.03.2016
whether these methods can help mitigate the risks. It will be especially helpful if we
test these methods to see if the really helped to solve the risks. It can help the project
managers to design correct risk reduction strategies based on the risks they discovered.
- RQ4: What OTS component-level risks have been identified and their
relationships with OTS component characteristics? Which risk reduction
methods have been done for specific OTS components and the results?
In the process of OTS component selection, integration and maintenance, the
project managers may identify a lot of risks for specific OTS components, since
different OTS components may have different related risks. This is on a case-by-case
basis. The possible risks are:
1. Overly optimistic expectation of OTS component quality attributes
2. Inadequate provider support for a OTS component
3. Faulty provider claims about the functionality of a OTS component
4. …
In order to identify these risks in advance, we need to find the relationship
between them and the characteristics of specific OTS components, such as the
complexity of the interface, the market share of the provider, etc. After the risks for
specific OTS components have been identified, some proper risk reduction methods
should be performed to solve them. There are also many risk reduction methods
proposed for the OTS component level. For example, use formal selection methods to
select and evaluate OTS components, and use a wrapper to protect OTS components
etc. There are few empirical studies to test whether these methods really work or not.
Using these methods blindly may cost a lot.
3.2 Detailed design
3.2.1 RQ1: Motivation of using COTS/OSS components instead of
building in-house. (Exploratory study)
RC: check that all this is covered, especially for OSS issues??
What is the driving force (i.e. if it is cost-driven, schedule-driven, technology driven) for
the project to use COTS/OSS components? To what extend will COTS/OSS components
are expected to contribute to achieving such objection?
For this research question, we just want to know the distribution of motivation for using
COTS/OSS component. These motivations are the key factors in their build vs. buy
decisions. Motivations may have a relationship with risks they may meet in their project
and the corresponding risk reduction methods they use.
H_OSS_1: Most of the time OSS is used as closed source, i.e. source code is not
modified or even looked at.
Version 1.15 24-May-2004
12
-1:08 AM 08.03.2016
H_OSS_2: The main motivation for using OSS vs. COTS is reduction of the risk of
product evolving in an unwanted direction (which includes producer going out of
business, producer being acquired and changing market strategies, etc.)
H_OSS_3: Other motivation for choosing OSS are (in this order??):
- risk reduction (see H_OSS_2)
- philosophical (OSS process grants better quality)
- economical (OSS can be acquired at no cost)
- technical (OSS can be modified)
3.2.2 RQ2: Actual OTS-based process (Descriptive study)
For this research question, we need to answer the following sub-questions:

Is the main process undisturbed by OTS, with rather small variations? Was the main
process decided before/after OTS component decision?
 What main process was used: waterfall, incremental, prototyping …? Who decided
the main process: the customer, the developer, or the project manager?
 What process changes/activities were introduced in the development process, by
whom (developer, manager, customer) and when (in what phase: Requirements,
Design, Implementation):
- buy vs. build decision
- component selection?
- company policy
 What new roles were added/needed?
- e.g. knowledge keeper, ...
- person competence/experience
We gathered some theses and hypotheses for this research question. So, the main research
method is descriptive study (hypothesis testing).
The hypotheses relevant to this research question are:
 New Hx1: Using or not using OTS components has no relationship with main
processes used (Appendix B).
3.2.3 RQ3: Project level risks (Descriptive study)
Project managers may identify the risks in two phases: before the start of the project and
at the end of the project. In our study, we focus on the risks discovered at the end of the
project with some possible risk reduction, i.e. the area marked by “X” in the following
table.
Did not perform risk Performed risk reduction
reduction
Risks identified before the
project
Risks discovered during or
X
after the project
Version 1.15 24-May-2004
13
-1:08 AM 08.03.2016
To answer the research question, we need to answer three sub-questions:



RQ3-1: What project-level risks have not been discovered in the projects?
RQ3-2: What are the relationships between those project-level risks and the project
contexts?
RQ3-3: What risk reduction methods have been used, and whether these risk
reduction methods solved the corresponding risks?
We gathered some project-level risks that have been proposed by papers in the literature
survey [Rose03] [CBSEnet] [FAA03] [Boehm03a]. We categorized them based on
different issues: system requirement, human resource, cost estimation, etc.
Project level risks management performance
Dimension
Reference
PRisk 1: Because system requirements change continuously, Requirement [Rose03]
it is hard to change OTS components to keep up with these
causes OTS [Boehm03
changes.
component
a]
change
PRisk 2: Because system requirements are not flexible, it is OTS
[Rose03]
hard to negotiate the requirements with the customer even
component
[Boehm03
most-fit OTS components from the market cannot be found
causes
a]
requirement [CBSEnet]
to change
PRisk 3: Overly optimistic expectations about the effort of
Cost
[Rose03]
OTS component selection.
estimation
[CBSEnet]
in
OTS
component
selection
PRisk 4: Overly optimistic expectations about the effort of
Cost
[Rose03]
OTS component integration and integration testing.
estimation
[CBSEnet]
in
OTS
component
integration
PRisk 5: The performance/reliability/security of the
Quality of [Rose03]
integrated system is worse than expected because of
integrated
[Boehm03
interoperability problems between OTS components.
system.
a]
[CBSEnet]
PRisk 6: When a defect is identified, it may not be very
Locate and [Rose03]
obvious where the cause of the problem is located (inside or report
outside OTS components).
defects
PRisk 7: Different OTS components have different upgrade OTS
[Rose03]
lifecycles. It is difficult to plan the maintenance activities in component
[CBSEnet]
advance.
change
coordination
For sub-question RQ3-1, we just want to calculate the distribution of risks.
Version 1.15 24-May-2004
14
-1:08 AM 08.03.2016
For the sub-question RQ3-2 (i.e. relationship risks and project contexts), we designed the
following possible hypotheses based on our literature survey:

Possible PHx2: There is no relationship between the number of OTS
components used in the project and the cost-estimation related risk (PRisk 3,
4) occurrences [COCOTS00] [Boehm03a].

Possible PHx3: There is no difference in requirement causes OTS component
change risk (PRisk 1) occurrences between different domains [Ropponen00].

Possible PHx4: There is no difference in OTS component causes requirement
to change risk (PRisk 2) occurrences between different domains
[Ropponen00].
For the sub-question RQ3-3 (i.e. the effect of risk reduction methods), we would like to
suggest that the proposed risk reduction methods can solve the corresponding risks.
We gathered some possible project-level risk reduction methods as follows:
Project-level risks
PRisk 1,2
PRisk 4
PRisk 4
PRisk 5
PRisk 6
PRisk 6
Project-level risk reduction methods
Involve users in the requirement and OTS
component selection phase.
Select OTS components mainly based on
integration easiness (architecture and
standard match) instead of expected
functionality
A most likely OTS component (package)
learning curve must be accounted during
plan and schedule
Significant OTS component quality
features must be tested
First integrate the OTS components that
you had no previous experience
Do integration testing immediately after
an OTS component has been integrated
References
[FAA03]
[Rose03]
[Li04b]
[Boehm03a]
[Boehm03a]
[Rose03]
[Rose03]
The possible Null hypotheses are: There is no relationship between whether risks may
happen and whether the corresponding risk reduction methods have been used. For
example:

New Hx2: There is no relationship between OTS component causes requirement
to change risk occurrences (PRisk 2) and customer involvement (Appendix B)

H10: OTS component integration risks (PRisk 4) and selection strategy
(Appendix B)
Version 1.15 24-May-2004
15
-1:08 AM 08.03.2016

Possible PHx5: There is no relationship between different project-level risks and
corresponding risk reduction methods used. (It actually includes 7 independent
hypotheses. We will test the relationship between PRisk1-PRisk 7 and their
corresponding proposed risk reduction methods)
3.2.4 RQ4: OTS component-level risks (Descriptive study)
Most research questions and hypotheses in the prestudy focus on this part. To
investigate this research question, we need to answer two sub-questions:
 RQ4-1: What are the relationship between the possible OTS component-level risks
and OTS component characteristics?
 RQ4-2: What are the results of using OTS component-level risk reduction methods?
The OTS component-level risks are those low-level risks that are different between
different OTS components in the same project. The possible OTS component level risks
are as following:
OTS component-level risks
Dimension
CRisk 1: Faulty provider claims about the standard/ Standard
version of standard that OTS component follows
mismatch
Reference
[Torchiano04]
CRisk 2: Components have different architecture
assumption either between component to component or
between components to application.
CRisk 3: OTS component can not provide all expected
functionality
CRisk 4: OTS component is not backward compatible.
(i.e. new version did not support the functionality of the
old version)
CRisk 5: Large OTS component provider companies do
not usually allow negotiation about their products.
Middle size companies provide the best support. OSS
provides better communication between developers and
integrators.
Architectural
mismatch
[Torchiano04]
Functionality
deficiency
Maintenance
[Torchiano04]
Provider support
++ later
[CBSEnet]
The characteristics of an OTS component include:
1. OTS component full name
2. OTS component origin (COTS or OSS)
3. OTS component type (COM, CORBA, EJB, .Net, Web Service, Library etc.)
4. OTS component functionality
5. Experience with the OTS component
6. Whether the source code has been modified or not
The possible hypotheses are:
Version 1.15 24-May-2004
16
-1:08 AM 08.03.2016

H01: standard mismatch risk / architecture mismatch risk (CRisk 1 and 2)
and OTS component type (Appendix B)

H02: Glueware (CRisk 1 and 2) vs. addware (CRisk 3) and OTS component
type (Appendix B)
Glueware is needed solve both standard mismatch (crisk1) and architecture
mismatch (CRisk 2).
Because of time limitation, we will ask the respondent to select only max. 3 OTS
components, which they are familiar with and used in the projects
To answer the second sub-question, we identified many risk reduction methods in the
OTS component level, such as:
OTS
component- OTS component-level risk reduction
References
level risks
methods
CRisk 1:
Hand-on trial beyond literature review or [Boehm03a]
provider provides documentation
CRisk 2,3:
Identify the core requirement;
[Boehm03a]
Filter all the candidates;
[Li04b]
Proceed for a more detailed assessment
with the remaining OTS component
candidates
CRisk 2,3:
Involve OTS component knowledgeable
[Li04b]
individuals in OTS component selection,
[Torchiano04]
design, integration
CRisk 2,3:
Selecting evaluation criteria (factors)
[ComellaCollecting and assigning values to these
Dorda02]
criteria
Making decision using formal decision
making methods, such as MAUT or
MCDA
CRisk 2,3:
Select unfamiliar OTS components in the [Rose03]
early phase
[Li04b]
[Moriso00]
CRisk 2,3:
Add glueware/addware
[Torchiano04]
CRisk 4:
Have a senior person to follow the OTS
[Rose03]
component’s future updates and the
[Li04b]
possible effects
CRisk 4:
Maintain a continual watch on the market [Rose03]
(newsgroup, OSS, cooperate with
[Conradi04]
knowledgeable persons) and look for
substitute products that might replace an
old one
CRisk 5:
Have research (newsgroup, OSS,
[Rose03]
cooperate with knowledgeable persons)
on provider support ability and reputation.
Version 1.15 24-May-2004
17
-1:08 AM 08.03.2016
The possible are:

H04: The frequency of using familiarity process is more than the frequency of
using formal process (Appendix B)

Possible PHx6: There is no relationship between the phase of OTS component
selection and the project members’ experience with the OTS components [Li04b].
Version 1.15 24-May-2004
18
-1:08 AM 08.03.2016
4. Questionnaire design
4.1 General questionnaire design
The questionnaire will include the following sections:
4.1.1 The project background
-
The domain
The development language
The development effort
How large a percentile of functionalities is provided by OTS components vs.
functionalities provided by application
- How many OTS components were used in the project
- How many developers have experience with OTS-based development
- The constrains on the project (time-to-market, reliability, security, …)
4.1.2 Motivation of using OTS components
- What is the driving force (i.e. if it’s cost-driven, schedule-driven, technology
driven) for the project to use COTS/OSS components?
- To what extent will COTS/OSS components be expected to contribute to
achieving these objectives?
4.1.3 The project process
- What main process: waterfall, incremental, prototyping …? Who decided the
main process: customer, developer, or project manager?
- Was the main process decided before/after the OTS component decision?
- When did they do the buy vs. build decision and component selection (in what
phase: Requirements, Design, coding)?
- What new roles were added / needed? - e.g. knowledge keeper, (Dedicated or not
dedicated?)
4.1.4 Project level risks
What is your opinion on the following aspects of your project?
Corresponding risk
a) The project was not completed according to schedule
PR3
b) Effort to select OTS components were not satisfactorily PR3
estimated
Version 1.15 24-May-2004
19
-1:08 AM 08.03.2016
c) Effort to integrate
OTS components were not
satisfactorily estimated
d) OTS components negatively affected system reliability
e) OTS components negatively affected system security
f) OTS components negatively affected system
performance
g) Requirements were continuously changed
h) OTS components could not be sufficiently adapted to
changing requirements
i) Project could not (re)negotiate requirements with the
customer, if OTS components could not satisfy all
requirements
j) It was difficult to identify whether defects were inside or
outside the OTS components
k) It was difficult to plan system maintenance, because
different OTS components had asynchronous release cycles
l) It was difficult to update the system with the last OTS
component version
m) OTS components were not satisfactorily compatible
with production environment when the system was
deployed
n) Information on the reputation and technical support
ability of provider were inadequate
o) Provider did not provide enough technical support/
training
PR4
PR5
PR5
PR5
PR1
PR1
PR2
PR6
PR7
PR7
PR6
Provider related risk
Provider related risk
4.1.5 Project level risks management
What actions had been performed in the development or maintenance process of the
project?
Corresponding risk
a) Customer had been actively involved in making the PR1,2
“acquire” vs. “build” decision
b) Customer had been actively involved in OTS component PR 1,2
selection
c) OTS components were selected mainly based on PR 5
architecture and standards compliance, instead of
expected functionality
d) OTS components qualities (reliability, security etc.) were PR 5
seriously considered in selection
e) Effort in learning OTS component was seriously PR 3
considered in effort estimation
f) Effort in black-box testing OTS component was PR 4
seriously considered in effort estimation
Version 1.15 24-May-2004
20
-1:08 AM 08.03.2016
g) Unfamiliar OTS components were integrated first
h) Did integration testing incrementally (after each OTS
component was integrated)
i) Local OTS-experts followed the OTS component
updates and possible consequences actively
j) Maintained a continual watch on the market and looked
for possible substitute components
k) Maintained a continual watch on provider support
ability and reputation.
PR 6
PR 6
PR 7
Provider related risk
Provider related risk
4.1.6 The OTS components background
OTS component ID
Comp.1
OTS component full name
OTS component main functionality
OTS component origin (COTS or OSS)
Source code had been changed or not
OTS component model (COM, CORBA, EJB, .Net, Web
service, Library etc.)
Experience with the OTS component
Impact on your project/process (when the component was
selected)
4.1.7 OTS component-level risks
Comp.1
a) The OTS component did not follow the
industry standard as provider claimed
b) OTS component have different architecture
assumption with other component in the system
c) OTS component cannot provide all expected
functionality
d) OTS component’s new version is
incompatible with the old one
e) Provider does not want to change the OTS
component functionality as we required
Version 1.15 24-May-2004
Risk
CRisk 1
CRisk 2
CRisk 3
CRisk 4
CRisk 5
21
-1:08 AM 08.03.2016
4.1.8 OTS component-level risk reduction methods
a) Searched Internet for possible OTS component candidates.
b) Got recommendation of possible OTS component
candidates from customer
c) Got recommendation of possible OTS component
candidates from local colleague/OTS-expert
d) Assigned values to selected evaluation criteria, calculated a
number using formal decision making algorithm/method to
compare possible candidates
e) Limited possible candidate into 1-3 components and
investigated them locally by reading literature or other
documentation
f) Did “hands-on” try-out of 1-3 components, e.g. on a downloaded demo version.
Corresponding risks
CRisk 4, 5
CRisk 4, 5
CRisk 1, 2,3
CRisk 1,2,3
CRisk 1,2,3
CRisk 1,2,3
4.1.9 The background information of the respondent
-
The role in the project
The experience with OTS-based projects
The experience with software development
The education background
4.2 Data analysis methods

New Hx1: Using or not using OTS components has no relationship with main
processes used (Appendix B).

Corresponding question
Question 3.3: when was the main development process decided

Analysis method
T-test: to compare the differences between two options

Sample size

Reliability and validity issues??

Possible PHx2 (May be a research question instead of a hypothesis): There is no
relationship between the number of OTS components used in the project and the
cost-estimation related risk (PRisk 3, 4) occurrences [COCOTS00] [Boehm03a].

Corresponding question
Question 5.1 and 4.1 (alternative a, b and c, i.e. cost-estimation related risk)

Analysis method
Two variables: Number of components used in the project (Interval)
Version 1.15 24-May-2004
22
-1:08 AM 08.03.2016


Cost-estimation related risk occurrences (Ordinal)
F-test or just describe the distribution
Sample size
Reliability and validity issues

Possible PHx3 (May be a research question instead of a hypothesis): There is no
difference in requirement causes OTS component change risk (PRisk 1)
occurrences between different domains [Ropponen00].

Corresponding question
Question 1.2.3 and 4.1 (alternative h): domain vs. adapt component

Analysis method
Two variables: Different domains (Nominal)
Requirement causes OTS component change risk (Ordinal)
Statistical Test
Kruskal-Wallis One-way ANOVA (If the distribution of the result doesn’t
follow the required distribution, we just describe the distribution)

Sample size

Reliability and validity issues

Possible PHx4 (May be a research question instead of a hypothesis): There is no
difference in OTS component cause requirement to change risk (PRisk 2)
occurrences between different domains [Ropponen00].

Corresponding question
Question 1.2.3 and 4.1 (alternative l): domain vs. adapt requirement

Analysis method
Two variables: Different domains (Nominal)
OTS component cause requirement change risk (Ordinal)
Statistical Test
Kruskal-Wallis One-way ANOVA (If the distribution of the result doesn’t
follow the required distribution, we just describe the distribution)

Sample size

Reliability and validity issues

New Hx2: There is no relationship between requirement management risk
occurrences and customer involvement (Appendix B)

Corresponding question
Question 4.1(alternative h,i) and 4.2 (alternative a,b): requirement management
risk vs. customer involvement

Analysis method
Two variables: requirement management risk(Ordinal)
Customer involvement (Ordinal)
Statistical Test
Spearman correlation Test

Sample size
Version 1.15 24-May-2004
23
-1:08 AM 08.03.2016

Reliability: Alpha analysis will be used to test the reliability of variable
requirement management risk (i.e. to see if alternatives h, and i in 4.1 are
consistent)

Construct Validity: Factor analysis will be used to test the alternatives h
and i in 4.1.

H10: OTS component quality risks and selection strategy (Appendix B)

Corresponding question
Question 4.1 (alternatives d,e,f) and 4.2 (alternative c,d)

Analysis method
Two variables: quality risks (Ordinal)
Selection strategy (Ordinal)
Statistical Test
Spearman correlation Test

Sample size

Reliability: Alpha analysis will be used to test the reliability of variable
quality risk (i.e. to see if alternatives d, e, and f in 4.1 are consistent)

Construct Validity: Factor analysis will be used to test the alternatives d, e,
and f in alternative 4.1.

Possible PHx5: There is no relationship between different project-level risks and
corresponding risk reduction methods used. (It actually includes several
independent hypotheses. We will test the relationship between PRisk1-PRisk 7
and their corresponding proposed risk reduction methods, see chapter 4.1.4. and
4.1.5)

Corresponding question
Questions 4.1 and 4.2

Analysis method
Two variables: Specific project-level risk occurrences (Ordinal)
Frequency of corresponding risk reduction methods used (Ordinal)
Statistical Test
Spearman correlation Test

Sample size

Reliability: Alpha analysis will be used to test the reliability of variable,
which is corresponding risk (similar to H10).

Construct Validity: Factor analysis will be used to test the alternatives in
4.1 (similar to H10).

H01 (May be a research question instead of a hypothesis): standard mismatch
risk / architecture mismatch risk (CRisk 1 and 2) and OTS component type
(Appendix B)

Corresponding question
Question 5.7 and 5.10 (alternatives a and b): architecture/standard mismatch vs.
component type
Version 1.15 24-May-2004
24
-1:08 AM 08.03.2016

Analysis method
Two variables: Integration risk occurrences (Nominal)
OTS components type, i.e. COM, CORBA, EJB, .Net or Library
(Nominal)
Statistical Test
Distribution
(Because the distribution of OTS component type may not follow required
discribution, we just describe the distribution instead of hypothesis testing)

Sample size

Reliability and validity issues

H02 (May be a research question instead of a hypothesis): Glueware (CRisk 1
and 2) vs. addware (CRisk 3) and OTS component type (Appendix B)
- Corresponding question
Question 5.7 and 5.10 (glueware needed, i.e. alternative a in 5.10 and alternative
b in 5.10 vs. addware needed, i.e. alternative c in 5.10) : glueware/addware vs.
component type

Analysis method
Two variables: Custom code related risk occurrences (Nominal)
OTS components type, i.e. COM, CORBA, EJB, .Net or Library
(Nominal)
Statistical Test
Distribution
(Because the distribution of OTS component type may not follow required
discribution, we just describe the distribution instead of hypothesis testing)

Sample size

Reliability and validity issues

H04: The frequency of using familiarity process is more than the frequency of
using formal process (Appendix B)

Corresponding question
Question 5.11 (select both alternative d are formal process) and 5.11 (select
alternative e or f is familiar process): formal process vs. familiar process

Analysis method
Two variables: The frequency of formal process used (Interval)
The frequency of familiar process used (Interval)
Statistical Test
T-test

Sample size

Reliability and validity issues??

Possible PHx6: There is no relationship between the phase of OTS component
selection and the project members’ experience with the OTS components [Li04b].

Corresponding question
Question 5.8 and 5.9

Analysis method
Version 1.15 24-May-2004
25
-1:08 AM 08.03.2016


Two variables: The phase of OTS component selection (Ordinal)
The project members’ experience with OTS component (Ordinal)
Statistical Test
Spearman
Sample size
Reliability and validity issues??
H_OSS_1 (May be a research question instead of a hypothesis): Most of the time OSS
is used as closed source, i.e. source code is not modified or even looked at.

Corresponding question
Question 5.5, 5.6
H_OSS_2 (May be a research question instead of a hypothesis): The main motivation
for using OSS vs. COTS is reduction of the risk of product evolving in an unwanted
direction (which includes producer going out of business, producer being acquired and
changing market strategies, etc.)

Corresponding question
Question 2.2
H_OSS_3 (May be a research question instead of a hypothesis): Other motivation for
choosing OSS are:
- risk reduction (see H_OSS_2)
- philosophical (OSS process grants better quality)
- economical (OSS can be acquired at no cost)
- technical (OSS can be modified)

Corresponding question
Question 2.2
Version 1.15 24-May-2004
26
-1:08 AM 08.03.2016
References:
[Abst00] Chris Abst, Barry W. Boehm, and Elisabeth B. Clark, “COCOTS: A COTS
Software Integration Lifecycle CostModel - Model Overview and Preliminary Data
Collection Findings", Technical report USC-CSE-2000-501, USC Center for Software
Engineering, 8 March 2000, 10 p. Available on:
http://sunset.usc.edu/TechRpts/Papers/usccse2000-501/usccse2000-501.pdf.
[Boehm88] Barry W. Boehm, “A Spiral Model of Software Development and
Enhancement”, IEEE Computer, 21(5):61-72, May 1988.
[Boehm99] Barry W. Boehm and Chris Abts, “COTS integration: Plug and Pray?” IEEE
Computer, 32(1):135-138, Jan. 1999.
[Boehm03a] Barry W. Boehm, Dan Port, Ye Yang, and Jesal Bhuta, ” Not All CBS Are
Created Equally COTS-intensive Project Types”, Proc. Second International Conference
on COTS-Based Software Systems (ICCBSS’03), Ottawa, Canada, February 10-13, 2003,
Springer Verlag LNCS 2580, ISBN 3-540-00562-5, pp. 36-50.
[Boehm03b] Donald J. Reifer, Victor R. Basili, Barry W. Boehm, and Betsy Clark,
“Eight Lessons Learned in COTS-based Systems Maintenance”, IEEE Software,
20(5):94-96, Sept./Oct. 2003. RC: p.t. not used??
[CBSEnet] CBSE Work Package WP4 D8.2.1 – COTS Chapter. Available on
www.cbsenet.org. Add more detail??
[COCOTS00] Chris Abts, Barry W. Boehm, and Elizabeth Bailey Clark, “COCOTS: A
COTS Software Integration Lifecycle Cost Model- Model Overview and Preliminary
Data Collection Findings”. Technical report: USC-CSE-2000-501. Available on:
http://sunset.usc.edu/publications/TECHRPTS/2000/usccse2000-501/usccse2000-501.pdf.
[Comella-Dorda02] Santiago Comella-Dorda, John C. Dean, Edwin Morris, and Patricia
Oberndorf, “A Process for COTS Software Product Evaluation”, Proc. First International
Conference on COTS-Based Software Systems (ICCBSS’02), Orlando, FL, USA,
February 4-6, 2002, Springer Verlag LNCS 2255, ISBN 3-540-43100-4, pp. 176-187.
[Conradi04a] Reidar Conradi, Jingyue Li, Finn Olav Bjørnson, Dag Sjøberg, and Vigdis
By Kampenes, “A State-of-the-Practice Survey on COTS component Based Development
in IT Companies in Norway” (prestudy report). NTNU, Internal report, 27 Feb. 2004, 61
p. Available on
http://www.idi.ntnu.no/~epos/cots/public_html/COTS_survey_report_internal_version.do
c.
Version 1.15 24-May-2004
27
-1:08 AM 08.03.2016
[FAA03] Federal Aviation Administration, Software Engineering Resource Center,
“FAAA COTS Risk Mitigation Guide: Practical Methods For Effective COTS
Acquisition and Life Cycle Support”, 5 March 2003, 113 p. Available on
http://www.faa.gov/aua/resources/cots/Guide/CRMG.htm.
[Kotonya01] Gerald Kotonya and Awais Rashid, “A Strategy for Managing Risk in
Component-based Software Development”, 27th EUROMICRO Conference 2001: A Net
Odyssey, 4-6 Sept. 2001, Warsaw, Poland. IEEE CS Press, ISBN 0-7695-1236-4, pp. 1221.
[Li04a] Jingyue Li (Ed.), “Summary of COTS hypotheses & theses”, NTNU, Internal
report, 4 March 2004, 4 p. Available on
http://www.idi.ntnu.no/~epos/cots/pub/Summary-of-hypotheses-04March.doc.
[Li04b] Jingyue Li, Finn Olav Bjørnson, Reidar Conradi, and Vigdis By Kampenes, “An
Empirical Study of Variations in COTS-based Software Development Processes in
Norwegian IT Industry”, NTNU, 25 March 2004, 12 p. Draft paper submitted to
Metrics’2004. Available on
http://www.idi.ntnu.no/~epos/cots/public_html/An_empirical_study_in_variances_of_C
OTS_based_development.pdf (prev. version!).
[Morisio00] Maurizio Morisio, Carolyn B. Seaman, Amy T. Parra, Victor R. Basili, Steve
E. Kraft, and Steven E. Condon, “Investigating and Improving a COTS-Based Software
Development Process”, Proceeding of 22nd International Conference on Software
Engineering (ICSE’00), Limerick, Ireland, IEEE CS Press, June 2000, pp. 31-40.
[Ropponen00] Janne Ropponen and Kalle Lyytinen, “Components of Software
Development Risk: How to Address”, IEEE Transactions on Software Engineering,
26(2):98-111, February, 2000.
[Rose03] Louis C. Rose, “Risk Management of COTS based System development”, In
Alejandra Cechich, Mario Piattini, and Antonio Vallecillo (Eds.): Component-Based
Software Quality - Methods and Techniques. Springer Verlag LNCS 2693, ISBN 3-54040503-8, 2003, pp. 352-373.
[Torchiano04] Marco Torchiano and Maurizio Morisio, "Overlooked Facts on COTSbased Development", IEEE Software, 21(2):88-93, March/April, 2004.
Version 1.15 24-May-2004
28
-1:08 AM 08.03.2016
Appendix A. Final questionnaire
Invitation letter (in English) ++ email/postal mail address??
Dear industrial colleague!
NTNU in Trondheim and the Simula Research Laboratory in Oslo are jointly participating in the research
projects SPIKE (on process improvement) and INCO (on incremental/component-based system
development). As part of this research, we wish to perform a questionnaire-based survey in Norwegian
ICT industry on OTS-based system development (Off-The-Shelf, including Open Source Software and
Commercial Off-The-Shelf Software). The subject is highly relevant for both research and industry.
We have already had a test run with a trial questionnaire in August 2003 with 6 persons, and 16 structured
interviews in November 2003 - January 2004. We now want to expand the survey to a larger part of the
Norwegian ICT sector, and to perform parallel surveys in cooperation with research colleagues at IESE in
Kaiserslautern in Germany and Politecnico di Torino in Italy. The revised questionnaire has now been
tested on about 15 persons in April-May 2004 in these three countries.
We now want to start the main survey, involving over 200 ICT companies (large, medium, and small) in
each country. We therefore look for software developers in your company with hands-on experience with
OTS-based system development. In this request, we attach an electronic version of the questionnaire as a
word file. We will also contact you by phone. If you or a colleague volunteer to participate in this survey,
we will send you (called a “respondent”) a userid and password to access a web-based version of the
questionnaire. Although we prefer that you fill in the web-version, you can also fill in the questionnaire on
paper (after printing out the word file) and return it by postal mail, or fill in the word file electronically and
return to us as an email attachment.
All provided information will be treated strictly confidential. A summary of the results will be returned to
the respondents as a technical report, and by a web file. Respondents will also be invited to a public sumup seminar to discuss the findings. We will also arrange a lottery among the respondents, where the
winner will receive a travel of 10 000 NOK. Two PhD fellows at NTNU, Jingyue Li and Odd Petter N.
Slyngstad, will be responsible for carrying out the survey. The main responsible in Norway is prof. Reidar
Conradi at NTNU.
The questionnaire is divided into 6 sections, where section 1 and 6 contain questions regarding
context/background, and sections 2-5 ask for information more specific to your OTS-based project. There
are ca. 40 question in total, and most are multiple-choice questions. This final version of the questionnaire
will require 20-30 minutes to fill out.
So, would you be willing to participate in such a survey?
As mentioned above, we attach an electronic copy of the questionnaire. If you or a colleague volunteer to
participate in the survey, we will in a few days send you a user login and password to access the webversion.
- Best regards, Reidar
Version 1.15 24-May-2004
29
-1:08 AM 08.03.2016
Please read carefully before completing the questionnaire
Survey
Objective
Coverage
Confidentiality
Language
Benefits
Some general
instructions
The purpose of this questionnaire is to survey the current state of
practice in component based software engineering (CBSE) in various
organizations and projects, involving the ICT companies in several
countries.
The background we want to measure, are hands-on experience
with CBSE-development in concrete and completed projects,
preferably with a maintenance phase afterwards. We want
answers based on concrete experience (by your team etc.), not
more general company information or a private/textbook opinion
on CBSE.
This questionnaire deals with reuse of external components, i.e.
COTS or OSS components (see definitions later).
The responses of this questionnaire will be handled confidentially.
Whether a company has participated is also confidential.
The conclusion of this survey provides only the current state of
practice in CBSE in the ICT industry. Detailed information of
each company will not be included in the final report. It is not
possible to draw conclusions about your company from the
research results.
You are expected to answer all the questions in your native language,
but you may also respond in English.
The final report of the survey will be available to all respondents, and
sent by email or through a web-link. A hard copy of the report can be
sent if a post address is provided. In addition, a public, sum-up
seminar will be arranged.
Use an “X” when you mark an answer to a question, e.g. in a
table. Filling-in of the questionnaire should not take more than
half an hour.
Name of respondent: Last Name (Family Name)
Middle Name
First Name
Age:
Gender (Male/Female):
Email address:
Phone:
Fax:
Name of the Company /Business unit:
Postal address of Company/Business unit:
Postal
Town
Country
Code
Date of response
Time used to fill in this questionnaire (in minutes):
(mm/dd/yy):
NB: The above information will be treated confidentially.
Version 1.15 24-May-2004
30
-1:08 AM 08.03.2016
Our definition of concepts used in this questionnaire:
Project: A software development project, in which the respondent has been involved.
We aim for completed projects, possibly with maintenance, and possibly with several releases.
Customer: Representing either an end-user, or a main contractor to which your organization is a
subcontractor.
Component: Software components are program units of independent production, acquisition, and
deployment that can be composed into a functioning system. We limit ourselves to components that
have been explicitly decided to either be built from scratch, or to be acquired externally as an OTScomponent. That is, to components that are not shipped with the operating system, not provided by
the development environment, and not included in any pre-existing platform. That is, platform
(“commodity”) software are not considered, e.g. an OS like Linux, DBMSes, various servers, or
similar software. Furthermore, components usually follow some component model, as expressed by
the COM, CORBA, EJB, .Net, or Web Service standards, or they can be a C++/Java library.
OTS (Off-The-Shelf) component: A component that:
 is provided (by a so-called provider) from:
- a commercial vendor (Commercial-Off-The-Shelf, COTS), or
- the Open Source community (Open Source Software, OSS).
The components may come with certain obligations, e.g. payment or licensing terms.
NB: we do not consider proprietary components developed by some other organization in
the same company, i.e. internal reuse.
 is not controllable, in terms of provided features and their evolution.
 is used as closed source, i.e. no source code is modified, even it is available.
Typical COTS components are: Vector draw professional (3D modeling), Shamman Reports for
Excel (Database reporting), Dynamic Cube (OLAP) etc.
Typical OSS components are: Xerces (XML parser), Kiwi Toolkit (Class library), OpenEJB,
Python (Interpreter) etc.
System: Product delivered by the actual project, consisting of application plus components, but again
excluding platform software.
Application: Code built in-house to provide certain functionalities of the system.
Glueware: Code to solve possible mismatch between components, or between components and
application/platform.
Addware: Code to add the functionalities which are expected but not provided by the components.
The relationship between the four last concepts is shown in the following figure:
Question for comments.Can you give an example of OTS components based on your understanding?
_____________________________________________________________________
Version 1.15 24-May-2004
31
-1:08 AM 08.03.2016
Section 1: Questions on the context of the project and system
1.1 Information about the project
The projects we want to investigate in this survey are completed software development
projects using one or more OTS components. If you have experience with several such
projects, please select the one that you are most familiar with, and base your answers
on that project.
1.1.1 What was the total staff size of the project (full-time and part-time persons)?
Full time: _________________
Part time: _________________
1.1.2 How large a percentage of the project members have previous experience with
general OTS-based development __ (%)?
1.1.3 What was the starting time of the project (mm/yy)
/
?
1.1.4 What was the time of first complete delivery from the project (mm/yy) ___/____?
1.1.5 What was the development environment of the project (describe textually)?
 Programming Languages: ____________________
 CASE (Computer Assistant Software Engineering) tools: _______________
1.2 Information about the system
1.2.1 What were the main functionalities of the system (describe textually)?
_____________________________________________________________________
_____________________________________________________________________
1.2.2 What was the emphasis on the following characteristics of the system?
[Mark one alternative per row]
Very
Low Medium
High
Very
Don’t
low
high
know
a) Time-to-market
b) Effort (cost)
c) Reliability
d) Security
e) Performance
f) Maintainability
g) New functionality (first
launch in the market)
h) Improved functionality
(over competitors)
i) Other (please specify):
___________________
Version 1.15 24-May-2004
32
-1:08 AM 08.03.2016
1.2.3 What were the major types of industry / services (application domain) of the
customers of the system?
[Mark one or more alternatives]
□ Bank/ Finance/ Insurance
□ Other private services (wholesale, retail, entertainment, …)
□ Public sector
□ ICT industry
□ Traditional industry/engineering/construction
□ Other (please specify): _______________
1.2.4 What is the executing environment of the system (describe textually)?
 OS (Operating System):_____________________________________
 DBMS (Database Management System): _______________________
 Other Supporting Software: __________________________________
1.2.5 What is the current version/release of the system (Version/Release) ___/___?
Version 1.15 24-May-2004
33
-1:08 AM 08.03.2016
Section 2: Questions on the motivation of using OTS components
2.1 To what extent did you expect, that the chosen OTS component(s) would contribute
to achieve the following project/system targets?
[Mark one alternative per row]
Don’t Hardly Agree Agree Stron Don’t
Project/system targets
agree
agree
some
mostly gly
know
at all
what
agree
a) Shorter time-to-market
b) Less development effort/cost
c) Less maintenance effort/cost
d) Larger market share
e) Compliance with industrial
standards
f) Keeping up with the newest
technology
g) Better system reliability
h) Better system security
i) Better system performance
j) Other (please specify):
________________________
2.2 What were the specific motivations of using OSS Component(s) in your project?
[NB: If you did not use a OSS Component in the project,please skip this question]
[Mark one alternative per row]
Don’t Hardly Agree Agree Strong Don’
Motivations
agree
agree
some
mostly ly
t
at all
what
agree
kno
w
a) Reduce risk of provider going
out of business
b) Reduce risk of provider
changing market strategies
c) Reduce risk of selected
component evolving into an
unwanted direction
d) Component can be acquired
for free
e) Source code is available and
can easily be changed
f) It was decided by customer
g) Political reasons
h) Other (please specify):
______________________
Version 1.15 24-May-2004
34
-1:08 AM 08.03.2016
2.3 What were the specific motivations of using COTS Component(s) in your project?
[NB: If you did not use a COTS Component in the project,please skip this question]
[Mark one alternative per row]
Don’t
Motivations
agree
at all
Hardly Agree
agree
some
what
Agree
mostly
Stro
ngly
agre
e
Don’t
know
a) Reduce risk of bad quality,
because components were paid
for
b) Reduce risk of poor security,
because source code was closed/
unavailable
c) Provider could give adequate
technical support
d) Provider may have investigated
the market trends, and will give
a component following them
e) It was decided by customer
f) Political reasons
g) Other (please specify):
__________________________
Version 1.15 24-May-2004
35
-1:08 AM 08.03.2016
Section 3: Questions on the general development processes
3.1 Please describe the main development process (e.g. waterfall, prototyping,
incremental etc. – describe textually):
____________________________________________________________
____________________________________________________________
3.2 Who decided the main (i.e. lifecycle) process?
[Mark one or more alternatives]
□ Company/department rules
□ Project manager
□ Software architect
□ Software developer
□ Customer
□ Other (please specify): _______________
3.3 When was the main process decided?
[Mark one alternative]
□ Before the “acquire vs. build” decision, i.e. before the decision whether to acquire
OTS components instead of building them in-house
□ After such an “acquire vs. build” decision
3.4 Were there any dedicated personnel in your project to keep track of the OTS
component knowledge/information?
[Mark one alternative]
□ No
□ Yes
If Yes, what is the role and OTS component-relevant experience of this person?
_______________
Version 1.15 24-May-2004
36
-1:08 AM 08.03.2016
Section 4: Questions on the project level risk management
4.1 What is your opinion on the following aspects of your OTS-based project?
[Mark one alternative per row]
Don’t
agree
at all
Hardl
y
agree
Agree
some
what
Agree Strong Don’t
mostly ly
know
agree
a) The project was not completed
according to schedule
b) Effort to select OTS
components
were
not
satisfactorily estimated
c) Effort to integrate
OTS
components
were
not
satisfactorily estimated
d) OTS components negatively
affected system reliability
e) OTS components negatively
affected system security
f) OTS components negatively
affected system performance
g)
Requirements
were
continuously changed
h) OTS components could not be
sufficiently adapted to changing
requirements
i) Project could not (re)negotiate
requirements with the customer,
if OTS components could not
satisfy all requirements
j) It was difficult to identify
whether defects were inside or
outside the OTS components
k) It was difficult to plan system
maintenance, e.g. because OTS
components had asynchronous
release cycles
l) It was difficult to update the
system with the last OTS
component version
m) OTS components were not
satisfactorily compatible with
production environment when
the system was deployed
n) Information on the reputation
Version 1.15 24-May-2004
37
-1:08 AM 08.03.2016
and technical support ability of
provider were inadequate
o) Provider did not provide
enough
technical
support/
training
4.2 What actions were performed during development or maintenance of the project?
[Mark one alternative per row]
Don’t
agree
at all
Hardl
y
agree
Agree
some
what
Agree Strong Don’t
mostly ly
know
agree
a) Customer had been actively
involved in the “acquire” vs.
“build”
decision
of
OTS
components
b) Customer had been actively
involved in OTS component
selection
c) OTS components were selected
mainly based on architecture
and standards compliance,
instead of expected functionality
d) OTS components qualities
(reliability, security etc.) were
seriously considered in selection
e) Effort in learning OTS
component
was
seriously
considered in effort estimation
f) Effort in black-box testing
OTS component was seriously
considered in effort estimation
g) Unfamiliar OTS components
were integrated first
h) Did integration testing
incrementally (after each OTS
component was integrated)
i) Local OTS-experts followed
the OTS component updates and
possible consequences actively
j) Maintained a continual watch
on the market and looked for
possible substitute components
k) Maintained a continual watch
on provider support ability and
reputation.
Version 1.15 24-May-2004
38
-1:08 AM 08.03.2016
Section 5: Questions on the OTS component level risk management
5.1 How many different OTS components were used in the project (Please fill in a
number) ________?
5.2 Which were the name, version and functionality of OTS components used in the
project?
[NB: If there were more than five OTS components used in the project, please choose
five you are familiar with, and select what you think is the most important OTS
component and fill it into component Comp.1 for later use.]
OTS
component
ID
Full name
and version
(descriptive
free text)
Origin
(write
COTS or
OSS)
Main functionality
(descriptive free text)
Comp.1
Comp.2
Comp.3
Comp.4
Comp.5
5.3 How large a percentage of the application functionality was provided by the
selected OTS component Comp.1 in the whole system ___________________(%)?
5.4 What was the source code status of the selected OTS component Comp.1?
[Make one alternative per row]
OSS, i.e. with source
code available
COTS, but with source
code available
COTS, without source
code
Comp.1
5.5 Have you read parts of the source code of the selected OTS component Comp.1?
[NB: If the source code of Comp.1 is not available, please skip this question]
[Make one alternative per row]
Never
Rather
Sometim Frequent Almost
Don’t know
seldom
es
ly
always
Comp.1
5.6 Have you modified parts of the source code of the selected OTS component Comp.
1?
[NB: If the source code of Comp.1 is not available, please skip this question]
[Make one alternative per row]
Never
Rather
Sometim Frequent Almost
Don’t know
seldom
es
ly
always
Comp.1
Version 1.15 24-May-2004
39
-1:08 AM 08.03.2016
5.7 What was the type of the selected OTS component Comp.1?
[Mark one or more alternatives]
○
○
○
○
○
○
○
.Net
ActiveX
Java Class/ Library
Java Applet/ Servlet
Java Bean/Enterprise Java Bean
C/C++ class/Library
MFC
○
○
○
○
○
○
○
DLL
VBX/ VB Class Library
COM/DCOM
CORBA
Windows Static Link Library
Windows Foundation Class (WFC)
Others (specify):____________
5.8 What was your project members’ experience with the selected OTS component
Comp.1?
[Make one alternative per row]
Very little
Little
Some
Much
Very
Don’t
much
know
Comp.1
5.9 When were the selected OTS components Comp.1 selected?
[Make one alternative per row]
Pre-study Requirem
Overall
Detailed
Coding
phase
ents/Analy
design
design
phase
sis phase
phase
phase
Comp.1
Don’t
know
5.10 Have you encountered the following aspects with the selected OTS components
Comp.1?
[Make one alternative per row]
Yes
No
Don’t
know
a) The OTS component did not follow the industry standard
(such as COM, CORBA etc.) as the provider claimed (glueware
needed)
b) The OTS component had mismatching architectural
assumptions towards other parts of the system (glueware
needed)
c) The OTS component could not provide all required
functionality (addware needed)
d) The recent OTS component versions were not backwardscompatible with the previous version
e) The provider did not want to or could not change OTS
component functionality as requested
Version 1.15 24-May-2004
40
-1:08 AM 08.03.2016
5.11 Have you performed some of the following actions for the selected OTS
components Comp.1?
[Make one alternative per row]
Yes
No
Don’t
know
a) Searched Internet for possible OTS component candidates.
b) Got recommendation of possible OTS component
candidates from customer
c) Got recommendation of possible OTS component
candidates from local colleague/OTS-expert
d) Used a formal decision-making method to compare
possible OTS component candidates, e.g. with weighted
evaluation criteria
e) Limited possible candidates into 1-3 components, by
reading literature or other documentation
f) Did “hands-on” try-out of 1-3 components, e.g. on a downloaded demo version.
5.12 What is your opinion on the following aspects of the selected OTS component
Comp.1?
[Make one alternative per row]
Don’t Hardly Agree Agree Strongl Don’t
agree
agree somew mostly y agree know
at all
hat
a) Comp. 1 was integrated
successfully
b) Comp. 1 was maintained
successfully so far
Version 1.15 24-May-2004
41
-1:08 AM 08.03.2016
Section 6: Questions regarding the context of respondent and business unit
[NB: The given information in 6.1 and 6.2 will be treated confidentially.]
6.1 Personal Information about the respondent
6.1.1 What was your position in this project?
[Mark one or more alternatives]
□ Project manager
□ Software architect
□ Software developer
□ Other level (please specify): _______________
6.1.2 What is your current position in your business unit?
[Mark one or more alternatives]
□ IT manager
□ Project manager
□ Software architect
□ Software developer
□ Other level (please specify): _______________
6.1.3 How long have you been working in the current business unit?
Years: __________________________
6.1.4 How long have you been working with software development?
Years: _________________________
6.1.5 How long have you been working with OTS components?
Years: __________________________
6.1.6 How many OTS component based projects have you been involved in?
Projects: __________________________
6.1.7 What is your principal education degree?
[Mark one alternative]
□ Bachelor
□ Master
□ Ph.D.
□ Other education: ___________________
6.1.8 Is your principal degree in software (informatics/computer science/telematics)?
[Mark one alternative]
□ No
□ Yes
Version 1.15 24-May-2004
42
-1:08 AM 08.03.2016
6.2 Information about your company
6.2.1 What is the type of company?
[Mark one alternative]
□ Stand-alone: The highest reporting entity with no parent organization above it.
□ Subsidiary: An independent entity with majority interest held by a parent.
6.2.2 What is the ownership of your company?
[Mark one alternative]
□ Publicly traded on a stock exchange
□ Privately held company
□ Government, education, or nonprofit organization
6.2.3 What is the staff size of your embedding company (full- & part-time persons) ____?
What is the staff size of your local business unit (full- & part-time persons) ___?
[This staff size may be the same for small and medium sized companies]
6.2.4 What is the main business areas of your company?
[Mark one alternative]
□ IT/ Telecom industry (Ericsson, Nokia, NERA, Nordic VLSI etc.)
□ Telecom service provider (Telenor, NetCom etc.)
□ Software House / Software Vendor (Opera, MicroSoft etc.)
□ Software / IT consulting company (Accenture, Cap Gemini E&Y, TietoEnator etc.)
□ Other software-intensive company (please specify): ___________________
Version 1.15 24-May-2004
43
-1:08 AM 08.03.2016
Appendix B. Summary of hypotheses and theses in the prestudy
1. Hypotheses
(For these hypotheses, some of them have to be revised for this study. Only
the hypotheses marked by bold will be included in this study [Conradi04]).
Retaining four old hypotheses (H01,H02, H04, H10) from prestudy,
possibly revised:

H01: Integration mismatch risk and OTS component type (included)
There is no relationship between the OTS component type (COM, CORBA vs. C++,
Java Library) and the architecture/standard mismatch risk occurrences.
Old version: H01: Standard mismatch vs. architecture mismatch
There is no difference between the number of integration problems caused by lack of
standard compliance and lack of architectural style compliance.
Comments: For the C ++ / Java library, there is no specific standard. Another problem is
that the respondents don't want to tell us the integration problems they encountered.

H02: Custom code related (addware/glueware) risks and OTS component type
(included)
There is no relationship between the OTS component type (COM, CORBA vs. C++,
Java Library) and the custom code (addware/glueware) occurrences.
Old version: H02: Addware vs. glueware
There is no difference of the proportion of addware and glueware in the custom code
Comments: How to tell the difference between the addware and the application. How to
calculate the exact lines of code of glueware and addware

Old H03: Integration easiness vs. function completeness (not included here)
There is no difference between the importance of easiness of integrating the components
into system and completeness of components functionality for the initial selection of OTS
components.
Comments: For waterfall and incremental process, the easiness of integration may be
more important than the functionality completeness. For the prototype process, the
functionality completeness is more important than the ease of integration in some cases.
Version 1.15 24-May-2004
44
-1:08 AM 08.03.2016

H04: Formal selection process and familiarity process (included)
There is no difference in frequency of formal selection process and familiarity process.
Old version: H04: Formal selection method vs. familiarity methods
There is no difference in the frequency of using formal decision making methods and
using familiar with components method in the component selection process
Suggestions: Define the formal selection methods (formal decision making methods or
formal selection process) clearly. Define the familiarity methods (demonstration,
experiment,…) clearly

Old H05: The role of COTS components and selection methods (not included here)
There is no relationship between the role of COTS components and the evaluation
methods used.
Comments: We cannot find out the obvious relationship between the role?? of COTS
components and the evaluation methods used so far. It is also hard to find a project using
COTS components as a core part

Old H06: The experience and provider control (not included here)
There is no relationship between the level of developers' experience with the COTS
component and the level of provider control.
Comments: If it is the first time of using the COTS component, they would like to control
the provider. However, it is not possible to control the provider in some cases.

Old H07: COTS component origin and provider control (not included here)
There is no relationship between the origin of COTS component and the level of provider
control.
Suggestions: It looks like an obvious conclusion, the context validity need discussion.

Old H08: Developer's experience and manager commitment (not included here)
There is no relationship between the level of developers' experience with COTS
component and the level of manager commitment.
Suggestions: The more experience, the less commitment were needed. It is possible to
test it. It looks like an obvious conclusion, the context validity need discussion.
Version 1.15 24-May-2004
45
-1:08 AM 08.03.2016

Old H09: The role of COTS component and manager commitment (not included here)
The role of COTS component will affect the level of manage commitment.
Comments: It is very difficult to find a project using OTS components as the core part of
the system. The needed sample size may be very huge.

H10: OTS component integration risks and selection strategy (included)
There is no difference in integration related risks occurrences between projects which
select OTS components mainly based on integration easiness (architecture and
standard match) rather and projects select OTS components based on expected
functionality
With no or little experience on a specific OTS component, if the selection is based on the
integration easiness, it is easy to success. If the selection is mainly based on the
functionality completeness, it is easy to fail.
Old version: H10: There is no relationship between the selection strategy and the result of
the project.
Suggestions: The definition of functionality completeness?? need to be made clear.
2. Two revised hypotheses
We also propose two new hypotheses based on some open questions in the questionnaire.

New Hx1: Using or not using OTS components and main process used (revised)
Using or not using OTS components will not change the selection of the main
development process (waterfall, prototype, and incremental).
The old version: New Hx1: Using or not using COTS components and process selection
Using or not using COTS components will not change the selection of traditional
development process (waterfall, prototype, and incremental).
Comments: Our results show that there none projects selected the development based on
using or not using COTS components.

New Hx2: Requirement negotiation and customer involvement (revised)
There is no difference in OTS component causes requirement risks (Not be able to
negotiate requirements with customer) occurrences between project involved customer
in OTS component selection and evaluation and the project didn’t involve customers.
Version 1.15 24-May-2004
46
-1:08 AM 08.03.2016
The old version: New Hx2: Requirement negotiation and customer involvement
The possibility of requirement negotiation has no relationship with the customers'
permission of using COTS components
Comments: None of the project we interviewed negotiated the requirement with customer
because a COTS component cannot satisfy some functionality. From customers' point of
view, it is not a good excuse.
3. Four new theses
We also proposed some new theses. We have not figured out how to formulate formal
hypotheses from them.

New thesis 1: the main cost of using the unfamiliar OTS component is in the
learning and understanding (new).

New thesis 2: if possible, the integrators prefer to use replaceable OTS
components instead of trying to control the provider (new).

New thesis 3: if it is the first time of using the same OTS component, it is
required to modify some processes (new).
The process change is different between waterfall, incremental and prototype. If the
project members are familiar with the OTS component, the process change is very
few, and there are no differences between processes.

New thesis 4: reusing the same OTS components in the future project will give
future benefits, such as easy to be learned, shorter time to market and ease of
change (new).
4. Summary of old hypotheses from prestudy
ReidarC: Note, that in point 1 above we have taken four old hypotheses - H01 and
H02 (both reconsidered), H04, and H10, but not H07 and H08 as indicated below.
See [Li04a] and resume from Oslo meeting 3. Feb.2004 on:
www.idi.ntnu.no/~epos/cots/pub/cots-meeting-3feb04.txt.
Summary of existing hypotheses:
We proposed totally 10 hypotheses H01-H10 before the prestudy.
We deleted six hypotheses (H01-03, H05-06, H09) after the prestudy because they are
not easy to be quantified.
Version 1.15 24-May-2004
47
-1:08 AM 08.03.2016
H01 (standard mismatch vs. architecture mismatch):
There is no difference between the number of integration problems caused by lack of
standard compliance and lack of architectural style compliance.
Comments: We found it is not possible to compare under our definition of COTS
components. For the C++ / Java library, there is no specific standard. The architecture
mismatch is definitely the main cause of integration problems.
For the components following COM, CORBA, or EJB standard, this hypothesis can be
tested. But the difficulty is how to define the architecture mismatch. Another difficulty is
that the respondents don't want to tell us the integration problem they encountered.
H02 (addware vs. glueware):
There is no difference of the proportion of addware and glueware in the custom code.
Comments: The difficulty is how to tell the difference between the addware and the
application. Another difficulty is that it is not easy to get the exact lines of code of
glueware and addware. More than half projects cannot give out the exact number of the
glueware and addware.
H03 (integration easiness vs. functionality completeness):
There is not difference between the importance of easiness of integrating the components
into system and completeness of components functionality for the initial selection of
COTS components.
Comments: The challenge for this hypothesis is that it may have relationship with the
development process. For waterfall process and incremental process, the easiness of
integration may be more important than the functionality completeness. For the prototype
process, the functionality completeness is more important than the ease of integration in
some cases.
H05 (connection between the role of COTS components and selection methods):
There is no relationship between the role of COTS components and the evaluation
methods used.
Comments: We cannot find out the obvious relationship between the role of COTS
components and the evaluation methods used so far. The most affect factor in using the
evaluation method is the project members' experience on the COTS components. Because
we haven't found any projects using formal selection method, it is not easy to be tested.
H06 (connection between the experience and provider control):
There is no relationship between the level of developers' experience with the COTS
components and the level of provider control.
Comments: We found it is difficult to be tested. If it is the first time of using the COTS
component, they would like to control the provider.
However, it is not possible to
control the provider in some cases.
H09 (connection between the role of COTS components and manager
commitment):
The role of COTS components will affect the level of manage commitment.
Version 1.15 24-May-2004
48
-1:08 AM 08.03.2016
Comments: The risk is that it is very difficult to find a project using COTS components
as the core part of the system. The needed sample size may be very huge.
We keep four hypotheses (H04, H07, H08, H10) that can be tested. Colleagues gave us
some suggestions on these hypotheses, we will modify them to make them better.
H04 (formal selection method vs. familiarity methods):
There is no difference in the frequency of using formal decision making methods and
using familiar with components method in the component selection process.
Comments: We haven't found any projects using the formal selection methods so far.
This hypothesis can be tested. The challenge is how to define the formal selection
methods? Is it the formal decision making methods and formal selection process?
Another challenge is how to define the familiar with components?
Suggestions: the question in the questionnaire should be modified because it doesn't
catch the meaning of formal selection methods and familiar with components.
H07 (connection between COTS component origin and provider control):
There is no relationship between the origin of COTS components and the level of
provider control.
Comments: For this hypothesis, we found there is some relationship. If the COTS
component was bought from market, it is not easy to control the provider. If the
component was got by the contract, it is possible to control the provider in some cases.
Suggestions: It looks like an obvious conclusion, the context validity need discussion.
H08 (connection between developer's experience and manager commitment):
There is no relationship between the level of developers' experience with COTS
components and the level of manager commitment.
Comments: We the more experience, the less commitment were needed. It is possible
to test it.
Suggestions: It looks like an obvious conclusion, the context validity need discussion.
H10 (connection between selection strategy and final result):
With no or little experience on a specific COTS component, if the selection is based on
the integration easiness, it is easy to success. If the selection is mainly based on the
functionality completeness, it is easy to fail.
Suggestions: The definition of functionality completeness need to be made clear.
Version 1.15 24-May-2004
49