Research in Software Engineering
– methods, theories,…
basta o cerchiamo?
NTNU, IDI, SU group PhD seminar, 23 Nov. 2007
(rev. 2 Dec. 2007, 12 Jan. 2009, 2.2.09)
Reidar Conradi
Dept. Computer and Information Science (IDI)
NTNU, NO-7491 Trondheim
www.idi.ntnu.no/grupper/su/publ/ese/res-methods.ppt
conradi@idi.ntnu.no, Tel +47 73.593444, Fax +47 73.594466
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
1
Ex. Software crisis - bad software
Some recent Software Engineering (SE) incidents/risks
in Norway:
• Sparebank 1 Midt-Norge (20 Oct. 2007): 200 000 netbank
users got most of their scheduled, monthly transactions
run twice. To be reversed the next days.
• Adresseavisen (2007): several printing delays due to
computer problems.
• Skandiabanken (Spring 2007): Electronic burglary of one
account.
• Jernbaneverket, Sandvika (20 April 2005): Almost train
collision due to a stop signal not showing red.
• CHAOS Report (1995 and later) by Standish Group.
• See www.idi.ntnu.no~/conradi/IT-debate/risks.html
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
2
SU motivation
•
Software essential in many important societal activities.
50-60,000 system developers in Norway – many without formal SW education.
Still many challenges wrt. software quality and delivery on time and budget; cf. [US
Standish report, 1995], cited in [PITAC, 1999], on projects for tailored software:
– 31% stopped before finish, 81 bill. $ loss/year (1% of GNP!)
– 53% have serious overruns (189% average), 59 bill. $/year
•
Some challenges:
– Web-systems: Manage time-to-market (TTM) vs. reliability?
– Component-based development (OSS, COTS): quality, risks
– Business critical systems
– How do software systems evolve over time, cf. Y2K?
– What is empirically known about software products and processes?
– How can small companies carry out systematic improvement?
– How to perform valid sw.eng. research in a university
-- by student projects and having industry serving as a lab?
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
3
Software development project risk factors (1)
Top ten risk factors for a software development project [Boehm 1989] :
• Personnel shortfalls (inexperience with domain, tools, personnel turnover
etc.).
• Unrealistic schedule or budget estimates.
• Incorrect functionality caused by e.g. misinterpretation of customer needs.
• Incorrect user interface.
• Implementing “nice” features not requested by the customer.
• Requirements volatility – requirements changes increase rework.
• Quality or functionality problems with external components.
• Subcontractor problems – inadequate quality in the work/skills.
delivered/provided.
• Real-time performance shortfalls of (parts of) the software system.
• Unstable environment/untried technology affected the development schedule
negatively.
See [Boehm 1989] Barry Boehm, Software Risk Management, IEEE, 1989.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
4
Public project success factors (2)
However, the critical 12 factors are the ”softer” ones :
•
•
•
•
•
•
•
•
•
•
•
•
Effective project planning
Effective project cost estimating
Effective project measurements
Effective project milestone tracking
Effective project quality control
Effective project change management
Effective development processes
Effective communications
Capable project managers
Capable technical personnel
Significant use of specialists
Substantial volume of reusable materials
See Capers Jones: ”Government Software Projects Rank High in Major Critical
Success Factors”, Crosstalk, Jan. 2002, p. 19 (factors taken from own 1995
book), www.stsc.hill.af.mil/crosstalk/2002/01/jones.pdf.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
5
General project failure factors (3)
Why do projects fail so often? - the most common 12 factors :
• Unrealistic or unarticulated project goals
• Inaccurate estimates of needed resources
• Badly defined system requirements
• Poor reporting of the project's status
• Unmanaged risks
• Poor communication among customers, developers, and users
• Use of immature technology
• Inability to handle the project's complexity
• Sloppy development practices
• Poor project management
• Stakeholder politics
• Commercial pressures
See Robert N. Charette: “Why software fails - continued” (part 2), IEEE Spectrum,
42(9):42-49, Sept. 2005, see http://www.spectrum.ieee.org/sep05/1685/2 with over 30
failed projects in “software hall of shame”.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
6
10 factors, with 3
factor estimates
1.
commitment.
Summary of projct factors
[Boehm89]’s 10 risk
software projects:
factors
for
[Jones02]’s 12 non-success factors for
public projects:
Goals,
[Charettes05]’s 12 failure factors:
Unrealistic or unarticulated project goal.
2.
Project
management.
Incapable project managers.
Poor project management.
Unmanaged risks.
Inability to handle the project's complexity.
3. Communication.
Ineffective communications.
Poor communication
developers, and users.
4.
Technical
personnel.
Personnel shortfalls or turnover.
Incapable technical personnel.
Insignificant use of specialists.
5.
Estimation,
tracking
Unrealistic schedule or budget estimates.
Ineffective project cost estimating.
Ineffective project measurements.
Ineffective project milestone tracking.
6. Bad requirements.
Incorrect functionality vs. needs.
among
customers,
Inaccurate estimates of needed resources.
Poor reporting of the project's status.
Badly defined system requirements.
Implementing “nice”, but non-requested
features.
Incorrect user interface.
7.
Changing
requirements.
Unstable requirements.
Ineffective project change management.
8. Product quality.
Problems with external components.
Subcontractor quality problems.
Real-time performance shortfalls.
Ineffective project quality control.
9.
Development
technology.
Unstable envir./untried technology.
Effective development processes.
10. Software reuse.
Sloppy development practices.
Use of immature technology.
Non-substantial
materials.
volume
of
reusable
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
7
Proposed ”silver bullets” [Brooks87] (1)
What almost surely works:
• Software reuse/CBSE/COTS: yes!!
• Formal inspections: yes!!
• Systematic testing: yes!!
• Better documentation: yes!
• Versioning/SCM systems: yes!!
• OO/ADTs: yes?!, especially in domains like
distributed systems and GUI.
• High-level languages: yes! - but Fortran, Lisp,
Prolog etc. are domain-specific.
• Bright, experienced, motivated, hard-working,
…developers: yes!!! – brain power.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
8
Proposed ”silver bullets” (2)
What probably works:
• Better education: hmm?
• UML: often?; but need tailored RUP.
• Powerful, computer-assisted tools, Eclipse: partly?
• Incr./agile methods, involve users; XP, SCRUM: partly?
• More ”structured” process/project (model): probably?, if
suited to purpose. But beware of OSS.
• Software process improvement; TQM, ISO-9001, CMM:
depends?, assumes stability.
• ”Structured programming”: not clear wrt. maintenance?
• Formal specification/verification/code-generation: does
not scale up? – only for safety-critical systems, so
constructive CBSE has ”won”.
=> Need further studies (”eating”) of these ”puddings”
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
9
The next best “silver bullet”:
Empirical Software Engineering (ESE)
• Lack of systematic validation in computer science / software
engineering vs. other disciplines: [Tichy98] [Zelkowitz98].
• (New) technologies not properly validated: OO, UML, …
• Empirical / Evidence-based Software Engineering since 1992:
writings by [Basili94], [Wohlin00], [Rombach93], Juristo??.
• Int’l Software Engineering Research Network (ISERN) group, 1992.
ESERNET EU-project in 2001-03.
• SE group at NTNU since 1993, at UiO from 1997 – both with ESE
emphasis.
• SE at Simula Research Laboratory from 2001: attn/ Dag Sjøberg,
in coop. with NTNU, SINTEF et al.
• SPIQ, PROFIT, SPIKE, EVISOFT, norskCOSI, ... projects on
empirical and practical SPI in Norway, 1996-2010.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
10
But ESE not easy, since SE is “special”
• Problems in being more “scientific”:
– Most industrial SE projects are unique (goals, technology, people,
…), otherwise just reuse software with marginal copy cost!
– Fast change-rate between projects: goals, technology, people,
process, company, … – i.e. no stability, meager baselines.
– Also fast change-rate inside projects: much improvisation, with
theory serving as back carpet.
– So never enough time to be “scientific” – with theory building,
hypotheses, metrics, data collection, analysis, … and actions.
• Tens of context factors in software projects: 3**N for trinary factors.
• Strong “soft” (human and organizational) factors.
• SE learnt by “doing”, not by “reading” experience reports; need realistic
projects in SE courses [Brown91].
• So how to show effect and causality? Realism vs. rigor?
• How can we overcome these obstacles, i.e. to learn and improve
systematically? – ESE as the answer? – Or action research? Or …
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
11
Ex. “Context” factors/variables
• To understand a discipline: must build and validate
theories/models that relate some key concepts/factors –
incl. context factors.
• People factors: number of people, level of expertise, group
organization, problem experience, process experience, …
• Problem factors: application domain, newness to state of
the art, susceptibility to change, problem constraints, …
• Process factors: life cycle model, methods, techniques,
tools, programming language, other notations, …
• Product factors: deliverables, system size, required
qualities such as time-to-market, reliability, portability, …
• Resource factors: target and development machines,
calendar time, budget, existing software, …
• Example: 29 factors to predict sw productivity [Walston77].
(from Basili’s CMSC 735 course at Univ. Maryland, fall 1999)
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
12
Ex. Four basic parameters in a study (from topdown GQM-method)
• Object: a process, a product, any form of model.
• Purpose: characterize, evaluate, predict, control,
improve, …
• Focus (relevant object aspect): time-to-market,
productivity, reliability, defect detection,
accuracy of estimation model, …
• Point of view (stakeholder): researcher, manager,
customer, …
- all this involves many factors/variables.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
13
Ex. “U”-model of fault rate vs. size
• [Basili84]: the fault rate of modules shrunk as module size and
complexity grew in the NASA-SEL environment; other authors
had inverse observation – who was right?
Fault
Rate
Basili: Actual
in NASA
Beleived intuitively
Others: Hypothesized
Size/Complexity
• Explanation: smaller modules are normally better, but involve
more interfaces and often chosen when “(re-)gaining” control.
• Above result confirmed by similar studies - but many more
factors …
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
14
Ex. Estimation models, e.g. by Barry Boehm
• Effort = E1 * Size ** 1.07 + E2
% Diseconomy of scale
• Duration = D1 * Effort ** 0.38 + D2 % ca. cube root
• And many other magic formulaes!
• Question: Can “E1” express 29 underlying factors?
• And how to calibrate for an organization, and use
with sense?
• Formal vs. informal (expert) estimation
[Jørgensen03]?
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
15
Theory building and ESE (1)
• Theory: set of related concepts to describe / understand a certain
phenomenon, e.g. as a law or to summarize experiences or
lessons-learned.
• Theory must be operationable or ”fruitful”, enabling design and
prediction of concrete ”empirical studies” to possibly verify or
falsify itself; otherwise just brain spin. So gain trust and
generalization over time.
• Law has four parts:
• Phenomena/concepts: what?
% V, r, I (see below)
• Relations/propositions/operators: how? % Δ, =, *
• Explanation: why?
% Maxwell …
• Constaints/validity: where?
% not in plasma/quantum
Ex. Ohm’s law: ΔV = r * I
• Empirical study: to explore or verify some phenomena/theory;
chosen research goal and scope w/ e.g. artifacts, actors and
processes, pertinent research method(s), ethical concerns.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
16
Theory building and ESE (2)
− Technology:
− ”what” – concepts, models, languages, executable tools,
techniques, methods; related to a theory?
− ”how” and ”why”– entire processes.
− Cost/benefits, with given project context?
– Our phenomena or main study subjects and objects:
• Technology: UML, Java, agile methods, process models, …
• Technical artifacts: rqmnts, designs, code, test data, …
• Actors: humans w/ roles, projects, external stakeholders.
• Context: part of project, in lab exercise, or freestanding.
– Our data/experiences: very diverse, hard and soft, partly
controllable and valid, costly!
– Our research methods: superset of those in science, social
sciences, engineering!! – over 20 such.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
17
Over 20 research methods for (E)SE [Sindre07]
qualitative
Philosophical
discussion
Participative
preparations
Grounded Theory
Action Research
Field study/Observation
Case study
Post Mortem Analysis
Literature review
Structured interview
Proof-of-concept
Prototyping
analytical
Design science /
VR
Math.
modellling
Mathematical
proof
Simulation
Benchmarking
Testing
quantitative
Data mining /
Archival studies
empirical
Quasiexperiment
Survey
Controlled
experiment
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
18
ESE: common research methods (1)
1. Philosophical discussion: refreshing, but no end.
2. Literature review: fetch abstracts first, then read and
classify papers, costly, boring? Use google/ontologies more?
3. Proof-of-concept: developer herself makes feasibility demo.
4. Prototyping: interactive and gradual refinement of goals and
solutions in fast steps.
5. Design science/Virtual Reality: like prototyping - build a
system (oil rig) using an executable and graphic model.
6. Mathematical modelling: make a mathematical model,
often partial differential equations, Newtonian mechanics, or
applications of this. GPS satellites apply General Relativity!
7. Mathematical proof: (manually) verifying a formal model /
specifations. Does not scale up, sorry.
8. Simulation: executing a mathematical/ stochastic model (by
math.modelling in 6), to predict and learn – ex. weather,
world climate via IPCC.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
19
ESE: common research methods (2)
9. Benchmarking: comparing different algorithms/models.
10. Testing: special runs of a program to check its dynamic
properties, long time before stabilization – as for large
physics experiments – don’t stop too early to falsify!
11. Participative preparations (”Scandinavian school”):
workshop-like design and planning.
12. Grounded Theory: Generalize words/concepts from texts.
13. Action Research: researcher & developer overlap in roles.
14. Field study/Observation: being a “fly on the wall”, or also by
automatic logging tools.
15. Case study: try out new technology in real project.
16. Structured interview: more open questions than in surveys,
brings up lots of insights, transcription takes time, apply
Grounded Theory later?
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
20
ESE: common research methods (3)
17. Post Mortem Analysis: collect lessons-learned, by
interviews [Birk02].
18. Data mining / Archival studies: dig out historical data,
bottom-up metrics, costly.
19. Quasi-experiment, in “vivo”, in industry: costly and hard
logistics. Use Simula’s SESE web-tool [Sjøberg02]?
20. Survey: by phone, emailed questionnaires or web servers,
costly randomization with “unaccessible” respondents,
unreliable census data.
21. Controlled experiment, “in vitro”, often among students:
can control the artifacts, process and outer context.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
21
ESE: different data categories
• Quantitative (“hard”) data: mainly numbers
according to a predefined metrics, both direct and
indirect data. Need suitable analysis methods,
depending on the metrics scale – nominal, ordinal,
interval, and ratio.
Often objective. But: “10000vis av regninger” – false.
• Qualitative (“soft”) data: prose text, pictures, …
Often from observation and interviews. Need much
human interpretation.
Often subjective. But: “Norge beat Malta 4-1” - true.
Specific data for a given study (e.g. reuse rate) vs.
Common data (cost, size, #faults, …) - “baseline”?
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
22
ESE: validity problems
• Construct validity: the “right” (relevant,
precise, minimal, …) metrics - use GoalQuestion-Metrics?
• Internal validity: the “right” data values.
• Conclusion validity: the right (appropriate)
data analysis.
• External validity: the “right” (representative)
context.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
23
ESE: combining different studies/data (1)
•
•
Meta-studies: aggregations over previous single
studies. Cf. medicine with Cochran reporting
standard. Need shared experience databases?
A composite study may combine several study
types and data, sequentially to track SPI:
1.
2.
3.
4.
5.
Prestudy, doing a survey or post-mortem
Initial formal experiment, on students
Sum-up, using interviews
Final case study, in industry
Sum-up, using interviews or post-mortem
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
24
ESE: combining different studies/data (2)
A composite study may also combine data
concurrently, by triangulation to verify
status:
1.
2.
3.
4.
Interviews of project personnel.
Data mining of ongoing project.
Case study of same project.
Independent observation of same project.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
25
Three slides from Tor Stålhane, April 2007:
Correlations vs. Causality - 1
Several published papers have an argument
roughly as follows:
Corr(A, B) > v, v >> 0, and A precedes B in time.
 A ”causes” B.
A
B
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
26
Correlations vs. Causality - 2
An observed correlation (v) can however be
explained in many ways:
1. A => B
Either this.
2. X => A, B. Or this, see below figure.
That is, mere coincidences in 1 - see next slide.
A
B
X
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
27
Correlations vs. Causality - 3
?
Stork density, A
Birth rate, B
Low degree of urbanisation, X
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
28
Achieving validated knowledge: by ESE
• Learn about ESE: [Rombach93] [Conradi03].
• Set goals, e.g. use QIP [Basili95]?
• Need operational methods to perform studies: general
[Kitchenham02], on GQM [Basili94]?
• Cooperate with others on repeatable studies / experiments
(ISERN, ESERNET, …) [Vokác03].
• Perform meta-analysis across single studies.
Need reporting procedures, databases etc.
• Need more industrial studies, not only with students.
• Have patience, allocate enough resources. Industrial
studies will run into unexpected problems; SPI initiatives
have 30-70% “abortion” rate [Conradi02] [Dybå03].
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
29
Ex. Some NTNU studies
(per 2003, all published)
CBSE/reuse:
• Assessing reuse in 15 companies in REBOOT, 1991-95.
• Modifiability of C++ programs and documentation, 1995.
• Ex3, INCO: COTS usage in Norway, Italy, and Germany 2002-04 (many).
• Assessment of COTS components, 2001-02.
• Ex2, INCO: CBSE at Ericsson-Grimstad, 2001-04 (many).
Inspections:
• Perspective-based reading, at U. Maryland and NTNU, 1995-96.
• Ex1, NTNU diploma theses: SDL inspections at Ericsson, 1993-97.
• UML inspections at U.Maryland, NTNU and at Ericsson, 2000-02.
SPI/quality:
• Role of formal quality systems in 5 companies, 1999.
• Comparing process model languages in 3 companies, 1999.
• Post-mortem analysis in two companies, 2002.
• SPI experiences in SMEs in Scandinavia and in Italy and Norway, 19972000.
• SPI lessons-learned in Norway (SPIQ, PROFIT), 1997-2002.
And many more!
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
30
Ex1. SDL inspections at Ericsson-Oslo 1993-97,
data mining study in 3 MSc theses (Marjara et al.)
General comments:
• AXE telecom switch systems, with functions around * and #
buttons, teams of 50 people.
• SDL and PLEX as design and implementation languages.
• Data mining study of internal inspection database.
No previous analysis of these data.
• Study 1: Project A, 20,000 person-hours. Look for general
properties + relation to software complexity (by Marjara
being a previous Ericsson employee).
• Study 2: Project A + Project-releases B-F, 100,000 personhours. Also look for longitudinal relations across phases and
releases, i.e. “fault-prone” modules - seems so, but not
conclusive (by Skåtevik and Hantho)
• When results came: Ericsson had changed process, now using
UML and Java, but with no inspections.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
31
Ex1. General results of SDL inspections at EricssonOslo 1993-97, by Marjara
Activity
Inspection
design
Yield =
Number
of Defects
[#]
preparation,
928
Total effort
on defect
detection
[h]
786.8
Costefficiency
[defect/h]
Total effort
on defect
correction
[h]
Estimated saved
effort by early
defect removal
(“formulae”) [h]
311.2
8200
1.17
Inspection meeting, design
29
375.7
0.077
Desk Check (Unit Test and
Code Review)
404
1257.0
0.32
-
-
89
7000.0
0.013
-
-
Total so far
1450
9419.5
0.15
-
-
System Test
17
-
-
-
-
Field Use (first 6 months)
35
-
-
-
-
Function Test
Table 1. Yield, effort, and cost-efficiency of inspection and testing, Study 1.
Study 1 overall results:
- About 1 person-hour per defect in inspections.
- About 3 person-hours per defect in unit test, 80 p-h/defects in function test.
- So inspections seem very profitable.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
32
Defects found in unit test
Defects found during inspections
Ex1. SDL-defects vs. size/complexity (#states) at
Ericsson-Oslo 1993-97, by Marjara
States
Study 1 results, almost “flat” curve -- why?:
- Putting the most competent people on the hardest tasks!
- Such contextual information is very hard to get/guess.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
33
Ex1. SDL inspection rates/defects at Ericsson-Oslo
1993-97, by Marjara
>1
Recommended rate
actual rate
0.66
8
Study 1: No internal data analysis, so no adjustment of insp. process:
- Too fast inspections: so missing many defects.
- By spending 200(?) analysis hours, and ca. 1250 more inspection hours:
will save ca. 8000 test hours!
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
34
Ex2. INCO, studies and methods by PhD student
Parastoo Mohagheghi, NTNU/Ericsson-Grimstad
• Study reusable middleware at Ericsson, 600 KLOC, shared
between GPRS and UMTS applications:
– Characterization of quality of reusable comp. (pre-case study)
– Estimation of use-case models for reuse – with Bente Anda,
UiO (case study)
– OO inspection techniques for UML - with HiA, NTNU, and
Univ. Maryland (real experiment)
– Attitudes to software reuse – with two other companies (survey)
– Evolution of product families (post-mortem analysis)
– Improved reuse processes (proposal for case study)
– Reliability and stability of reusable components, based on
13,500 (!) change requests – with NTNU (case study/data
mining), next three slides
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
35
Ex2. GPRS/UMTS system at Ericsson-Grimstad
Application A
Application B
Application-specific
components
Business Specific
Middleware
(& Component Framework)
System Platform
Reused components
in our study
Reused, but
considered as
COTS and OSS here
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
36
Ex2. Research design (data mining)
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
37
Ex.2 Hypotheses testing (as null-hyp.)
• H01: Reused components have same fault-density as nonreused components. Rejected - reused more reliable.
• H02a: There is no relation between #faults and component size
for all components. Not rejected - not incr. with size.
• H02b: There is no relation between #faults and component size
for reused components. Not rejected - not incr. with size for
reused.
• H02c: There is no relation between #faults and component size
for non-reused components. Rejected - incr. with size for nonreused.
• H03a/b/c: There is no relation between fault-density and
component size for all/reused/non-reused components. Not
rejected.
• H04: Reused and non-reused components are equally modified.
Rejected - reused more stable.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
38
Ex3. COTS usage contradicts “common wisdom”
In INCO, structured interviews of 7 Norwegian and Italian SMEs:
• Thesis T1: Open-source software is often used as closed source.
• Thesis T2: Integration problems result primarily from lack of
compliance with standards; not architectural mismatches.
• Thesis T3: Custom code is mainly devoted to add functionalities.
• Thesis T4: Formal selection seldom used; rather familiarity with
product or generic architecture.
• Thesis T5: Architecture more important than requirements to
select components. - Reidar: no longer true; better standards.
• Thesis T6: Tendency to increase level of control over vendor
whenever possible.
See [Torchiano04].
Extended with larger Norwegian OSS/COTS survey by NTNU and
Simula, later repeated in Germany and Italy [Li08].
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
39
•
•
•
•
•
•
From 50 software “laws” [Endres03]:
L1, Glass: Requirement deficiencies are the
prime cause of project failures.
L5, Curtis: Good designs require deep
application domain knowledge.
L12, Corbató: Productivity and reliability
depend on the length of a program’s text,
independent of language level used.
L16, Conway: A system reflects the
organizational structure that built it.
L23, Weinberg: A developer is unsuited to test
his or her code.
L27, Lehman-1: A system that is used will be
changed.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
40
More from 50 software “laws”:
• L30, Basili-Möller: Smaller changes have a
higher error density than large ones.
• L36, Brooks: Adding manpower to a late
project makes it later.
• L45, Moore: The price/performance of
processors is halved every 18 month.
• L47, Cooper: Wireless bandwidth doubles
every 2.5 years.
• L49, Metcalfe: The value of a network
increases with the square of its users.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
41
Some of the 25 hypotheses, also from [Endres03]:
• H2, Booch-2: Object-oriented designs reduce
errors and encourage reuse.
• H5, Dahl-Goldberg: Object-oriented
programming reduce errors and encourage
reuse.
• H9, Mays: Error prevention is better than error
removal.
• H16, Wilde: Object-oriented programs are
difficult to maintain.
• H25, Basili-Rombach: Measurements require
both goals and models.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
42
Conclusion (1)
• Best practices: depend on context, so must know more
about that relation!!
• Need feedbacks from and cooperation with industry to
be helpful – our “laboratory”! Compensate industry.
• Seek relevance of data to actual goal/hypothesis!
But unused data worse than no data?
• ESE: promising, but hard. Research design? Statistics?
• High ESE / SPI activity in Norway since 1997.
• Much international cooperation.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
43
Conclusion (2)
• Higher R&D spending in Norway?: only 1.55% of GNP
(2005), in spite of parliamentary promises from April 2000 on
reaching OECD-level (2.25%) in 4 years.
• Ex. NFR is using 150 MNOK per year on basic software
research – as much as the three best Norwegian football
players earn per year!
• Standardized formats for reporting empirical studies? Ex.
Kreftregisteret for medicine, SSB for general data, Air traffic
authority, Water research institute etc. – what public
“bureau” is for (empirical) software engineering?
• Chinese proverb:
– invest for one year - plant rice,
– invest for ten years – plant a tree,
– invest for 100 years – educate people.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
44
Appendix 1: Some useful web addresses
• Fraunhofer Institute for Experimental Software Engineering (IESE),
Kaiserslautern: www.iese.fhg.de
• International Software Engineering Research Network (ISERN):
www.iese.fraunhofer.de/isern
• Fraunhofer Center for Experimental Software Engineering, Univ. Maryland
(FC-MD): http://fc-md.umd.edu
• EU-network on Experimental Software Engineering (ESERNET, 2001end-2003): www.esernet.org
• Software engineering group (SU) at IDI, NTNU:
www.idi.ntnu.no/grupper/su/
• Industrial software engineering group (ISU) at UiO: www.ifi.uio.no/~isu/
• SINTEF Telecom and Informatics: www.sintef.no
• Simula Research Laboratory, Oslo: www.simula.no (see under “research”
and then “Software Engineering”)
• EVISOFT project: www.idi.ntnu.no/grupper/su/evisoft.html (NTNU
one).
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
45
Appendix 2: Literature list (1)
[Basili84] Victor R. Basili, Barry T. Perricone: “Software Errors and Complexity: An Empirical Investigation”,
Commun. ACM, 27(1):42-52, 1984 (NASA-SEL study).
[Basili94] Victor R. Basili, Gianluigi Caldiera, and Hans Dieter Rombach: "The Goal Question Metric Paradigm", In
John J. Marciniak (Ed.): Encyclopedia of Software Engineering -- 2 Volume Set, John Wiley and Sons, 1994,
p. 528-532, 1994.
[Basili95] Victor R. Basili and Gianluigi Caldiera: “Improving Software Quality by Reusing Knowledge and
Experience”, Sloan Management Review, 37(1):55-64, Fall 1995 (on Quality Improvement Paradigm, QIP).
[Basili01] Victor R. Basili and Barry Boehm: “COTS-Based Systems Top 10 List”, IEEE Computer, 34(5):91-93,
May 2001.
[Birk02] Andreas Birk, Torgeir Dingsøyr, and Tor Stålhane: "Postmortem: Never leave a project without it", IEEE
Software, 19(3):43-45, May/June 2002.
[Brooks87] Frederick P. Brooks Jr.: No Silver Bullet - Essence and Accidents of Software Engineering. IEEE
Computer, 20(4):10-19, April 1987.
[Brown91] John Seely Brown and Paul Duguid: "Organizational Learning and Communities of Practice: Toward a
Unified View of Working, Learning, and Innovation, Organization Science, 2(1):40-57 (Feb. 1991).
[Conradi02] Reidar Conradi and Alfonso Fuggetta: "Improving Software Process Improvement", IEEE Software,
19(4):92-99, July/Aug. 2002.
[Conradi03] Reidar Conradi and Alf Inge Wang (Eds.): Empirical Methods and Studies in Software Engineering -Experiences from ESERNET, Springer Verlag LNCS 2765, ISBN 3-540-40672-7, Aug. 2003, 278 pages.
[Dybå03] Tore Dybå: "Factors of SPI Success in Small and Large Organizations: An Empirical Study in the
Scandinavian Context", In Paola Inverardi (Ed.): "Proceedings of the Joint 9th European Software
Engineering Conference (ESEC'03) and 11th SIGSOFT Symposium on the Foundations of Software
Engineering (FSE-11)“, Helsinki, Finland, 1-5 September, ACM Press, pp. 148-157.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
46
Appendix 2: Literature list (2)
[Endres03] Albert Endres and Hans-Dieter Rombach: A Handbook of Software and Systems
Engineering: Empirical Observations, Laws, and Theories, Fraunhofer IESE / Pearson
Addison-Wesley, 327 p., ISBN 0 321 154207, 2003.
[Jørgensen03] Magne Jørgensen, Dag Sjøberg, and Ulf Indahl: “Software Effort Estimation by
Analogy and Regression Toward the Mean”, Journal of Systems and Software, 68(3):253-262,
Nov. 2003.
[Kitchenham02] Barbara A. Kitchenham, Susan Lawrence-Pfleeger, L.M. Pickard, P.W. Jones, D.C.
Hoaglin, Khalid El Emam, and J. Rosenberg: "Preliminary guidelines for empirical research in
software engineering", IEEE Trans. on Software Engineering, 28(8):721-734, Aug. 2002.
[Li08] Jingyue Li, Reidar Conradi, Christian Bunse, Marco Torchiano, Odd Petter N. Slyngstad, and
Maurizio Morisio: "Development with Off-The-Shelf Components: 10 Facts", Forthcoming in
IEEE Software in 2008, 11 p.
[PITAC99] President’s Information Technology Advisory Committee: “Information Technology
Research: Investing in Our Future”, 24 Feb. 1999, http://www.hpcc.gov/pitac/.
[Rombach93] Hans-Dieter Rombach, Victor R. Basili, and Richard W. Selby (Eds.): Experimental
Software Engineering Issues: Critical Assessment and Future Directives, Springer Verlag
LNCS 706, 1993, 261 p. (from International Workshop at Dagstuhl Castle, Germany, Sept.
1992).
[Sjøberg02] Dag Sjøberg, Bente Anda, Erik Arisholm, Tore Dybå, Magne Jørgensen, Amela
Karahasanovic, Espen Koren, and Marek Vokác: ”Conducting Realistic Experiments in
Software Engineering”, ISESE’02, Nara, Japan, October 3-4, 2002, pp. 17-26, IEEE CS Press
(about SESE web-tool – an Experiment Support Environment for Evaluating Software
Engineering Technologies).
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
47
Appendix 2: Literature list (3)
[Sindre07] Guttorm Sindre: “Forelesningsfoiler til DT8108 IT-emner –2007, IDI, NTNU,
http://www.idi.ntnu.no/emner/dif8916/pdfs/forelesninger/Analytic.pdf (til felles
metodekurs for alle IDIs dr.studenter, foil 1, senere tilpasset av Reidar C.).
[Tichy98] Walter F. Tichy: "Should Computer Scientists Experiment More", IEEE
Computer, 31(5):32-40, May 1998.
[Torchiano04] Marco Torchiano and Maurizio Morisio: "Overlooked Facts on COTS-based
Development", Forthcoming in IEEE Software, Spring 2004, 12 p.
[Vokác03] Marek Vokác, Walter Tichy, Dag Sjøberg, Erik Arisholm, and Magne Aldrin: “A
Controlled Experiment Comparing the Maintainability of Programs Designed with
and without Design Patterns – a Replication in a real Programming Environment”,
Journal of Empirical Software Engineering, 9(3): 149-195 (2004).
[Walston77] C. E. Walston and C. P. Felix: "A Method of Programming Measurement and
Estimation“, IBM Systems Journal, 16(1):54-73, 1977.
[Wohlin00] Claes Wohlin, Per Runeson, M. Höst, M. C. Ohlsson, Björn Regnell, and A.
Wesslén: Experimentation in software engineering: An introduction, Kluwer
Academic Publishers, 2000. ISBN 0-792-38682-5, 224 pages.
[Zelkowitz98] Marvin V. Zelkowitz and Dolores R. Wallace: "Experimental Models for
Validating Technology", IEEE Computer, 31(5):23-31, May 1998.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
48
Appendix 3: SU group at NTNU
IDI’s software engineering (SU) group:
• Five faculty members: Reidar Conradi, Tor Stålhane,
Letizia Jaccheri, Monica Divitini, Alf Inge Wang.
• Five researchers/postdocs: Sobah A. Petersen, Anna
Trifonova, Jingyue Li, Sven Ziemer, Thomas Østerlie,
• 12 active PhD-students, 4 more from 2008; common core
curriculum in empirical research methods.
• 30-40 MSc-cand. per year, 2-3 PhDs per year.
• Research-based education: students participate in projects,
project results are used in courses.
• A dozen R&D projects, basic and industrial, in all our
research fields – industry is our lab.
• Half of our papers are based on empirical research, and
25% are written with international co-authors.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
49
Research fields of SU group (1)
• Software Quality: reliability and safety, software process
improvement, process modelling
• Software Architecture: CBSE: OSS and COTS,
versioning, evolution
• Co-operative Work: learning, awareness, mobile
technology, computer games, project work
In all this:
• Empirical methods and studies in industry and among
students, experience bases.
• Software engineering education: partly project-based.
• Tight cooperation with Simula Research Laboratory/UiO
and SINTEF, 15-20 active companies, Telenor R&D,
Abelia/IKT-Norge etc.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
50
Research fields of the SU group (2)
Software
quality
CBSE: OSS,COTS,
Evolution, SCM
Software
architecture
Reliability,
safety
SPI, learning
organisations
Computer games
Co-operative
work
Software
Engineering
Education
Mobile
technology
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
51
SU research projects since 2000, part 1
Supported by NFR, basic research:
1.
CAGIS-2, 1999-2002: distributed learning environments, COO lab,
Ekaterina Prasolova-Førland (Divitini).
2.
MOWAHS, 2001-04: mobile technologies, Carl-Fredrik Sørensen
(Conradi); coop. with DB group.
3.
INCO, 2001-04: incr. and comp.-based development, Parastoo
Mohagheghi at Ericsson (Conradi); with Simula/UiO.
4.
WebSys, 2002-05: web-systems – reliability vs. time-to-market,
Sven Ziemer and Jianyun Zhou (Stålhane).
5.
BUCS, 2003-06: business critical software, Jon A. Børretzen, Per T.
Myhrer and Torgrim Lauritsen (Stålhane and Conradi).
6.
SEVO, 2004-2007: software evolution, Anita Gupta and Odd Petter
N. Slyngstad (Conradi), with Statoil-IT.
7.
FABULA, 2006-09, mobile learning, Ilari Canovaca Calori, Basit
Ahmed Khan, NN (Divitini).
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
52
SU research projects, part 2
Supported by NFR, user-driven:
8.
9.
10.
11.
SPIQ & PROFIT, 1996-2002: industrial sw process improvement,
Tore Dybå, Torgeir Dingsøyr (Conradi); with Simula/UiO, SINTEF,
Abelia, and 10 companies.
SPIKE, 2003-05: industrial sw process improvement, Finn Olav
Bjørnson (Conradi); with Simula/UiO, SINTEF, Abelia, and 10
companies - successor of SPIQ and PROFIT. Book on Springer.
EVISOFT, 2006-10, empirically-driven process improvement, Vital,
10 companies, Simula & SINTEF, Geir Kjetil Hanssen, NN (Conradi,
Stålhane) – successor of SPIKE etc.
NorskCOSI, 2006-2008: OSS in Europe, IKT-Norge and three
companies, Sven Ziemer, Thomas Østerlie, Øyvind Hauge by IDI
(Conradi).
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
53
SU research projects, part 3
IDI/NTNU-supported:
•
Software security, 2002-06: Siv Hilde Houmb (Stålhane).
•
Component-based development, 2002-06: OSS survey, Jingyue Li
(Conradi).
•
ESE/Empirical software engineering, 2003-07 (SU funds): open source
software, Thomas Østerlie (Jaccheri).
•
KRITT, Sart: Creative methods in education/software and art, 2003-09
(NTNU): novel educational practices, Salah Uddin Ahmed (Jaccheri).
•
MOTUS, 2002-2006 (NTNU), pervasive and cooperative computing, Birgit
R. Krogstie, Eli M. Morken (Divitini), Telenor R&I.
•
GAMES, Computer games, 2007-10,Telenor R&I and IME-faculty, NN1,
NN2, NN3 (Alf Inge Wang).
Supported from other sources:
•
ESERNET, 2001-03 (EU): network on Experimental Software Engineering,
no PhD, Fraunhofer IESE + 25 partners. Book on Springer.
•
Net-based cooperation learning, 2002-06 (HiNT): learning and
awareness, CO2 lab, Glenn Munkvold (Divitini).
•
ASTRA, 2006-09 (EU), awareness and mobile technology, Otto Helge
Nygård (Divitini).
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
54
Ex. EVISOFT: Evidence-based Software
Improvement
• NFR industrial R&D project, 2006-10. NTNU, SINTEF, UiO/Simula,
Vital. 3 PhD stud. (NTNU, UiO), 5-10 researchers, 10 active
companies. NFR funding: 8 mill. NOK/year, covers direct expenses.
• Project manager: Tor Ulsund, Vital ex.Geomatikk.
• Builds on SPIQ (1996-99), PROFIT (2000-02), SPIKE (2003-2005)
• Help (“facilitate”) IT companies to improve, by pilot projects in
each company: e.g. on cost estimation and risk analysis, UML-driven
development, agile methods, component-based software
engineering (CBSE) – coupled with quality/SPI efforts.
• Couple academia and industry: win-win in profile and effect, by
action research.
• Empirical studies – in/across companies and with other projects
• General results: Method book, reports and papers, experience
clusters, shared meetings and seminars
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
55
Project model in EVISOFT
Dissemination
Common projects (generalization)
Dissemination
Company project (pilot project)
Plan
Do
Check
Act
Next company project
Development/implementation project
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
56
Student assignments: linked to ongoing R&D
projects
• Conradi: process improvement, SCRUM,
CBSE / open source, sw evolution. Companies:
Vital, EDB, Opera, Skattedirektoratet.
• Divitini: Coop. technology,awareness. Telenor,
NTNU and pedagogics.
• Jaccheri: open source, software and art,
pedagogics, research methods. Falanx.
• Stålhane: reliability, safety, defect analysis.
Vital, EDB, Opera.
• Wang: Computer games, mobile systems, sw
architecture. Telenor.
R. Conradi: Research in Software Engineering, SU's PhD day, 23 Nov. 2007
57