Experimental
Software
EngineeRing
NETwork
Empirical Methods and Studies in
Software Engineering Experiences from ESERNET
Reidar Conradi and Alf Inge Wang (Eds.)
Ch.0: About this book
Alf Inge Wang and Reidar Conradi (Eds.)
Norwegian University of Science and Technology (NTNU),
NO-7491 Trondheim, Norway
alfw@idi.ntnu.no / conradi@idi.ntnu.no
The book “Empirical Methods and Studies in Software Engineering – Experiences
from ESERNET” is a result of the ESERNET project. ESERNET is a thematic
network project in 2001-03 in the European Union’s 5th Framework Programme under
contract number IST-2000-28754, see www.esernet.org. It has the ambition, in
cooperation with related activities, to gradually change the mentality of software
engineers and their organizations towards systematic empirical studies, for the
purpose of long-term learning. The overall goal is therefore to collect, systematize
and disseminate relevant and valid insight. ESERNET is lead by IESE in
Kaiserslautern (Germany). The project responsible is Dieter Rombach, with project
manager Christian Bunse. ESERNET has five founding members (contractors):
 Blekinge Institute of Technology (BTH), Sweden: Claes Wohlin and Kennet
Henningsson.
 European Software Insitute (ESI), Spain: Elixabete Ostolaza and Elisa Gallo.
 Fraunhofer Institute for Experimental Software Engineering (IESE), Germany:
Christian Bunse, Andreas Jedlitschka , Markus Nick, and Holger Westing.
 Norwegian University of Science and Technology (NTNU), Norway: Reidar
Conradi, Letizia Jaccheri, Tor Stålhane, and Alf Inge Wang.
 The Technical Research Centre of Finland; Finnish National Institute of
Technology (VTT), Finland: Toni Sandelin and Matias Vierimaa.
Sodalia in Trento (Italy) was originally a partner, but decided to withdraw from the
ESERNET consortium due to an internal reorganization activity.
In addition to the contractors, ESERNET has the following 22 participating
members, with contact person indicated:
 DaimlerChrysler AG, Germany: Frank Houdek.
 DELTA Danish Electronics, Light & Acoustics, Denmark: Jørgen Bøegh.
 Engineering Ingegneria Informatica S.p.A., Italy: Stefano de Panfilis.
 Fraunhofer Center for Experimental Software Engineering, Maryland (FC-MD),
USA: Victor R. Basili.
 FZI Forschungszentrum Informatik, Germany: Thomas Genssler.
 Lund University, Sweden: Martin Höst.
 MARKET MAKER Software AG, Germany: Martin Verlage.
 methodpark Software AG, Germany: Christian Knuevener.
 Politecnico di Torino, Italy: Maurizio Morisio
 PSIPENTA Software Systems GmbH, Germany: Torsten Sander.
 Robert Bosch GmbH, Germany: Eberhard Hübner.
 Simula Research Laboratory/University of Oslo, Norway: Dag Sjøberg.
 Softlab, Germany: Wolfgang Koch.
4









SOLID Information Technology, Finland: Janne Järvinen.
Universidad Politécnica de Madrid, Spain: Natalia Juristo.
Università degli Studi dell´Insubria, Italy: Sandro Morasca.
Università degli Studi di Bari, Italy: Giuseppe Visaggio.
Università degli Studi di Roma "Tor Vergata", Italy: Giovanni Cantone.
University of Calgary, Canada: Yingxu Wang.
University of Castilla La Mancha, Spain: Mario Piattini.
University of Kaiserslautern, Germany: Dieter Rombach and Marcus Ciolkowski.
University of Strathclyde, UK: Marc Roper and Murray Wood.
Of the 22 participating members, 8 are industrial and 2 come from outside Europe.
Many of the partners and members already work together.
5
Table of Contents
Ch.0: About this book ........................................................................... 3
Ch.1: Introduction ................................................................................. 6
Ch.2: Empirical Research Methods in Software EngineeringError! Bookmark not defined.
Ch.3: Challenges and Recommendations when Increasing the
Realism of Controlled Software Engineering ExperimentsError! Bookmark not defined.
Ch4. Empirical Studies in ESERNET Error! Bookmark not defined.
Ch.5: Software Engineering Knowledge RepositoriesError! Bookmark not defined.
Ch.6: Using Empirical Studies during Software CoursesError! Bookmark not defined.
Ch.7: Practical Experiences in the Design and Conduct of Surveys
in Empirical Software EngineeringError! Bookmark not defined.
Ch.8: Post Mortem - An Assessment of Two ApproachesError! Bookmark not defined.
Ch.9: Evaluating Checklist-Based and Use-Case-Driven Reading
Techniques as Applied to Software Analysis and Design
UML Artifacts .......................... Error! Bookmark not defined.
Ch.10: Effectiveness of Code Reading and Functional Testing with
Event-Driven Object-Oriented SoftwareError! Bookmark not defined.
Ch.11: Experimentation with Usage-Based ReadingError! Bookmark not defined.
Ch. 12: Functional Testing, Structural Testing and Code Reading:
What Fault Type do they Each Detect?Error! Bookmark not defined.
Ch. 13: COTS Products Characterization: Proposal and Empirical
Assessment ............................... Error! Bookmark not defined.
Ch.14: Reuse Based Software Factory Error! Bookmark not defined.
Ch.15: Appendix - Glossary ............... Error! Bookmark not defined.
Ch.1: Introduction
Reidar Conradi and Alf Inge Wang (Eds.)
Norwegian University of Science and Technology (NTNU),
NO-7491 Trondheim, Norway
conradi@idi.ntnu.no / alfw@idi.ntnu.no
1
Motivation
Our societies crucially depend on high-quality software for a large part of their
functions and activities. That means that software researchers, managers, and
practitioners have to be familiar with what software technologies to use for which
purpose. This again means that we have to systematically study and collect evidence
on what technologies (UML etc.) will work – or not work – with what effects in
which contexts (application areas, project models, societal/technological systems
etc.).
The field of software engineering and software technology is developing very fast.
Perhaps as a consequence, there is seldom enough interest or opportunity for
systematic investigation of how the underlying technology will actually perform. That
is, we constantly introduce new concepts, methods, techniques and tools – or change
existing ones – without much scientific reflection. But how can we learn and improve
without systematic studies?
Systematic empirical studies in software engineering first of all assume wellproven research methods to plan and carry out investigations, and to collect relevant
data. Thereafter comes data analysis – quantitative and/or qualitative – and
discussion. Finally comes generalisation for later reuse and learning, and follow-up
activities such as changed work practices and dissemination through scientific
articles, technical reports and experience bases. In other words, we apply the scientific
method [1]! Common study types are controlled experiments, case studies,
postmortem analyses, and surveys.
Such a study usually starts with some hypotheses to be tested (verified or falsified),
based on some previous insight or observations, or some external need. The
hypotheses (“goal”) will set the premises for what kind of activities and metrics that
are relevant. For instance, if the goal is to improve reliability, we may do a case study
as part of an existing project. A suitable development activity as part of the study may
be to try out (improved) inspection techniques for e.g. UML design artifacts. A partial
metrics for this is the number and type of defects in different phases or subsystems,
with data coming from previous studies and the recent one. However, a persistent
problem in our field is lack of previous data, i.e. no solid “baseline” to compare with.
All this means that there is a need for improved and better qualified empirical
methods. But most importantly, there is a need to learn about and to apply such
methods in practice, and to spread the results. That is, we should promote an
empirical approach to our field of software engineering. The present book aims to
7
contribute to this goal. It has six chapters describing different kinds of empirical
studies, and methods for performing such studies. For instance, chapter 2 describes
the common study types, and chapter 3 describes the ESERNET “template” for
planning, performing, analysing, and reporting empirical studies. The book has seven
more chapters where actual studies are described and discussed. These studies mainly
deal with inspections, testing, object-oriented techniques, and component-based
software engineering.
2 ESERNET, Experimental Software Engineering Research
NETwork
ESERNET has been a thematic network project in 2001-03 in the European
Union’s 5th Framework Programme under contract number IST-2000-28754, see
www.esernet.org. It has the ambition, in cooperation with related activities, to
gradually change the mentality of software engineers and their organisations towards
systematic empirical studies, for the purpose of long-term learning. The overall goal
is therefore to collect, systematise and disseminate reliable insight. This type of
knowledge will serve as a scientific base for assessing, understanding, changing,
innovating, and using software technologies. This task assumes a joint effort between
academia, technology providers, software developers, and possibly end-users. One
step in this direction is the current ESERNET Method and Experience Book to
convey the insights from the ESERNET project. The target audience of the book is
software engineers and managers, and corresponding researchers, teachers and
students.
ESERNET is lead by IESE in Kaiserslautern (Germany). The project responsible is
Prof. Dieter Rombach, with project manager Dr. Christian Bunse. ESERNET has p.t.
four other founding members: VTT Electronics in Oulu (Finland), NTNU in
Trondheim (Norway), Blekinge Institute of Technology or BTH in Ronneby
(Sweden), and ESI in Bilbao (Spain). Sodalia in Trento (Italy) was a sixth founding
member, but has later withdrawn. Per June 2003, ESERNET has 27 participating
members (including the five founding members). Of these, 8 are industrial and 2
come from outside Europe. Many of the partners and members already worked
together before the project.
ESERNET has actively cooperated with and brought synergy to many other
initiatives and projects in software engineering, as illustrated by Fig. 1 below:
8
SMEs
SW-Intensive
Industries
Applied SE
Research
SE Education, Training,
and Conferences
ESERNET
users
ESERNET
virtual network organisation
ISERN
PROFES, CORONET, VASIE,
network PERFECT, CLARIFI
ESPINODEs,
REBOOT
PROFIT
ESSI/PIEs
SE initiatives and projects
(company-wide, national,
European, global)
...
Fig. 1. ESERNET and related initiatives and projects
In Fig. 1, SW means software and SE means Software Engineering. ISERN is the
International Software Engineering Research Network, established in 1992, see
www.iese.fhg.de/ISERN. PROFES, PERFECT, REBOOT, CORONET, and
CLARIFI have been EU-supported software engineering projects in the 1990s (see
www.cordis.lu). PROFIT is a Norwegian industrial project on software process
improvement (SPI) in 1999-2003 (www.geomatikk.no/profit). VASIE stands for
Value Added Software Information for Europe, and is an information and experience
base for EU-projects maintained by ESI in Bilbao, see www.esi.es/VASIE/vasie.html.
ESPINODEs and ESSI/PIEs encompass over 400 small EU-projects on SPI for Smalland Medium-sized Enterprises (SMEs), many registered by VASIE. Lastly, to
promote exchange of ideas and results, ESERNET has supported a researcher
exchange program among its members.
3
ESERNET goals, workpackages, results, and empirical studies
The actual project goals are:
 Establish an infrastructure for a collaboration network.
 Evaluate software engineering practices and package the gained experience for
reuse.
 Use Experimental Software Engineering (ESE) methods to facilitate the systematic
introduction of new practices.
 Accelerate community-wide learning in software engineering.
 Foster collaboration and exchange within and between industry and research.
9
 Create synergy from existing projects, networks, and organisations.
 Initially focus on inspections, testing, object-oriented technologies, and
Component-Based Software Engineering (CBSE).
ESERNET results and deliverables include internal reports, four open public
workshops, a web portal (via the mentioned www.esernet.org) with relevant links and
including a knowledge base of performed empirical studies, and of course this method
and experience book.
ESERNET has applied especially four kinds of empirical studies: post-mortems
and partly surveys (often in industry), controlled experiments (e.g. using students),
and industrial studies (e.g. real case studies in industry). As described in chapter 4,
such studies are often done in the following sequence: First, post-mortems or surveys
are used to pre-analyse practical experience by using a specific software technology
or process. Then, controlled experiments are used to test out more concrete
hypotheses about the chosen study object (e.g. novel inspection techniques), often
using university students. Lastly, case studies are used to investigate refined and
adapted hypotheses in an industrial setting. Overall methods to support such studies
are discussed in chapters 2 and 3, and lessons-learned and recommendations from
using two such methods stand in chapters 6 and 7.
The term Experimental Software Engineering or ESE covers six phased
activities, taken from the Quality Improvement Paradigm (QIP) [2]:
1-3. Characterise/define/plan: Make an explicit improvement goal and a plan to
try out some new process or technique. Extra training may be given and
measurements should be planned.
4. Operate (do) the study, observe the effects, and collect the specified data.
5. Analyse/interpret the results: analyse and generalise.
6. Present/package: package and disseminate for reuse, often in a
web/experience base.
The QIP principle on project vs. corporate learning, using a double improvement
loop each with the above six activities, is illustrated by Fig. 2 below. As mentioned,
the ESERNET project is supporting its own experience and knowledge repository to
support even cross-corporate learning, see chapter 5.
10
Corporate Learning
Package and
store experience
Analyze
results
Set goals
Execute
Project
Learning
Characterize
and understand
Provide process
with feedback
Choose processes,
methods, tools,
and techniques
Analyze
results
Fig. 2. Project vs. organisational learning in QIP
4
About the book content
As mentioned, the book reports on method and experimental work in the project. It
is organised in four parts, numbered I-IV.
Part I contains a short introduction by the book editors, as chapter one.
Part II contains six method chapters, numbered 2-7. Chapters 2, 3 and 4
describe general and specific methods for running empirical studies. Chapter 5 deals
with supporting an experience and knowledge base for such work. Chapters 6 and 7
describe experience by using two empirical methods, respectively controlled
experiments and surveys, from chapter 2. The six method chapters are:
Chapter 2 contains an introduction to “Empirical Research Methods in Software
Engineering” by Claes Wohlin, Martin Höst, and Kennet Henningsson from Blekinge
Institute of Technology and Lund University (Sweden).
Chapter 3 is about “Challenges and Recommendations when Increasing the
Realism of Controlled Software Engineering Experiments” by Dag I. K. Sjøberg,
Bente Anda, Erik Arisholm, Tore Dybå, Magne Jørgensen, Amela Karahasanović,
and Marek Vokáč from Simula Research Laboratory (Norway).
Chapter 4 explains “Empirical Studies in ESERNET” by Toni Sandelin and Matias
Vierimaa from VTT Electronics (Finland).
11
Chapter 5 discusses how to establish, use and maintain “Software Engineering
Knowledge Repositories” by Andreas Jedlitschka and Markus Nick from Fraunhofer
Institute for Experimental Software Engineering (IESE) in Kaiserslautern (Germany).
Chapter 6 explains how to “Using Empirical Studies during Software Courses” by
Jeffrey Carver, Letizia Jaccheri, Sandro Morasca, and Forrest Shull from (in order of
appearance) University of Maryland (USA), Norwegian University of Science and
Technology (Norway), Università degli Studi dell'Insubria (Italy), and Fraunhofer
USA Center for Experimental Software Engineering Maryland (USA).
Chapter 7 sums up “Practical Experiences in the Design and Conduct of Surveys in
Empirical Software Engineering” by Marcus Ciolkowski, Oliver Laitenberger, Sira
Vegas, and Stefan Biffl from (in order of appearance) University of Kaiserslautern
(Germany), Droege & Comp. GmbH (Germany), Universidad Politécnica de Madrid
(Spain), and TU Wien (Austria).
Part III contains seven experience chapters, numbered 8-14. Chapter 8 deals
with using two variants of post-mortem analyses in industrial case studies. Chapters 9,
10, 11 and 12 all report on university experiments with novel inspection techniques
and partly testing, applied to either object-oriented designs (in UML) or programs (in
Java). Chapters 13 and 14 report, respectively, on a university experiment and an
industrial case study with CBSE. Thus, all the topics from the original ESERNET
work plan are present. The seven experience chapters are:
Chapter 8 reports on a study on “Post Mortem - An Assessment of Two
Approaches” by Tor Stålhane, Torgeir Dingsøyr, Geir Kjetil Hanssen, and Nils Brede
Moe from Norwegian University of Science (Norway) and Technology and SINTEF
(Norway).
Chapter 9 reports on “Evaluating Checklist-Based and Use-Case-Driven Reading
Techniques as Applied to Software Analysis and Design UML Artifacts” by Giovanni
Cantone, Luca Colasanti, Zeiad A. Abdulnabi, Anna Lomartire, and Giuseppe
Calavaro from University of Rome at Tor Vergata and Rational Software (Italy).
Chapter 10 reports on an experiment with “Effectiveness of Code Reading and
Functional Testing with Event-Driven Object-Oriented Software” by Giovanni
Cantone, Zeiad A. Abdulnabi, Anna Lomartire, and Giuseppe Calavaro from
University of Rome at “Tor Vergata” and Rational Software (Italy).
Chapter 11 reports on “Experimentation with Usage-Based Reading” by Thomas
Thelin, Magnus Erlansson, Martin Höst, and Claes Wohlin from Lund University and
Blekinge Institute of Technology (Sweden).
Chapter 12 reports on “Functional Testing, Structural Testing and Code Reading:
What Fault Type do they Each Detect?” by Natalia Juristo and Sira Vegas from
Universidad Politécnica de Madrid (Spain).
Chapter 13 reports on “COTS Products Characterization: Proposal and Empirical
Assessment” by Alessandro Bianchi, Danilo Caivano, Reidar Conradi, Letizia
Jaccheri, Marco Torchiano, and Giuseppe Visaggio from University of Bari (Italy),
Norwegian University of Science and Technology (Norway), Politecnico of Turin
(Italy).
Chapter 14 reports on a study on “Reuse Based Software Factory” by Manuel
Prego from European Software Institute Bilbao (Spain).
Part IV with chapter 15 contains an Appendix with a glossary, assembled by the
book editors.
12
NTNU, Trondheim, 13 May 2003
On behalf of the ESERNET project,
Reidar Conradi and Alf Inge Wang, coordinating editors.
5
References
[1] Victor R. Basili, "The Experimental Paradigm in Software Engineering", in H. D. Rombach,
V. R. Basili, and R. W. Selby (Eds.): Experimental Software Engineering Issues: Critical
Assessment and Future Directives, Springer Verlag, Lecture Notes 706 in Computer
Science, 1993, 261 p. From International Workshop, Dagstuhl Castle, Germany, Sept. 1992.
[2] Victor R. Basili and Gianluigi Caldiera: “Improving Software Quality by Reusing
Knowledge and Experience”, Sloan Management Review, 37(1):55-64, Fall 1995.