Software testing research
Ossi Taipale
November 2011
Lappeenranta University of Tech.
Contents
Background
Research process
Research methods
Research projects, research objectives and results:
 Basic research of software testing, ANTI -project
 Reference model of software testing, MASTO project
 Software testing in the cloud, STaaS-project
 Intended quality, STX-project
 Research co-operation




Background, why testing
 During the last decades productivity of Finland has
been higher than in other EU countries, but now the
productivity in the product and service sectors is
slowing down meaning lower growth rate of GDP.
 The key to increase productivity is in the hands of
industry and service sectors that produce or apply ICT.
 Productivity can be increased by creating tangible
products or service products.
 Testing enables product creation.
Background, software testing
research
Top-down approach:
 1. Years 2004–2007 ANTI-project: Basic research of
software testing, affecting factors.
 2. Years 2008–2011 MASTO-project: Software
testing strategy and management. Theses: Software
Testing Strategy and Management and Software
Testing in the Cloud. The reference model of
software testing ISO/IEC 29119 Software Testing
Standard. 1. Vocabulary, 2. Testing Processes,
3. Reporting. Part 4. Methods is under
development. Ossi Taipale is a member of the
WG26.
 3. Years 2011-2014, new STX-project: Testing level
and ”Testing Spice”.
Software testing research
projects
 Top down approach in the research projects :
ANTI-project:
Basic research
of SW testing,
2004 - 2007
ISO/IEC 29119:
SW testing
standard
MASTO-project:
Reference model
of SW testing,
2008 - 2011
SW testing in
the cloud
STX-project:
Intended
quality,
2011 - 2014
ISO/IEC and IEEE :
29119 and Testing Spice,
15504, IEEE1012,
25010
Background, software testing
definition
 We use the definition: Software testing consists of
verification and validation.
 By testing, we try to answer two questions: are we
building the product right and are we building the
right product?
 Verification answers the question: are we building the
product right? Basic verification methods include
inspections, walkthroughs, and technical reviews.
Checklists are used as the tools of verification.
 Validation answers the question: are we building the
right product? Validation activities can be divided into
unit testing, integration testing, usability testing,
function testing, system testing, and acceptance
testing.
Research process
Ontology
Research
philosophy
Epistemology
Beliefs of the
essence of
phenomena
under
investigation
Background
assumptions
Beliefs of
the suitable
method
Selection of
standards and
constructs: e.g.
ISO/IEC 12207,
15504 and 29119
Empirical study
Concrete selections
of the study:
Delphi, survey, and
grounded theory
methods
Research methods
 Mixed methods approach consisting of quantitative
and qualitative analyses (Hirschheim 1985).
 The results of the quantitative and qualitative
analyses are triangulated to increase the validity of
the study. The principle of triangulation means that
more than one method, observer or data set is used
in a study to complement each other and to verify the
findings (Denzin 1978).
Research methods
 The objective of the Delphi method is to achieve the
most reliable consensus of opinion of a group of
experts. Delphi method can be used in finding good
arguments about an ongoing process (Directing the
study).
 The survey method is used to gather information
about feelings, motivations, plans, and beliefs. Tools
for gathering such information are usually
questionnaires or interviews (Fink & Kosecoff 1985).
 Grounded theory (Strauss & Corbin 1990). The
objective of a qualitative study is rather to find or
reveal facts than to prove existing theorems. Strauss
and Corbin (1990) define qualitative research as any
kind of research that produces findings not arrived at
by means of statistical procedures or other means of
quantification.
Some results of the ANTI-project
(2004-2007)
 Hypotheses for improvement, process view:
• Testing ought to be adjusted according to the business
orientation of the OU. Product oriented organizations should
adopt a formal planned testing process. Service oriented
organizations should adopt a flexible testing process that
allows appropriate coordination with their clients.
• Enhanced testability of software components. Consider
testability as an important factor in the selection of
components. Review the testing process of your suppliers.
• Ef ficient communication and interaction between development
and testing seems to reduce costs and improve quality. Product
oriented OUs could develop straightforward processes because
knowledge transfer is predictable. Service oriented OUs could
develop innovative solutions because knowledge transfer varies
according to the customer.
• The early involvement of testing seems to reduce costs and
improve quality. The planning of testing is more
straightforward in product oriented than in service oriented
OUs.
• The risk-based testing strategy helps in avoiding ad hoc
decisions on testing, because the decision on what to test is
based on a predefined testing strategy.
Some results of the ANTI-project
(2004-2007)
 Hypotheses for improvement, knowledge management view:
• Business orientation af fects the testing organization. In
product oriented OUs, the organization model ought to support
repetitive testing tasks that enable development of a deeper
expertise. In service oriented OUs, the organization model
ought to support adaptation to the processes of the customers
that demand broader expertise.
• Business orientation af fects the knowledge management
strategy. OUs ought to adjust their knowledge management
strategy according to the business orientation.
• Business orientation and the knowledge management strategy
af fect outsourcing. Codification of knowledge enables testing
outsourcing.
• Identifying and avoiding barriers and using enablers improve
knowledge transfer.
Some results of the ANTI-project
(2004-2007)
 Associations between testing outsourcing and knowledge
management:
• Outsourcing seems to be more ef fective when independent
testing agencies have enough domain knowledge.
• Outsourcing verification tasks is more dif ficult than
outsourcing validation tasks.
Publications of the ANTI-project
(2004-2007)
1. Finding and Ranking Research Directions for Software Testing, EuroSPI, Budapest,
Taipale, O., K. Smolander, H. Kälviäinen.
2. Cost Reduction and Quality Improvement in Software Testing, SQM, Southampton,
UK, Taipale, O., K. Smolander, H. Kälviäinen.
3. A Survey on Software Testing, SPICE Conference , Luxembourg, SPICE ,Taipale, O.,
K. Smolander, H. Kälviäinen (2006).
4. Factors Affecting Software Testing Time Schedule, ASWEC, Sydney, Australia,
Taipale, O., K. Smolander, H. Kälviäinen (2006).
5. Improving Software Testing by Observing Practice, ISESE, Rio de Janeiro, Brazil,
Taipale, O., K. Smolander.
6. Observing Software Testing Practice from the Viewpoint of Organizations and
Knowledge Management, ESEM, Madrid, Spain, Taipale, O., K. Karhu, K. Smolander.
7. Triangulating Testing Schedule Over-runs from Knowledge Transfer Viewpoint,
Lappeenranta University of Technology, Research Report 104, Taipale, O., K. Karhu, K.
Smolander .
8. Outsourcing and Knowledge Management in Software Testing, EASE, Staffordshire,
UK, K. Karhu, O. Taipale, K. Smolander.
Overview on MASTO-project (20082011)
Jussi Kasurinen
MASTO
 The project was a 3-year academic
joint project conducted by LUT and
Aalto University, funded by Tekes
(Technology and Innovation Agency of
Finland) and a group of companies
and organizations.
 The focus on the project was on
software testing in different
organizational levels.
 MASTO at LUT focused on test
process and organizational
aspects.
 Aalto focused on the testing work
and project-level aspects.
 Researcher exchange, Jussi
Kasurinen – Lund, Sweden, Leah
Riungu-Kalliosaari – Limerick,
Ireland.
Software-producing
organization
Organizational level,
one company has
one set of these
Project-level work,
one company may
Pof these
have several
of these
Projecese
Collection of qualitative and
quantitative data, 65 interviews
Round: Type of Interviewee
interview
organization
role
in the
Interview themes
1st round: Semistructured interview,
12 OUs
Software designer or people
responsible for software design
and architecture.
Design and development methods, Testing
strategy and -methods, Agile methods,
Standards, Outsourcing, Perceived quality
2nd round:
Structured survey,
additional semistructured
questions, 31 OUs,
including 1st.
Manager, test manager or
project leader responsible for
development project or test
process of a product.
Test processes and tools, Customer
participation, Quality and Customer, Software
Quality, Testing methods and -resources
3rd round: Semistructured interview,
12 OUs, same as
1st.
Tester or people responsible
for doing testing in the
development project.
Testing methods, Testing strategy and –
resources, Agile methods, Standards,
Outsourcing, Test automation and –services, Test
tools, Perceived quality Customer in testing
4th round: Semistructured interview,
10 OUs, some new.
Manager, test manager or
project leader responsible for
development project or test
process of a product.
Test policies, Test strategies, Test plans, Testing
work, Software architecture, Delivery models,
New software development concepts
ISO/IEC 29119 Test process
Testing standard and publications
Journal article “Software Test Automation in
Practice: Empirical Observations”
Conference paper “How Test Organizations Adopt
Conference
New Testing Practices and
Methods?”paper: “A Study on Agility and Testing
Processes in Software Organizations”
Covered in article “Software Test Automation in
Practice: Empirical Observations”
Conference paper: “Test Case Selection and
Prioritization: Risk or Design-based?”
Conference paper “A Self-Assessment Framework
for Finding Improvement Objectives
with ISO/IEC
Conference
paper: “Exploring Quality Concepts in
29119 Test Standard”
Software Organizations”
Conference paper “Analysis of Problems in Testing
Practices”
Test strategy constructs (and layers)
Quality and quality objectives
Stability
Reliability
ab
fe
r
Operability
Tr
a
ity
ns
ur
Software
Product
Quality
ai
ilit
y
M
bi
y
lit
om
na
pa
tib
ai
nt
Efficiency
C
These quality attributes
define the quality objectives
for software
c
Se

ilit
y
 According to ISO/IEC 25010 (Software product quality), the
quality in the software product is a composition of several
quality attributes
Quality characteristics
• All at least somewhat
important
• Only in 9 out of 248
(3,6%) assessments
“not important”
• Perceived quality, not
absolute quality!
Functional suitability
4.2
Reliability
4.1
Performance
3.8
Operability
3.6
Security
4.0
Compatibility
3.9
Maintainability
3.5
Transferability
3.3
0.0
1.0
2.0
3.0
4.0
5.0
Important things that affect
perceived quality (Survey 2009)
• Trust between the customer and developer.
– Specifications are distributed freely, things are discussed and more
information is available if needed.
• Existing conformance with the ISO/IEC 29119 model
concepts.
– Workflow has a feedback system which is used, plans and made
decisions are assessed and changes can be made.
• Elaboration of quality
– Everyone in the organization knows what kind of quality is preferred
and tries to achieve that.
Things that do not affect the perceived quality
that much even if they seem to have a big
influence
• End-product criticality
– Obviously a rocket control system is tested more thoroughly than a
mobile game, but preferred quality comes from the domain, not
criticality.
• Development method
– An agile product can be as high quality as a “traditionally” made
product.
– Open source is not necessarily better (or worse) than closed
software.
• Outsourcing
– In large organizations, outsourcing does not seem to affect
perceived quality.
– In smaller organizations it may, but not always.
How do the organizations develop
their test process
• Organizations do not tend to try out new ideas.
• Sporadic development is done when the inconveniences
overcome acceptable losses.
• Even if the test process feedback is collected, it is often
neglected if the process is “good enough”.
Self assessment framework
 Combination of Test Improvement model (TIM) maturity
levels and ISO/IEC 29119 processes.
 Two results:
 General maturity and conformance with the standard model.
 Process improvement objectives to develop test process.
Maturity levels from TIM
Processes from ISO/IEC 29119
Individual assessment of each
process area, development ideas
General maturity/conformance
estimation
Publications from the MASTO
project (2008 -2011)
1. Test Case Selection and Prioritization: Risk-Based or
Design-Based? Jussi Kasurinen, Ossi Taipale and Kari Smolander, ESEM
2. A Self-Assessment Framework for Finding Improvement Objectives with ISO/IEC
29119 Test Standard, Jussi Kasurinen, Per Runeson, Leah Riungu and Kari
Smolander, EuroSPI
3. Software Test Automation in Practice: Empirical
Observations, Jussi Kasurinen, Ossi Taipale, Kari Smolander, AiSE
4. Exploring Perceived Quality in Software Organizations, Jussi Kasurinen, Ossi Taipale,
Jari Vanhanen and Kari Smolander, IEEE
5. Analysis of Problems in Testing Practices, Jussi Kasurinen, Ossi Taipale and Kari
Smolander, APSEC
6. A Study on Agility and Testing Processes in Software Organizations, Vesa Kettunen,
Jussi Kasurinen, Ossi Taipale, and Kari Smolander, ISSTA
7. How Test Organizations Adopt New Testing Practices and Methods? Jussi Kasurinen,
Ossi Taipale and Kari Smolander, TAICPART
8. Exploring the Perceived End-Product Quality in Software-Developing Organizations,
accepted for publication in International Journal of Information System Modeling and
Design, IGI Global, Jussi Kasurinen, Ossi Taipale, Jari Vanhanen and Kari Smolander.
Software Testing in the Cloud
(2010-2013)
Leah Riungu-Kalliosaari
Objectives and background
 Empirical study aimed at understanding how
organizations can successfully use the cloud for
testing
 Cloud computing: A model for enabling convenient, on -demand
network access to a shared pool of configurable computing
resources (e.g. networks, servers, storage, application and
services) that can be rapidly provisioned and released with
minimal management ef fort or service provider interaction
(NIST definition).
 STaaS: A model for software testing whereby testing of an
application is provided as a service across the internet.
 Cloud-based testing is provided on-demand and billed on pay per-use basis, so that the user pays only for the resources they
have used.
Initial observations
 The trend towards cloud services is growing
 Traditional desktop applications  online software services e.g.
SaaS, StaaS
 According to a 2009 Gartner report, cloud computing changes
the way software is delivered and used. This change means that
traditional licensing of software seems to loose ground where
as hiring of software services seems to be on the rise.
 Mixed reactions  Generally positive
 Positive – due to e.g. cost reduction, flexibility, performance,
access to global markets
 Negative – due to concerns regarding e.g. security, domain
knowledge, test data management
 Neutral – mainly explorative approach
Implications on Testing
 With the cloud becoming more common, the quality
expectations are growing.
 New issues and complexities surrounding these services
will need to be addressed.
 Testing will become more important
 Need for testing increases
 Testing is seen as arena of cloud computing where the
barriers to entry are relatively low. Testing in the cloud
simply entails the use of computing resources and models
in the cloud to perform testing. Cloud -based testing is
provided on-demand and billed on pay -per-use basis, so
that the user pays only for the resources they have used.
Testing in the Cloud
3. Testing the cloud
1. The system or
application under test is
available online
2. Testing environments
in the cloud
1a. SaaS
software
Facets of testing in the cloud
1b. Non-SaaS
software
2. Testing infrastructure
and platforms are
hosted in the cloud
(Including
crowdsourcing/Human
as a Service-(Haas))
3. Testing of the cloud
itself
A roadmap towards testing in the
cloud
 Develop understanding of cloud computing
Cloud computing is increasingly becoming a feasible choice for
testing.
 Carr y out pilot projects
We recommend that organizations carr y out pilot projects that
enable them to fully explore the potential benefits.
 Come up with elaborate strategies
Cloud testing vendor s as well as testing and quality assurance
consulting firms will be called upon to of fer advice and direction.
 Enhance team interaction and prepare for complexities
Organizations need to be prepared for additional testing brought
about by the complexities and new requirements for cloud -based
applications and systems.
 Enhance co -operation between research and industr y
These include, among others, application issues e.g. the types of
applications best suited for testing in the cloud; management
issues like how to organize the human resources for cloud -based
testing (e.g. crowdsourcing ); and legal and financial issues such as
the management of test data across dif ferent global jurisdictions
and how to device appropriate pricing models.
Current research problem
 To better understand how cloud based services can be
achieved with intended quality
 Identify the most important quality requirements for cloud -based
services
 Provide recommendations for achieving overall intended quality
of cloud-based services
 Difference between intended and achieved quality
 What are the tradeoffs, and what is the optimal balance.
Publications from the Testing in
Cloud – project (2010-2013)
1. Research Issues for Software Testing in the Cloud, Leah Muthoni Riungu, Ossi
Taipale, Kari Smolander, 2nd IEEE International Conference on Cloud Computing
Technology and Science.
2. Software Testing as an Online Service: Observations
from Practice, Leah Muthoni Riungu, Ossi Taipale, Kari Smolander, Third International
Conference on Software Testing, Verification, and Validation Workshops.
3. Testing in the Cloud: Exploring the Practice, IEEE Software, Leah Riungu-Kalliosaari,
Ossi Taipale, Kari Smolander.
Software testing and development
for intended quality, STX (20112014)
Tero Pesonen
Objective
 To show how software development, software testing and
intended quality depend on one another.
 Traditional software development and service models
 Emerging XaaS (Everything as a Service) architectures,
technologies and service models.
 The project results help the participating companies in
improving the ef ficiency of their quality management and
software testing and hence the ef ficiency of their software
development as a whole.
 Testing techniques
 Testing as a service
Research Problem
OU’s are evaluated through an assessment framework
Intended
Software Quality
Software
Development
•Software
Products
ISO/IEC 12207
ISO/IEC 15504
•New Services
SaaS
ISO/IEC 25010,
Software
Product Quality
Software Testing
ISO/IEC 29119, Testing
Spice, IEEE Std 1012
StaaS
Research Problem
 The framework used as a ”grid” through which each OU is
assessed.
 Intended quality vs. achieved quality.
 Top-down and bottom-up analysis (testing work  project
management  organizational level).
 The assessment framework is defined through international
software engineering standards .
 We need to anchor the assessment to a specific, concrete view
on what quality, software testing and software development
are.
 Based on standards, the model remains coherent throughout
the assessment.
 Project results and usability; contribution to standardization
work.
 The results include improvement proposals for the OU’s.
 Data collection methods include interviews and surveys.
Work division, entire project
 3 year project
 12 months / work package; WP 1 starts 8/2011
 WP1: Requirements for intended quality
 Identifies the quality requirements and attributes that the OU's
perceive as key for achieving the intended quality.
 Defining intended quality, the importance of different quality
characteristics in achieving it, traditional vs. SaaS
 WP2: Building the intended quality
 Achieved vs. Intended quality vs. the software development
process
 Action research
 WP3: Assessment
 How the OU’s can improve their quality management and
software testing to achieve the intended quality more
efficiently?
Contents of work package (WP) 1
 Study units are OU's.
 What are the quality requirements, and hence the quality
attributes, that the OU perceives as key for achieving the
intended quality (level & characteristics)?
 Are some quality attributes more important than others when
determining whether the targeted quality has been achieved?
Do OU's or projects differ here?
 Does SaaS development emphasize different kind of targeted
quality (level, characteristics) to traditional software
development?
 Data collection for the first qualitative study September –
November / 2011 .
 Additional data from other projects also incorporated.
 Data analysis may produce first leads.
 Second data collection round will constitute quantitative
studies based on round 1 observations.
Publications from the STX project
(2011-2014)
1 . Observations on eBusiness implementation capabilities in
heterogeneous business networks", 11th IFIP International
Conference on e-Business, 2011 , Kaunas, Lithuania Pesonen T.,
Smolander K.