TEST PROCESS AND TEST
PROCESS DEVELOPMENT
Jussi Kasurinen, Software Engineering Laboratory
WHAT WERE MASTO AND ESPA?
 ESPA was a 3-year academic project
conducted by LUT and Aalto University,
funded by Tekes and a group of
companies and organizations.
 The focus on the project was on software
testing in all dif ferent organizational
levels.
 MASTO at LUT was the part focusing on test
process and organizational aspects.
 SQUID at Aalto was the part focusing on the
testing work and project-level aspects.
Jussi Kasurinen, Software Engineering Lab. LUT
Information Technology
Software-producing
organization
Organizational level,
one company has
one set of these
Project-level work,
one company may
Pof these
have several
of these
Projecese
TEST STANDARD AND MASTO
PUBLICATIONS
Journal article “Software Test Automation in
Practice: Empirical Observations”
Conference paper “How Test Organizations Adopt
Conference
New Testing Practices and
Methods?”paper: “A Study on Agility and Testing
Processes in Software Organizations”
Covered in article “Software Test Automation in
Practice: Empirical Observations”
Conference paper: “Test Case Selection and
Prioritization: Risk or Design-based?”
Conference paper “A Self-Assessment Framework
for Finding Improvement Objectives
with ISO/IEC
Conference
paper: “Exploring Quality Concepts in
29119 Test Standard”
Software Organizations”
Conference paper “Analysis of Problems in Testing
Practices”
Test strategy constructs (and layers)
SELF-ASSESSMENT FRAMEWORK
 Combination of Test Improvement Model (TIM) ( Ericson et
al. 1997) maturity levels and ISO/IEC 29119 processes.
 Two results:
 General maturity and conformance with the standard model.
 Process improvement objectives to develop test process.
 Here is an example of the self -assessment results from one
case organization:
Maturity levels from TIM
Processes from ISO/IEC 29119
General maturity/conformance
estimation
Individual assessment of each
process area, development ideas
Jussi Kasurinen, Software Engineering Lab. LUT
Information Technology
OTHER MAJOR IMPLICATIONS FOR
INDUSTRY AND RESEARCH
 1 . The test strategy establishes a framework for testing work at the project
level. The following hypotheses promote the development of a test strategy
to address the factor s impor tant for the testing work:
 The development of a test plan can be characterised as applying two stereotypical
approaches. The first approach promotes a design -based approach, in which the testing
work focuses on validating the object under testing. The second approach promotes the
risk-based approach, where the testing work focuses on minimising the potential losses
caused by the object under testing.
 There is only a loose association between development methods and test processes.
The applied development method does not restrict the practical testing work to any
large degree, or require compromises in the test process definitions.
 The most important aspects in the test process which have positive association with
the perceived end-product quality are trust between the customer and producer, a test
process which conforms to the self -optimising processes similarly as defined in the
ISO/IEC 29119 standard and the communication of the preferred quality characteristics
to all of the process stakeholders.
 In the test process resourcing, the organisations have an average of 70% of their self defined ―optimal amount of resources and dedicate on average 27% of the total project
effort to testing. Based on the study results presented in the literature and the survey
data, the test process hindrances are also based on the efficiency factors and test
management, in addition to simple resourcing issues.
Jussi Kasurinen, Software Engineering Lab. LUT
Information Technology
OTHER MAJOR IMPLICATIONS FOR
INDUSTRY AND RESEARCH
 2. The ISO/IEC 29119 test standard is a feasible process
model for a practical organisation with the following
limitations as regards for applicability:
 The standard model can be characterised as being overtly detailed in
the definition of roles and activities. In the practical test
organisations, the boundaries of different levels, processes and roles
are less organised than the model presents.
 The process model is top heavy and places a considerable emphasis
on the organisational aspects of the test process. Based on the
qualitative analysis, the model defines several responsibilities for the
upper management, many of which are performed, in real -life
organisations, at the project-level management or at least not as
systematically as defined in the model.
 The current standard model does not include a roadmap or phased
process for adopting the model. This hinders the applicability of the
model in organisations, as the organisations had difficulties in
creating an approach which their existing process could adopt for the
concepts presented in the standard.
Jussi Kasurinen, Software Engineering Lab. LUT
Information Technology
OTHER MAJOR IMPLICATIONS FOR
INDUSTRY AND RESEARCH
 3. The organisations do not actively try out new test methods
and prefer a status quo in their test process. The following
hypotheses relate to the test process development at the
organisational level:
 The organisations do not test out new testing tools or apply new
testing methods unless they have a significant external incentive to
do so. Based on the qualitative analysis, these incentives are things
like the current state of the existing process or business needs in the
operating domain.
 The organisational test process may have feedback processes in
place to allow continuous development, but in practice, the
organisations tend to disregard the evidence of process enhancement
needs if the existing process still performs at least at an acceptable
efficiency.
 In test process development, the organisations need a way of relating
their existing process to the proposed changes to understand the
objectives, and more pragmatically, the requirements the process
improvement needs to succeed.
Jussi Kasurinen, Software Engineering Lab. LUT
Information Technology
Software Testing in the Cloud
Leah Riungu-Kalliosaari
Background
 Empirical study aimed at understanding how
organizations can successfully use the cloud for
testing
 Cloud computing: A model for enabling convenient,
on-demand network access to a shared pool of
configurable computing resources (e.g. networks,
servers, storage, application and services) that can be
rapidly provisioned and released with minimal
management effort or service provider interaction
(NIST definition)
 STaaS: A model for software testing whereby testing
of an application is provided as a service across the
internet.
 Cloud-based testing is provided on-demand and billed
on pay-per-use basis, so that the user pays only for
the resources they have used
Current Situation
 The trend towards cloud services is growing
 Traditional desktop applications  online software
services e.g. SaaS, StaaS
 Mixed reactions  Generally positive
 Positive – due to e.g. cost reduction, flexibility,
performance, access to global markets
 Negative – due to concerns regarding e.g. security,
domain knowledge, test data management
 Neutral – mainly explorative approach
Implications on Testing
 With the cloud becoming more
common, the quality expectations are
growing
 New issues and complexities
surrounding these services will need
to be addressed
 Testing will become more important
 Need for testing increases
Testing in the Cloud
3. Testing the cloud
1. The system or
application under test is
available online
2. Testing environments
in the cloud
1a. SaaS
software
Facets of testing in the cloud
1b. Non-SaaS
software
2. Testing infrastructure
and platforms are
hosted in the cloud
(Including
crowdsourcing/Human
as a Service-(Haas))
3. Testing of the cloud
itself
What next?
 To better understand how cloud based services
can be achieved with intended quality
 Identify the most important quality requirements for
cloud-based services
 Provide recommendations for achieving overall
intended quality of cloud-based services
 Difference between intended and achieved quality
 What are the tradeoffs, and what is the optimal
balance.
 HOW Through interviews during Autumn
 Collect data to uncover the needs, problems, and
practices requiring improvement
 Learn how OUs develop software, test software and
Questions and comments
 What would you like to achieve during
this project?
 What is most important for your
organization?
 What kind of results?
 What kind of collaboration do you
expect?
 Suggestions/improvement proposals
regarding our research approach so far.
Software testing and development for
intended quality
Tero Pesonen
Objective
 To show how software development, software
testing and intended quality depend on one
another.
 The research area includes, in addition to
traditional software development and service
models, also the emerging XaaS (Everything as
a Service) architectures, technologies and
service models.
 The project results help the participating
companies in improving the efficiency of their
quality management and software testing and
hence the efficiency of their software
development as a whole.
 Testing techniques
Research Problem
Assessment framework through which each OU is evaluated
Intended
Software Quality
Software
Development
•Software
Products
ISO/IEC 12207
ISO/IEC 15504
•New Services
SaaS
ISO/IEC 25010,
Software
Product Quality
Software Testing
ISO/IEC 29119, Testing
Spice, IEEE Std 1012
StaaS
Research Problem
The three-level view to software development:
 Software quality
 Specify the intended quality
 Quality attributes & metrics according to the
ISO/IEC 25010 quality model.
 Software development
 Product or service oriented development
 Traditional or SaaS software
 Software development methods, software life
cycle models
 ISO/IEC 12207, 15504 software development
process improvement
Research Problem
 Software testing
 Ensure that the software development process
produces the targeted quality
 Testing strategy & policy → testing processes
→ testing techniques
 ISO/IEC 29119, IEEE Std 1012,
 Testing process assessment, Testing SPICE
Research Problem
 The framework used as a ”grid” through which each
OU is assessed.
 Intended quality vs. achieved quality.
 Top-down and bottom-up analysis (testing work  project
management  organizational level).
 The assessment framework is defined through
international software engineering standards.
 We need to anchor the assessment to a specific, concrete view
on what quality, software testing and software development
are.
 Based on standards, the model remains coherent throughout
the assessment.
 Project results and usability; contribution to standardization
work.
 The results include improvement proposals for the
OU’s.
 Data collection methods include interviews and
Work division, entire project
 3 year project
 12 months / work package; WP 1 starts 8/2011
 WP1: Requirements for intended quality
 Identifies the quality requirements and attributes that
the OU's perceive as key for achieving the intended
quality.
 Defining intended quality, the importance of different
quality characteristics in achieving it, traditional vs.
SaaS
 WP2: Building the intended quality
 Achieved vs. Intended quality vs. the software
development process
 Action research
 WP3: Assessment
 How the OU’s can improve their quality management
and software testing to achieve the intended quality
more efficiently?
WP 1
 Begins in August 2011
 Quality requirements and the intended quality, software
development and software testing
 Study units are OU's.
 What are the quality requirements, and hence the quality
attributes, that the OU perceives as key for achieving the
intended quality (level & characteristics)?
 Are some quality attributes more important than others
when determining whether the targeted quality has been
achieved? Do OU's or projects differ here?
 Does SaaS development emphasize different kind of
targeted quality (level, characteristics) to traditional
software development?
 The data are collected from the participating companies
 Interviews
 Additionally, supplementary data from the previous
MASTO project may also be used.
 The findings are published as scientific publications
during the WP.