Agile Software Development and
ISO/IEC Software Quality Standards:
Measuring Economic Benefits and
Calculating Quantitative Yields
Anca-Juliana Stoica, Margus Nael
Royal Institute of Technology, KTH-ITC, Sweden
25th International Forum on Systems, Software, and COCOMO Cost
Modeling
University of Southern California, Los Angeles, CA
November 2-5, 2010
Outline
–
Research Background
• Agility vs. Discipline in Software Development
• ISO/IEC Software Quality Standards
–
–
–
Research Objectives
Research Design
Case Studies
•
•
•
•
•
–
–
Techniques and Features Used
Framework for assessing a project
Metrics and Characteristics
Quality Metrics
Integrated Models for Measuring Economic Benefits and
Calculating Quantitative Yields
Results
Conclusions
The Spectrum of Software
Engineering Methods
Examples of Software Engineering
Methods
• Agile methods:
–
–
–
–
Increased customer satisfaction
Lower defect rates
Faster development times
Solution to rapidly changing
requirements
Examples: XP(Extreme
Programming), Cristal
Methods, ASD(Adaptive
Software Development),
DSDM, FDD, Scrum
• Plan-driven methods:
– Predictability
– Stability
– High assurance
Examples: Cleanroom, PSP,
TSP, CMM methods (Capability
Maturity Models: Software
CMM, Integrated
CMM(CMMI)), ISO/IEC
Software Quality Standards
(9001 and the recent SQuaRE
standards)
Agility vs Discipline
Agility vs. Discipline
Challenge
Balance the two approaches to take
advantage of their strengths in a given
situation while compensating for their
weaknesses
Two suggested solutions
•
•
Software risk management approach
Software quality management
approach,
for structuring projects to incorporate
both agile and plan-driven features in
proportion to project’s needs.
Trends for the 21st Century
Software Systems
• For large – to very large scale software intensive
systems, the Emerging Scalable Spiral process
model
• For small – to medium scale software systems,
hybrid methods, using some other concepts in
addition to software risk management, like:
software quality management.
Our focus here is on small – to medium scale
software systems that need to balance agility
and quality assurance.
ISO/IEC SQuaRE Series
– Software product Quality Requirements and
Evaluation
– SQuaRE series of International Standards is
based on 2 predecessors:
•
•
ISO/IEC 9126 (Software product quality)
ISO/IEC 14598 (Software product evaluation)
– New general reference model consisting of 5
quality divisions:
•
•
•
•
•
Quality Management Division
Quality Model Division
Quality Measurement Division
Quality Requirements Division
Quality Evaluation Division.
Organisation of SQuaRE series of
standards
SQuaRE Common Models
–
SQuaRE general reference model - navigation guide
as a function of user’s task(s)
– Software product LC model – views of internal,
external software quality and software quality in use
during software LC
– Quality model structure –categorisation of software
quality attributes into characteristics, sub-characteristics,
and quality attributes.
Remarks
– Measuring and evaluating characteristics will assure
quality
– Organization must support the project with necessary
equipment and knowledge
– Create Quality Evaluation Project Plan
ISO/IEC SQuaRE
Software Product Quality Views
Research Objectives
– Study if there are any conflicts between the
application of agility principles and the
ISO/IEC SQuaRE series of software quality
standards
– Guidelines on how to apply SQuaRE to agile
(hybrid) software development
– Real-life case studies for proving major
benefits:
• Software quality
• Economic benefits.
Research Design
–
Data collection
•
•
–
Guidelines for applying ISO/IEC SQuaRE
•
•
•
–
Principles
How they apply to a specific project/development
environment
Use of inductive approach
Real-life case studies with/without using SQuaRE
•
•
–
Software project assessment framework based on
agile/plan-driven characteristics
Experience base
Software quality improvements
Economic benefits and other quantitative yields
Data analysis
Real-Life Case Studies
Focus on two separate but similar real-life
software development projects:
1. Project A -project without ISO/IEC SQuaRE
2. Project B -project with ISO/IEC SQuaRE
Applied comparative analysis of projects and
software metrics
Real-Life Case Studies
Projects’ characteristics:
• Modern Web-based small- sized software
business systems
• Stakeholder committment based on Theory W
principles
• Incremental development and delivery
• Balancing agility and discipline
• Software quality management done through the
whole life-cycle
• Use of a tailored software engineering method.
ProjectA/Project B
• To publish fresh
publications, e.g. news
articles from news
providers, via WAP or
GPRS to the customers
of a tecommunication
company
• Publications are delivered
to application by RSS and
server creates pages that
are suitable for mobile
stations to read.
•To publish Web Logs
(Blogs) via WAP or GPRS
to the custommers of a
telecommunication
company
•Publications are
delivered to application by
RSS and server creates
pages that are suitable for
mobile stations to read.
Company vision
–
–
by
–
–
capture a larger market share
become more attractive to customers
developing better software quality products
managing better software development
processes.
Project A
Used techniques:
–
–
–
–
–
–
–
–
–
–
–
–
Feature Driven Development
Small team
Osmotic communication
Spiking, Walking skeleton
Information radiators
Simple design
Access to expert user
Estimation using comparison and
analogy
Frequent delivery
Documentation is not so important
Kick-off meeting
Status reporting
Project A
Home ground polar chart
Project A
Framework
for
project
assessment
Included in the
experience
base
Project B
Used techniques:
–
–
–
–
–
Feature Driven
Development
Small team
Osmotic communication
Spiking, Walking
skeleton
Information radiators
(cont.)
Project B
Used techniques:
–
–
–
–
–
–
–
–
–
Simple design
Acess to expert user
Estimation using
comparison and analogy
Frequent delivery
Kick-off meeting
Status reporting
Code reviews
Measurements of quality
characteristics
Burn charts
Project B
Home ground polar chart
Project B
Framework
for
project
assessment:
Project A Quality Model
Dark colors
mm represent quality
n
characteristics
of the project
Project B Quality Model
Remark: We have two quality models, Project A and Project B. We just have to find a way
to measure it and prove that Project B is better.
Metrics and Characteristics
–
–
–
–
–
Project metrics: Personnel.
Product metrics: Lifetime, Downtime, Number of defects,
Complexity.
Product quality metrics: Mean time to failure, Defect density,
Customer satisfaction.
Other metrics: Lost amount of money to customer, Lost amount
of money to developer.
Quantitative economic yields of improving software quality
Remarks
1. Software quality metrics measured with Research Standard
Metrics (RSM) tool
2. Quantitative economic yields of improving software quality are
calculated (estimated) using a value-based framework.
RSM quality report
Project basic data
(RSM Reports)
Software quality report (1):
Total quality notices
Software quality report (2):
Maintainability(Readability)
Software quality report (3):
Quality notices
Software quality notices
Quantitative yields of improving
software engineering methods
Stoica, A.J. (2007). Exploring Quantitative Yields of Software
Engineering Methods in the Software Life-Cycle. The first IEEE
Computer Society EQUITY International Conference. 19-21
March 2007, Amsterdam, the Netherlands.
Motivation
The problem of exploring quantitative yields of a new
(improved) software engineering method in the software
life-cycle has to be addressed in a systemic manner
– relevant metrics and models have to be selected
– their interrelashionships have to be expressed in a
formal manner to detect possible inconsistencies and/or
incompletenesses.
The Model Systems (MS) concept is applied to
– analyze multiple facets of software development
– derive a value-based framework (VF) for exploring the
quantitative yields of a new (improved) software
engineering method in the software life-cycle.
Value-Based Framework (VF)
Interrelated software system metrics and
models that address:
– Costs (CM)
– Quality (QM)
– Productivity (WM)
– Benefits (BM) from applying better software
project quality management
– Resulting value estimation metrics (VM),
evaluated over the software life-cycle.
Quantitative Models for Estimating
Costs, Benefits, and Value
Applicable for software engineering methods that:
– Improve software quality via structured and facilitated group
reviews
– Certify companies to adhere to regulations for international trade
– Evaluate, qualify, certify suppliers to government agencies
– Perform individual and group-based project management via
technical challenging approaches
– Use sets of criteria designed for supplier selection and
establishment of priorities for internal software process
improvement.
VF Instantiation Example
•
•
•
•
Improving an agile (hybrid) software engineering
method is achieved as a result of applying ISO/IEC
software quality (SQuaRE) standards
Metrics and models are applied to: i) existing, and ii)
new (improved) software development process in order
to calculate all the possible quantitative yields
Improvements to an agile (hybrid) software engineering
process are in similar to applying Software Inspection
Method (SIM)
Guidance of practical use that ISO/IEC SQuaRE quality
standard contains is also quite similar.
Input parameters (Project B)
General metrics
Cost metrics
Benefit metrics
Value metrics
Main results
– The real return on investment is 4700%.
– The real gross benefit is 49701,6 Eur.
– Real net benefit when you apply Software
inspection Method (SIM) 48666,15 Eur.
– Real benefit to cost ratio i.e. the ratio
between benefits and costs is 48.
– Real breakeven point is 24288,90 Eur, or
202,12 hours.
Summary
–
–
–
–
–
–
Our research shows that there are no conflicts between
applying ISO/IEC SQuaRE software quality standards and agile
software development
Economic and quality benefits are demonstrated using a value
framework as well as software metrics and measurements
Simple guide to SQuaRE practical use with real-life example is
presented
Evaluation framework for software system projects to decide on
the balance between agile and plan-driven approaches is
outlined
Experience base for analysing software system projects is
created
Our work is a step forward in reconciling agile approaches and
formal software product quality evaluation that is discussed in
ISO/IEC software quality standards.
Conclusions
–
We showed that creatively using software quality standards in
agile software development is possible for
•
•
•
–
–
combining the best of agility and discipline
creating quality software
generating economic benefits.
During the last years, agile software development has become
a work facet in small and medium software development
organizations, but its lack of theoretical foundations require
practical methods, models, metrics, and/or processes to relate it
to software engineering research and application as well as to
the associated engineering standards.
Our work supports this effort and offers theoretical and practical
support for future advances in educational and software
development organizations.