Software initiatives 4
Quality Standards
See Word 97 file
“Software initiatives 4”
1
Quality (BURKS)
• The totality of features and characteristics
of a product or service that bear on its
ability to satisfy stated or implied needs.
Not to be mistaken for "degree of
excellence" or "fitness for use" which meet
only part of the definition.
• There is no single, unambiguous definition.
2
Quality Assurance (BURKS)
• A planned and systematic pattern of all
actions necessary to provide adequate
confidence that the product optimally fulfils
customer's expectations.
• This term can be used generically to refer to all
activities that affect quality in any way.
• It also has a number of more specific definitions,
such as the label assigned to quality assurance
departments and to quality assurance personnel.
3
Quality Control (BURKS)
• The assessment of product compliance.
Independently finding deficiencies assures
compliance of the product with stated
requirements.
• This term can be used generically for the same
concepts as "quality assurance" or more specifically
to specific combinations of inspections and test
steps applied to software projects.
4
Quality Standards (BURKS)
• "The nice thing about standards is that there
are so many of them to choose from",
– Standards are necessary for interworking,
portability, and reusability.
– They may be de facto standards for various
communities, or officially recognised national
or international standards.
5
Standards?
– Every country has a standards organisation.
– In addition, many industries and technologies also have standards
bodies, such as the well known standards published by the IEEE
(Institute of Electrical and Electronic Engineers) and the European
Computer Manufacturers Association (ECMA).
– Many large corporations have in-house standards. Indeed, for
software quality the internal standards used at companies such as
AT&T, IBM, Hewlett Packard, Motorola, and many others are
among the most effective in the world.
– In some cases, the in-house standards pioneered entirely new
concepts such as Motorola's well known Six-Sigma quality
standard.
6
Some standards-issuing
organisations
• Allied Quality Assurance Publications (AQAP) produced
by NATO
• American National Standards Institute (ANSI)
• British Standards Institute (BSI)
• Department of Defence (DoD) in the United States
• Department of Defence (Def) in the United Kingdom
• Deutsches Institut fur Normung (DIN)
• European Computer Manufacturing (ECMA)
• European Space Agency (ESA)
7
Some more standards-issuing
organisations
•
•
•
•
•
•
•
•
Federal Information Process Standards (FIPS)
International Organisation for Standards (ISO)
Institution of Electrical Engineers (IEE)
Institute of Electrical and Electronic Engineering (IEEE)
Japanese Industrial Standards Committee (JISC)
Japanese Union of Scientists and Engineers (JUSA)
Software Publishers Association (SPA)
Various in-house standards and guidelines by specific
companies
8
Oh Dear!
• “On the whole, the software standards
associated with quality have had only a
marginal effect on quality.”
• “Generally speaking, software standards are
based on the subjective views of the
standards committee, and seldom have even
much solid empirical data behind them.”
9
European Software Initiatives
• The combined software population of
Western Europe and the U.K. is somewhat
larger than the software population of the
United States and very energetic in terms of
quality initiatives, standards, conferences,
and Internet communication channels.
• Quality initiatives such as the ISO 90009004 standards, now have global impact.
10
The Good
• The same industries tend to achieve the best
quality results in every industrialised country in
Europe, North America, South America, the
Middle East, and the Pacific Rim.
• These industries share a common characteristic
that their software supports large and complex
physical devices which need close to zero-defect
software in order to operate successfully.
11
The Good






Aerospace manufacturers
Commercial systems software manufacturers
Computer manufacturers
Defence system manufacturers
Medical instrument manufacturers
Telecommunication manufacturers
12
The Bad
• Several industries where software quality
lags in the United States appear to be rather
good in much of Europe: insurance and
banking software, for example.
• In the USA the insurance and banking
software development practices tend to be
somewhat less formal and more casual than
the six industries with the best quality
results.
13
The Bad
• The financial and insurance industries have
also been rushing to get into the
• client/server world,
– where quality is often marginal and even worse
than equivalent mainframe applications.
– By contrast, financial and insurance software in Europe has a
higher probability of rigor during development, more care in
quality control, and a greater use of non-test defect removal
methods such as inspections.
14
The Ugly
• The quality of government software seems
somewhat better in several European countries
than here in the United States.
• Software produced by some of the civilian
agencies of the Federal Government in the United
States tends to lag the private sector in terms of
defect prevention and defect removal operations,
and hence gets deployed with substantially more
errors than commercial norms.
15
The Ugly
• By contrast, government-produced software
in Belgium, Italy, France, Sweden, and the
U.K. is sometimes equivalent to information
systems software produced by public
company or their outsourcing partners.
16
Differences
 Requirements and design approaches are often more
formal in Europe, and some of the European front-end
approaches are technically very sophisticated
 On the whole, formal methodologies are used more often
than in the United States, and much more often in the
information systems and client/server domains where U.S.
practices tend to be rather careless.
 The overall result is that European MIS software projects
often run about 10% to 20% below equivalent U.S.
projects in requirements and design errors.
17
Differences
 The patterns of programming language usage tend to
differ between the U.S. and Europe.
 Logic-based languages such as ALGOL, Prolog, and
Miranda have a higher frequency of use as do
strongly-typed languages such as PASCAL and
Modula-2.
 The usage of Ada83 for civilian projects is far more
common in Europe than in the United States.
 Programming language research is very strong in
much of Europe.
18
Differences
 Usage of commercial software cost estimating tools
and other software project management aids are less
common in Europe than in the United States,
although both sides of the Atlantic are now
experiencing rapid growth in project management
technologies.
 Usage of function point metrics for data
normalisation is roughly equal between the United
States and Europe in terms of frequency of use.
19
Differences
 Membership in various function point users' groups
is growing at about 50% per year in every
industrialised country, while usage of "1ines of
code" is declining by about 10% per year.
 In Europe, large function point associations have
been formed in France, Italy, the Netherlands, the
United Kingdom, and also a multi-country
Scandinavian function point group.
 Function point usage is growing but not yet formally
organised in Austria, Belgium, Germany, Greece,
Portugal, or Spain.
20
Differences
 Although the factor is more important for
productivity studies than for quality studies, the
rather lengthy European vacation periods (running
to more than 30 days per year in some cases) tend to
favour countries with more working days per year,
such as the United States, Japan, and India.
21
Differences
 Although a difficult topic to explore since many
tracking systems don't record it, the amount of
unpaid overtime applied to software projects (i.e.,
unpaid work in the evenings, on weekends, and on
public holidays) appears low in Europe and the
United Kingdom compared to the United States and
Japan.
 Unpaid overtime is so often part of software projects
that it can exert an overall productivity influence
that approaches 20%.
22
Differences
• One of the problems of doing direct international
comparisons between the United States and Europe is that
the metrics are sometimes different.
• A survey of refereed software engineering journals carried
out a few years ago noted that:
 one-third of the articles using "lines of code" were based on
physical lines;
 one-third were based on logical statements;
 the remaining third did not even state the basis of the lines of
code count.
 almost 20% of the articles did not even identify the languages
from which the study was based.
23
Differences
• Of course, there are function point variants too.
• Between the United States and Europe, at least a
dozen function point variations can now be
enumerated.
• On a global basis, however, standard IFPUG
function points have about 90% penetration in
terms of published books, articles, and data.
24
ISO 9000 (BURKS)
• A set of international standards for both quality
management and quality assurance that has been
adopted by over 90 countries world-wide.
• The ISO 9000 standards apply to all types and
sizes of organisations.
• Documentation is at the core of ISO 9000
– Say what you do.
– Do what you say.
– Write it down.
25
ISO 9000
 The standards require:
 a standard language for documenting quality processes;
 a system to manage evidence that these practices are
instituted throughout an organisation; and
 third-party auditing to review, certify, and maintain
certification of organisations.
26
ISO 9000
• Say what you do.
• Do what you say.
• Write it down.
27
Used where?
• The ISO standards are used intermittently in the
United States, Japan and the Pacific Rim, South
America, the Middle East, and Eastern Europe.
• The ISO standards have had far and away their
greatest deployment in Europe and the United
Kingdom.
• Indeed, they are now often required for marketing
products within the European Community (EC).
28
Used where?
• Not every company utilises the ISO standards in
Europe, but the penetration is probably now in
excess of 20% in Western Europe and the U.K. as
opposed to perhaps 3% elsewhere based on
interviews with software managers and QA
personnel.
• A few U.S. companies that produce only domestic
software respond to questions about ISO standards
with a question of their own: "What are they?"
29
ISO 9001 and ISO 9000.3
• ISO 9001 is the primary standard on how
products are built, and a guideline
• ISO 9000.3 explains the ISO approach in a
software context.
30
But ...
• “So far as can be determined, there is not
yet any tangible or perceptible difference in
the software quality levels of companies
which have been certified when compared
to similar companies in the same industries
which have not been certified.”
31
More But ...
• Currently the most tangible aspects remain
– the high costs of the ISO certification process,
– the large volumes of planning and control
documents, and
– the time required to achieve certification.
32
Motorola’s 2 Objections
• ISO certification was used for marketing purposes
with an implied assertion that achieving ISO
certification meant high quality levels regardless of
the lack of empirical data demonstrating that point.
Motorola's Vice President of Quality, bluntly stated
that was false advertising.
• The ISO standards themselves lacked some of the
key quality elements found in Motorola's own
quality standards and could be viewed as a step back
rather than a step forward in terms of quality
methodology.
33
What?
• Some ISO users have observed that it is not
the place of the ISO 9000-9004 standards to
actually improve quality.
• Their purpose is simply to ensure that
effective practices are followed, and are not
in and of themselves a compilation of best
current practices.
34
Does ISO 9001 ...










reduce software defect potentials below 5 per function point?
raise defect removal efficiency levels above 95% ?
reduce the number of had test cases below 15% ?
reduce the rate of "bad fix" injections below 7% ?
increase reliability under field conditions?
increase customer satisfaction levels?
have any affect on data quality?
reduce the number of cancelled or delayed projects?
reduce the incidence of litigation for breach of contract?
reduce the incidence of litigation for poor quality control?
35
Data?
• Software standards groups should consider the methods
through which medical and pharmaceutical standards are
set.
• Although medical standards are not perfect, their concept
is that a therapy should be proven to be effective and its
side effects understood before it becomes a standard.
• It would be a strong statement that software is approaching
the status of a real profession if software standards
included information similar to medical and
pharmaceutical standards on efficacy, counter-indications,
known hazards, and potential side effects.
36
Data?
• Software standards are normally totally published in a way
that is totally devoid of empirical data regarding their
effectiveness when the standards are first promulgated.
• The fact that there may possibly be negative or harmful
side effects from following the standards is usually not
even discussed, although negative side effects are a major
section of medical and pharmaceutical standards.
• Standards organisations should collect data, and should
encourage auditors and assessors to move toward
quantification and empirical results rather than subjective
opinions.
37
TickIT
• TickIT is a method of evaluating software quality that
originated in the late 1980s by the BCS.
• The TickIT approach is based on ISO 9000.3 and is
intended to serve as a consistency check on auditing ISO
standards as applied to software.
• The TickIT concept includes both training and certification
of auditors, who are then qualified to perform ISO audits.
• The TickIT approach includes on-site interviews and
assessments and envisions the production of a number of
quality-related plans and reports.
38
TickIT
• TickIT is widely used throughout the United Kingdom, and
especially so for projects involving the government.
• Since the ISO standards did not originate for software,
there is a strong need to customise or tailor the ISO
standards and audits to the software world. The TickIT
concept is to serve as a translator of ISO philosophy into
the software domain.
• Empirical data on the effectiveness of the TickIT approach
is sparse.
– The TickIT concept and the ISO standards ignore both functional
metrics and defect removal efficiency measurement.
39
SPICE
• "Software Process Improvement and Capability
Evaluation."
• The SPICE approach is a composite software assessment
method based in part on
– the Software Engineering Institute (SEI) capability maturity model
(CMM);
– the Bell Northern Research TRILLIUM assessment approach; and
– the International Standards Organisation (ISO) 9000-9004 standard
series.
40
SPICE
• The SPICE approach is moving toward an international
standard for software process assessments that is intended
to improve the relationship between software contractors
and clients.
• Many of the SPICE concepts are derived from the needs of
large corporations and large government agencies whose
software contracts might run to many millions of dollars
for key contracts, and billions of dollars in aggregate.
41
SPICE
• SPICE is a bold and ambitious program, and like many
bold programs, also has some risks associated with it.
• It is premature to judge the overall impact of SPICE since
it is still evolving.
• The direction is promising, but when so many
organisations are involved in standards the results are often
cumbersome and unsuitable for small projects and small
companies.
42
Software development in 1984
• More than half of large software systems were over 12 months late.
• The average costs of large software systems was more than twice the
initial budget.
• The cancellation rate of large software systems exceeded 35%.
• The quality and reliability levels of delivered software of all sizes were
poor.
• Software personnel were increasing by more than 10% per year.
• Software was the largest known business expense that could not be
managed.
43
Software Engineering Institute
• In order to explore software issues, and especially those
topics associated with defence contracts, the Department of
Defence in 1984 funded a new software research facility.
• This was the essential origin of the Software Engineering
Institute.
• The SEI is located on the campus of Carnegie Mellon
University in Pittsburgh.
• It is a non-profit software research organisation funded
primarily by the US government through the Defence
Advanced Research Project agency (DARPA).
44
SEI Assessment Approach
• The SEI assessment data is collected by means of
on-site interviews using both a standard
questionnaire and also observations and informal
data.
• Once collected, the assessment data is analysed,
and used to place a software organisation on one
of the five plateau of the now well-known SEI
“Capability Maturity Model”
• CMM
45
System Size?
• The SEI assessment approach originally dealt
primarily with the software processes and
methodologies used by very large companies that
produced very large systems.
• It was derived from the best practices used by
leading corporations such as IBM and ITT which
employ from 5,000 to more than 25,000 software
professionals, and which could safely build
systems in excess of 1,000,000 lines of code or
10,000 function points.
46
SEI CMM
Approximate Distribution of U.S. Software Enterprises
Based on the SEI Capability Maturity Model (CMM) Results
SE1 Maturity Level
1 = Initial
2 = Repeatable
3 = Defined
4 = Managed
5 = Optimising
Meaning
Chaotic
Marginal
Adequate
Good to excellent
State-of-the-art
Frequency of
Occurrence
75.0%
15.0%
8.0%
1.5%
0.5%
CMM
• The Software Engineering Institute's model of
software engineering that specifies five levels of
maturity of the processes of a software
organisation.
• CMM offers a framework for evolutionary process
improvement.
• Originally applied to software development (SECMM), it has been expanded to cover other areas
including Human Resources and Software
Acquitition.
48
Focii - key process areas:
Level - 1 Initial
• Heroes
• None.
49
Level 2 - Repeatable
• Project Management –
• Software Project Planning, Software Project
Tracking and Oversight, Software
Subcontract Management, Software Quality
Assurance, Software Configuration
Management, Requirements Management.
50
Level 3 - Defined
• Engineering Process
• Organisation Process Focus, Organisation
Process Definition, Peer Reviews, Training
Program, Inter-group Coordination,
Software Product Engineering, Integrated
Software Management.
51
Level 4 - Managed
• Product and Process Quality
• Software Quality Management, Quantitative
Process Management.
52
Level 5 - Optimising
• Continuous Improvement
• Process Change Management, Technology
Change Management, Defect Prevention.
53
P-CMM
• The SEI assessment program was criticised for
concentrating on too narrow a set of topics, and
ignoring other topics such as personnel,
compensation, hiring practices, and the like.
• In 1995 the SEI filled a notable gap by publishing
a People Capability Maturity Model or P-CMM.
• The lead researcher on the P-CMM was a
cognitive psychologist, Dr. Bill Curtis.
• He had formerly headed the SEI assessment
practice
54
Level 1: initial
• Characterised by random or chaotic development methods
with little formality and uninformed project management.
• Some smaller projects may be successful due to the
capabilities of individual staff members, large systems are
often failures and the overall results are marginal to poor.
• In terms P-CMM, such organisations are asserted to be
deficient in training at both the technical staff and
managerial levels.
55
Level 1: initial
• SEI does not recommend any key process areas for Level 1
organisations on the grounds that they are not sophisticated
enough to deploy methods consistently and will probably
botch things up if they try.
• Moving from Level 1 to Level 2 is the most
difficult transition point.
• For a large company with perhaps 1,000 software
personnel, the move from Level 1 to Level 2 can take from
18 to more than 36 months.
56
Level 2: repeatable
• Level 2 or Repeatable organisations have introduced at
least some rigor into project management and technical
development tasks.
• Approaches such as formal cost estimating are noted for
project management, and formal requirements gathering
are often noted during development.
• Compared to the initial level, a higher frequency of success
and a lower incidence of overruns and cancelled projects
can be observed.
57
Level 2: repeatable
• In terms P-CMM, the Level 2 organisations have
begun to provide adequate training for managers
and technical staff.
• Further, they have become aware of the
importance of professional growth and the need
for selecting and keeping capable personnel.
58
Level 2: repeatable
• The key process areas which SEI recommends for
Level 2 organisations include requirements
management, project planning, resource tracking,
quality assurance, and configuration management
plus subcontract management for projects that
utilise multiple contractors.
• The key process areas for Level 2 of the People
CMM include compensation, training, staffing,
communication, and work environment.
59
Level 2 to 3?
• Moving from Level 2 to Level 3 is not as difficult
as the first move, since organisations have usually
become used to the idea that change will be
necessary.
• However, the same size organisation illustrated
previously with 1000 software personnel, should
not expect to go from Level 2 to Level 3 in less
than a year, and it may take more than 18 months.
60
Level 3: defined
• Level 3 or Defined organisations have mastered a
development process that can often lead to
successful large systems.
• Over and above the project management and
technical approaches found in Level 2
organisations, the Level 3 groups have a welldefined development process that can handle all
sizes and kinds of projects.
61
Level 3: defined
• In terms of the P-CMM, the Level 3 organisations
have developed skills inventories and are now
capable of selecting appropriate specialists who
may be needed for critical topics such as testing,
quality assurance, web mastery, and the like.
• Level 3 organisations should be capable of
effective software reusability programs.
62
Level 3: defined
• The key process areas which SEI recommends for Level 3
organisations include establishing an effective
organisational infrastructure, training of both managers and
technical staff, inter-group co-ordination, and the
utilisation of formal design and code inspections.
• The key process areas for Level 3 of the P-CMM include
career development, competency-based practices, work
force planning, and analysis of the knowledge and skills
that are needed by the organisation.
63
Level 3 to 4?
• Organisations that achieve Level 3 sometimes rest
on their laurels, since Level 3 is the minimum
level needed to bid on some U.S. defence projects.
• Organisations that are pursing "best in class" goals
will find Level 3 to be only a way station.
• For these energetic groups, moving from Level 3
to Level 4 can sometimes be accomplished in less
than 12 months.
64
Level 4: managed
• Level 4 or Managed organisations have established a firm
quantitative basis for project management and utilise both
effective measurements and also effective cost and quality
estimation.
• In terms of the People CMM, the Level 4 organisations are
able to not only monitor their need for specialised
personnel, but are actually able to explore the productivity
and quality results associated from the presence of
specialists in a quantitative way.
65
Level 4: managed
• Long-range predictions of future needs are also part of the
Level 4 P-CMM assertions. In addition, Level 4
organisations would make extensive use of mentoring.
• The key process areas which SEI recommends for Level 4
organisations include quantification of defect levels,
productivity levels, and activity-based costing.
• Level 4 organisations should be approaching 50% levels of
software reuse for both design, code, and testing materials.
• The key process areas for Level 4 of the P-CMM include
mentoring, team building, organisational competency, and
the ability to predict and measure the effect of specialists
66
and teams in a quantitative manner.
Level 4 to 5?
• Moving from Level 4 all the way to Level 5
is not easy to quantify because the actual
criteria for Level 5 are somewhat abstract.
• There is no reason to assume that a
transition from Level 4 to Level 5 should
take more than 12 months.
67
Level 5: optimising
• Level 5 or Optimising organisations are assumed to have
mastered the current state-of-the-art of software project
management and development.
• Such groups are expected to be the pioneers who are
capable of building entirely new methods that advance the
state of the art beyond current levels.
• Although not explicitly stated, Level 5 organisations
should be approaching or exceeding a volume of about
75% of reusable design, code, and test materials.
68
Level 5: optimising
• In terms of the P-CMM, the requirements are an extension
of Level 4 and hence different more in degree than in kind.
• The P-CMM stresses both coaching and rewards for
innovation at Level 5.
• The key process areas recommended for Level 5 include
defect prevention and advancing the fundamental software
engineering and management technologies, plus rapid and
effective technology transfer and deployment of
improvement approaches.
• The key process areas for the P-CMM include
encouragement of innovation, coaching, and personal
69
competency development.
Same again?
• When the SEI capability maturity model was first
published it was asserted that both quality and productivity
levels would improve at the higher levels.
• This is an area where SEI shares the bad habits of the rest
of the software industry. The SEI assertions about quality
and productivity were initially made without any kind of
study or empirical evidence being collected.
• In fact, even in 1997 the SEI assessment approach itself
still does not include the collection of quantitative
information on either productivity rates or quality levels.
• This is a serious flaw in the SEI assessment procedure.
70
Level overlaps
Software delivered (to customer) defects (per function point) at each level of the CMM
Level
1
2
3
4
5
Minimum Average
0.150
0.750
0.120
0.624
0.075
0.473
0.023
0.228
0.002
0.105
Maximum
4.500
3.600
2.250
1.200
0.500
“Six-Sigma” Quality Levels
• The Motorola Corporation has received substantial interest
in its famous “six-sigma” quality program for hardware
and manufactured components.
• The phrase refers to defect densities of 3.4 in a million.
• Since the six-sigma definition is hard to visualise for
software, an alternative approach would be to achieve a
cumulative defect removal efficiency rate of 99.999999%
prior to delivery of a product.
• This method is not what Motorola uses but it helps to
clarify what would have to be done to achieve six-sigma
results for software projects.
72
“Six-Sigma” Quality Levels
• Since the current U.S. average for software defect removal
efficiency is only about 85%, and quite a few software
products are even below 70%, it may be seen that software
producers have some serious catching up to do.
• As it happens, there are a few software products which
appear to have achieved six-sigma quality levels.
• In 30 years Capers Jones has observed four projects that
achieved six-sigma results in their first year of deployment
out of a total of almost 10,000 projects.
73
“Six-Sigma” Quality Levels
• Surprisingly, the prognosis for achieving six-sigma quality
levels for software is fairly good.
• The best of the available technologies today can approach
this level if
– a full suite of defect prevention and
– defect removal operations are carried out synergistically.
• Indeed, for software projects where reuse of certified
materials top 75% by volume, and where formal
inspections are used to augment testing, six-sigma quality
levels should be achieved routinely.
74
“Six-Sigma” Quality Levels
• That does not mean that the software industry is
ready to try.
• There are both sociological and technical reasons
that quality is held back, such as
– the widespread lack of appreciation of the correlation
between high quality and short schedules and
– the lack of commercial reusable materials certified to
zero-defect levels.
75