Notes_011 Good Enough Quality
SE 3730 / CS 5730 – Software Quality
1 Philosophy of Quality

We never know the quality of a product for certain.

We make conjectures about the quality of a product.

A good conjecture is falsifiable but NOT provable.

Although a conjecture is not provable, it can be supported.

Testing is an attempt to refute or support that good quality exists.
2 Factors, Thoughts and more Philosophy about Quality

Market Driven software is looking for the shortest path to shipping features.

The first company out with a major product/tool can set the price (and make a killing) and/or
flood the market and make it hard for the second company to sell anything.

Quality of software may have very little to do with early success of a company.

“How good is good enough?”, Collins (1994)

Most/All commercial software contains defects at delivery time. Seems like most companies
are following “good enough to quality to ship”.

The “good enough quality“, is accepting from the beginning of a project, that we will be
shipping software with defects, and many of these defects will be known to us.

The skill comes in shipping the product with the “right defects”.

“Is good enough for me good enough for you?”, “Is it good enough for your customers?”

“Is it good enough to survive, but not good enough to succeed?”

Why leave bugs in the code?
o It shortens the release cycle – sometimes makes the release cycle finite rather than
infinite.
©2011 Mike Rowe
Page 1
3/15/2016 2:47 AM
From article by James Bach, IEEE Computer Society (1997), and late night discussions
Notes_011 Good Enough Quality
SE 3730 / CS 5730 – Software Quality
3 Do we have Good Enough Quality (GEQ)?
3.1 Four GEQ Factors:




Benefits
Problems
Quality
Logistics
Designed to help put product quality into perspective.
3.2 Assess the benefits of the product.

Identification – What are the known benefits or potential benefits for the stakeholders of the
product?

Likelihood – Assuming the product works as designed, how likely are stakeholders to realize
each benefit?

Impact – How desirable is each benefit to the stakeholders?

Individual criticality – Which benefits, all by themselves, are completely indispensable in the
product?

Overall benefit – As a whole, and assuming no problems, are there sufficient benefits for
stakeholders.
Question: If the above analysis is not positive – why are you doing the project?
3.3 Assess the problems of the product

Identification – What are the problems or potential problems for stakeholders of the product.

Likelihood – How likely are stakeholders to experience each problem?

Impact – How damaging is each problem to stakeholders? Are there workarounds?

Individual criticality – Which problems, all by themselves are completely unacceptable?

(Safety) – Are there human or property safety issues?
©2011 Mike Rowe
Page 2
3/15/2016 2:47 AM
From article by James Bach, IEEE Computer Society (1997), and late night discussions
Notes_011 Good Enough Quality
SE 3730 / CS 5730 – Software Quality

Overall problems – Are there too many non-critical problems?
Question: Are there problems that would prohibit shipping the product?
3.4 Assess product Quality

Overall quality – Do the benefits appear to outweigh problems?

Margin of safety – By how much do we need benefits to outweigh problems.

True value = Measured value
+
Error in Measurement
Question: Is the assessed estimate of quality, with margin of safety, high enough?
3.5 Assess the logistics of improving the product (or holding out for something better)

Strategies – What strategies can be used to improve the product?

Capabilities – How able are we to implement those strategies?

Costs – How much cost or trouble are improvements? Is this the best use of resources ROI,
ROA, IRR, payback periods, sales per employee, etc. Does it make sense to spend the money
here or someplace else?

Schedule – Can we ship now and improve later? Can we ship improvement in an acceptable
time? There may be critical events for which we need the product – drop dead dates.

Benefits – How specifically will the product improve? Are there any side benefits to improving
the product before or after shipping (delay shipping: better morale in shipping a quality item;
ship now: more money in upgrades or selling maintenance contracts that include later
upgrades)?

Problems – How might improvements fail? – in fixing something can we introduce more bugs,
will it hurt morale, starve other projects for resources?
©2011 Mike Rowe
Page 3
3/15/2016 2:47 AM
From article by James Bach, IEEE Computer Society (1997), and late night discussions
Notes_011 Good Enough Quality
SE 3730 / CS 5730 – Software Quality

(Logistics) – Are there barriers to upgrading customers to newer, less buggy or more beneficial
versions of the software? DB schema changes, hardware, conversions, … What are the costs to
us and to customers? If an upgrade is too expensive or hard a customer may consider switching
to a new vendor!
Question: How hard is it to improve and redeploy the product?
4
GEQ Perspective
Six major perspectives to be considered when making an assessment.
1. Stakeholders – Whose opinion about quality really matters? (project team’s, customers’,
competitors’, courts of law, the press, etc.)
2. Critical purpose – What must be achieved? (for a startup it might be immediate survival; others
may have longer scope, like market share (kill off competition), market growth, customer
satisfaction, profitability, etc.).
3. Time Frame – How time-sensitive is the quality decision? (immediate, long-term, event based
(trade shows, major customers drop dead date, …) ).
4. Alternatives – How does product compare to its alternatives?
5. Consequences of failure – What if quality is not good enough? Do we need a plan B? If a
startup fails with its first product it may not have a second chance.
6. Quality of assessment – What is our confidence in our assessment?
5
Assumptions
1. We are dealing with a very complex environment, it is not just the software we are developing,
but
a. our customers expectations (which we may or may not know for sure),
b. competitors that we may have only third hand knowledge of (often you find out what
competitors are doing by talking with your customers and seeing what your competitors
are trying to sell them).
c. Third party software tools
2. Everything has a cost, what we want will generally always exceed what we can afford.
©2011 Mike Rowe
Page 4
3/15/2016 2:47 AM
From article by James Bach, IEEE Computer Society (1997), and late night discussions
Notes_011 Good Enough Quality
SE 3730 / CS 5730 – Software Quality
3. Quality is quantitative and qualitative. There are big bug and small bugs, and there are big
benefits and small benefits.
4. Achieving excellent quality is not easy in complex software. We need to solve a lot of hard
problems, make trade-offs, and resolve contradictory motivations and values.
5. Software Engineering methods are useful to the extent that we keep the above assumptions in
view.
6
Summary
Our task is NOT to eliminate all defects, but to understand the defects and benefits enough to fix the
right problems and deliver the right benefits.
7 In Summary:
To claim that a product is good enough, we must agree that:
1. It has sufficient benefits.
2. It has no critical problems.
3. The benefits sufficiently outweigh the problems (many small problems may make even large
benefits unusable).
4. Understand whether further improvements would be more harmful than helpful (this could
be interpreted as would taking resources away from another project be economically
unfavorable).
If any of the above is not meet, than we do not have good enough quality.

Benefits and problems are not necessarily opposites.

If we have no problems, we still may not have a product with sufficient benefits to be
successful.
©2011 Mike Rowe
Page 5
3/15/2016 2:47 AM
From article by James Bach, IEEE Computer Society (1997), and late night discussions