Independent Validation and
Verification
for the
Pennsylvania Digital
Government Summit
2
Agenda
•
•
•
•
Speaker Introduction
IV&V Session Goals
Material Review
References
3
Introductions
Will Hurley
–
–
–
–
–
–
–
Service Director IT Quality Management
Domain Expert Quality Management
CISCO PIX Firewall for SOHO Cable modem
Fidelity FMR SEPG
Joint Service Common Operating Environment
USAF Electronic Systems Center Data Administrator
CMM Assessment Lead Theater Battle Management
Contingency Planning Source Selection
Early adopter
•
•
•
•
Blended assessments
Practical Software Measurement
Aspects for testing
Java, open source, XML
4
Session Goals
•
•
•
•
•
Definitions and perspective
Key concepts
Case studies
Application and techniques
Final thoughts
5
Definitions
Independent Verification & Validation
A systems engineering process
employing rigorous methodologies
for evaluating the correctness and
quality of the product throughout the
life cycle.
6
Definitions
Independent
– Technically: IV&V prioritizes its own
efforts within specialty
– Managerially: Independent reporting
route to program management,
sponsor or acquiring agency
– Financially: Budget is allocated and
controlled at high level such that IV&V
effectiveness is not compromised
7
Definitions
Verification: The process of determining that an
implementation and its associated data
accurately represent the conceptual description
and specifications.
Validation: The process of determining the
degree to which an implementation and its
associated data accurately represent of the real
world from the perspective of the intended uses
of the system.
Accreditation: The official certification that a
model, simulation, or federation of models and
simulations and its associated data is
acceptable for use for a specific purpose.
8
Definitions
English please!
– Verification – Did I build the thing right?
– Validation – Did I build the right thing?
– Accreditation - Should it be used?
Also, there is an underlying implicit
principle, and its key question:
– Credibility – Should it be trusted?
9
Perspective
Why do IV&V?
– Timely identification of errors and defects (via
associated failures)
– Provide accurate counsel regarding the quality and
readiness of the project to advance
– Deliver product with very high confidence
Staff
Knowledge
Scope
Strategy
Too little testing is a crime,
too much testing is a sin.
10
Perspective
• IV&V activities derive their importance
from the intended use of the project to
which it will be applied.
GRAVE
SUBSTANTIAL
MARGINAL
INSIGNIFICANT
Key thought: Probability of an undesired
event and its consequence drive IV&V
11
Key Concepts
IV&V ≠ Software Quality Assurance
Risk
IV&V candidates
1
SQA Baseline
2
3
N
Features
(Any Project Phase)
N+1
12
Key Concepts
• IV&V employs rigorous
methodologies
– Frameworks
• CMMI, CoBIT, RUP
– Measurement
• Design of Experiments
• ISO/IEC 15939
• Confidence Intervals
– Models, simulations, surveys
13
Key Concepts
• IV&V is effective in preventing ‘defect
leakage’ in all common frameworks
and development lifecycles
Planning & Organization
Requirements
Design
Acquisition &
Implementation
Construction
IV&V
QA
Delivery & Support
Acceptance
Warranty
14
Key Concepts
Importance of IV&V
– Reduce system lifecycle risk
– Increase customer satisfaction/retention
– Increase the long-term success of a project
and the long-term health of an organization
– Reduce Total Cost of Ownership (TCO)
– Improve vendor selection/relation
– Repeatability
– Predictability
– Manageability
– Usability
– Etc.
15
Case Studies
Good projects gone bad
or
Snatching defeat from the jaws of victory.
Plus;
IV&V success stories
You make the call…
16
BT Case Study
• Seeking to reduce costs and deploy a
standard helpdesk system for both
internal and external users, BT evaluated
a number of commercial solutions before
selecting Remedy®.
• More than half of the proposed users
were equipped with 21” screens and
drivers that did not support the Remedy
deployment.
17
XRT Case Study
• XRT, the Cross Retail Workstation, will provide
Financial phone representatives a complete
view of client positions and statutory guidance.
Using this information representatives will cross
sell products and deliver accurate actionable
recommendations. Although technologically
new in all respects (OOA, OOD, distributed
transactions, GUI) the decision was made to
bypass performance testing during the first
release.
• Institutional customers, managed 1000’s of
positions and regularly called representatives
using XRT. XRT screens could take more than
30 minutes to populate the callers positions.
18
Hubble Case Study
• When the Hubble Space Telescope was
being constructed, a decision was made
to save costs by not assembling it on the
ground to check all the alignments before
sending it into space.
• After launch “…NASA announced that
the telescope suffered from spherical
aberration … the problem concerned two
excellent yet mismatched mirrors …”
19
Digital Archive Case Study
• Digital assess under management were doubling
every three months. Projections show managed
storage exceeding 200 terabytes (1,000,000,000,000 bytes –
1000 or 10 = 1 terabyte) However customer commitments
were consistently missed and defect queues
were growing at 2:1.
4
12
• Defect histograms were mapped to ClearCase
file changes. A high correlation between 40% of
defects observed and 5 files was established.
The five files in question, and their functionality,
are currently part of a re-architecture effort to
establish a single archive asset format. A major
client was lost to a competitor due to feature
delay.
20
World Wide Rollout Case Study
• The Project Manager for the world wide rollout
of Oracle Financials, at large financial firm, has
lost confidence in his System Test team. The
team was unable to express how the second
three week cycle of system testing had gone nor
could the team express how future cycles of
testing could ensure a smooth delivery.
• Recovery includes developing standard
methods to track and assess progress and
predictive modeling to establish the reliability of
the system.
21
Predictive Modeling
• Improves counsel regarding quality and
readiness
– More than a gut feel
– Deliver critical systems with very high
confidence
• Part science; part art
– Lots of curves (equations)
– Experience helps
– Context of project required
• Process centric
– Good data; good models
– Everything looks like a nail when you only
have a hammer
22
Modeling Process
Step 1. Log past failures.
Step 2. Organize failure data to support modeling
and context.
Step 3. Copy or enter data using SMERFS3
Step 4. Execute models and plot the failures.
Step 5. Determine curve(s) that best fit project
context.
Step 6. Copy curve data to MS Excel and project
using polynomial trend line.
Step 7. Measure accuracy of curve models.
Step 8. Predict the future using the model.
Step 9. Repeat as necessary.
23
Results
Critical System Failure Predictions
7
Yamada Curve
Critical Failures
6
5
3
2
y = -0.0003x + 0.0218x - 0.6117x + 6.9529
4
3
Schneidewind's
Treatment 2 Curve
Poly. (Yamada Curve)
Poly. (Schneidewind's
Treatment 2 Curve)
2
1
y = -3E-05x4 + 0.0019x3 - 0.0488x2 + 0.5064x - 0.2344
0
1
3
5
7
9 11 13 15 17 19 21 23 25
Interval (7 days)
24
Analysis
• Predicting 13.3 critical faults remain in the
system.
• 5 additional weeks should allow us to remove
approximately 4.4 more critical faults.
• For the next testing period of 7 days, the
probability of operation without a critical failure
is 0.4.
• For now we’ll use Yamada’s curve for
prediction.
– Software only models tend to under predict due to
differences in software and hardware reliability.
– Earlier Yamada models delivered accurate measures
with statistical strength.
– Details
Standard Deviation
95% Confidence Interval
1.59
Total Number of Faults (TNOF)
(25.0, 64.5)
TNOF Remaining
(0.0, 39.5)
25
Application
• How do we start?
– Start with a know high priority problem
– Develop relationships with one or
more IV&V contractors
– Build-out a Probability Risk Factors
table and apply
• Repeat the process for each domain or
phase
26
Consequence of System Failure
Application
IV&V
Grave
IV&V
Substantial
IV&V
Marginal
96
Insignificant
16
32
64
128
250
Total Likelihood of Failure based on Analysis.
High Risk - IV&V Required
Intermediate Risk - Evaluate for IV&V
27
Application
Who’s on the team?
– Phase and scope determines team
composition and size
• Minimum team must include
– Senior Sponsor
– IV&V Lead
– Two domain experts (one if Project Lead is trained)
Key thought
– Executive sponsorship is required to champion
both the project and the findings and
recommendations developed by the team
28
Application
What dose it cost?
– Plan on 5 to 7 person months per 1K
Function Points
• Don’t use FPs Java users can multiply FPs
by 68.
Key thought
– Block IV&V activities throughout the
lifecycle to achieve highest ROI
29
Final Thoughts
• IV&V is a powerful and proven
approach to reduce delivery risks
• Executive\Senior management
support is essential
• Proper planning prevents poor
performance
30
Final Thoughts
• Quality of results increase with
organizational process maturity
• IV&V offers stakeholders impartial
evaluations and recommendations
as to how to best proceed in difficult
situations
• Schedule Slips
• Cost Increases
• Project Termination
31
Final Thoughts
• “The leading driver for cost savings
was internal process improvement,
not the vendor resource costs.”*
Source: Ventoro Offshore Outsourcing Research Report, Oct 11, 2004
32
References
• Dr. Linda H. Rosenberg, Software Quality
Assurance, NASA V&V presentation
10/2002
• Rogers, McCaugherty and Martin, Case
Study of IV&V Return on Investment
(ROI), NDIA presentation, 11/2000
• Hurley, Predictive Failure Arrival
Modeling to Support System Readiness,
Ajilon Presentation 2005
• Ventoro Offshore Outsourcing Research
Report, Oct 11, 2004
33
Thank you for attending.
Will Hurley
Service Director
IT Quality Management
800.654.2748