Independent Verification and
Validation
Ensuring Project Success
NASACT
August 24, 2015
© Grant Thornton LLP. All rights reserved.
Speakers
Cynthia Dannels, Deputy State Treasurer, SC State Treasurers
Office
Tony Hernandez, Principal, Grant Thornton, LLP
© Grant Thornton LLP. All rights reserved.
2
Agenda
• What is Independent Verification and Validation?
• The business case for independent oversight
• Scope of independent oversight
• Elements of effective oversight
• Case Study - South Carolina State Treasurers Office
• Question and answer
© Grant Thornton LLP. All rights reserved.
3
What is Independent Verification & Validation?
Definition
Independent Verification and Validation (IV&V) is a process
of employing rigorous methodologies for evaluating the
correctness and quality of a project throughout the project
lifecycle. Done correctly, IV&V is conducted by a third party
organization not involved in the development of the product.
© Grant Thornton LLP. All rights reserved.
4
What is Independent Verification & Validation?
Key Considerations
• Verification – "Did I build the thing right?"
• Validation – "Did what we build work as expected?
• Accreditation – "Should it be used?"
• Realization – "Are we getting the benefits we expected?"
• There is also an underlying implicit principle, and its key
question: Credibility – Should it be trusted?
© Grant Thornton LLP. All rights reserved.
5
The Business Case for Independent Oversight
Business Drivers
• Public sector technology projects are getting larger and more complex,
project successes and failures are increasingly more public, and overall
projects costs are increasing. This is especially true of large, complex
IT acquisitions.
• While the adoption of formal project management methodologies and
techniques is becoming more common, many public sector technology
projects continue to fail or deliver solutions that are late and over
budget.
© Grant Thornton LLP. All rights reserved.
6
The Business Case for Independent Oversight
Critical Success (and Failure) Factors
The reasons why technology projects fail are many and varied, but some
root causes appear consistently in troubled projects:
– Lack of a well-defined business case;
– Poorly defined or constantly changing requirements;
– A hurried or badly designed procurement that leads to the wrong
vendor being chosen or to a badly written contract;
– The inability to make timely, fact-based decisions; and
– Inadequate skills and experience of government and vendor staff.
© Grant Thornton LLP. All rights reserved.
7
The Business Case for Independent Oversight
Critical Success (and Failure) Factors (cont'd)
It is well accepted in the software
development industry that the cost of
fixing a defect increases exponentially
as a project moves through the
development lifecycle.
However, this principle applies equally well to stages of the project that
occur long before software development starts. In particular:
•
Errors or omissions in the project business case can lead to the wrong solution
being procured, so all the project budget may be wasted;
•
Errors in the procurement can cause a project to be effectively doomed from the
moment the contract is signed;
•
Weaknesses in project governance and decision-making can seal a project's fate
regardless of the quality of project team efforts.
© Grant Thornton LLP. All rights reserved.
8
The Business Case for Independent Oversight
How Independent Oversight Can Help
When the above types of issues impact projects, it is often because there
was a particular element missing from the project governance structure.
This element is a party with:
– The experience and foresight to anticipate the problems that could occur in the future;
– The ability and authority to communicate these problems and their consequences to
executive management;
– The expertise to recommend a practical course of action to avoid the future problems;
and
– The trust of executive management in the objectivity and independence of the
recommendations.
An independent oversight consultant can fulfill this role, and can provide
project leadership with practical, unbiased advice that can help projects
avoid major problems and improve the likelihood of project success.
© Grant Thornton LLP. All rights reserved.
9
The Business Case for Independent Oversight
How Oversight Should Pay for Itself
Independent oversight is analogous to preventative care in the medical
industry – a relatively small investment up front can reduce the likelihood
of significant expenses later on.
The level of investment in oversight should be a function of the size,
complexity and criticality to the mission of the project. An industry
average for oversight/Independent Validation & Verification (IV&V)
services ranges from 5 - 10% of total project development and
implementation cost. Similarly, the cost of IV&V is a function of the size,
risk and complexity of a project.
© Grant Thornton LLP. All rights reserved.
10
The Scope of Independent Oversight
Full Project Lifecycle
Issues and risks that impact project success can occur very early in the
project lifecycle, and go well beyond the traditional software development
activities that are the focus of standards such as the IEEE 1012 standard
for IV&V.
For this reason, the
scope of independent
oversight services
should include the full
project lifecycle, from
initial project concept
through sustainment in
production operations.
© Grant Thornton LLP. All rights reserved.
11
The Scope of Independent Oversight
Oversight Categories
•
Governance – To assess the level of executive stakeholder awareness, buy-in and
support, and the willingness and ability to make timely, fact-based decisions.
•
Project Management – To determine if the project is planned and managed in a fashion
that effectively defines and controls scope, schedule, budget and resources.
•
Risk and Issue Management – To determine whether risks and issues are accurately
and comprehensively identified and managed. Also to determine whether the culture of
the organization supports proactive and transparent communication of risks and issues.
•
Quality Management – To assess the activities involved in quality planning,
implementation of the quality plan, and the quality assurance functions within the project.
© Grant Thornton LLP. All rights reserved.
12
The Scope of Independent Oversight
Oversight Categories (cont'd)
•
Organizational Change and Training – To determine whether internal and external
stakeholders are being prepared for – and will be ready for – the impending changes.
•
Requirements Management – To assess the requirements management plan and
validate that requirements management activities are implemented in accordance with the
requirements management plan.
•
Procurement and Contract Management – To determine whether procurement
activities enable the best possible vendor selection and whether resulting contracts are
managed in a fashion that reinforces quality and accountability.
•
Solution Development and Implementation –To validate that solutions are being
designed, developed and implemented in accordance with approved plans and designs.
•
Maintenance and Operations – To determine whether the project is conducting
maintenance and operations activities according to plan and service agreements.
© Grant Thornton LLP. All rights reserved.
13
Elements of Effective Oversight
Success Factors and Approach
Oversight success factor
The experience and foresight to
anticipate the problems that could
occur in the future.
High value oversight approach
•
•
•
Be predictive, not reactive.
Leverage experience to foresee the future consequences of
current decisions and actions.
Findings must be factual and evidence-based.
The ability and authority to
communicate these problems and
their consequences to executive
management.
•
•
Direct line of communication to executive management.
Translate problems and recommendations into language
that is meaningful and relevant to decision-makers.
The expertise to recommend a
practical course of action to avoid
the future problems.
•
•
Identify pragmatic, actionable solutions to problems.
Be part of the solution, not just an identifier of problems.
The trust of executive management
in the objectivity and independence
of the recommendations.
•
•
Independence in fact and in appearance.
Integrity and courage of conviction to state the facts, even
(especially) if they are unpopular or uncomfortable.
© Grant Thornton LLP. All rights reserved.
14
Elements of Effective Oversight
Communications Planning
• Communication tools
– Initial review reports
– Status reports
– Briefings
– Dashboards
– Assessments
– Final review reports
– Close-out reports
© Grant Thornton LLP. All rights reserved.
15
Elements of Effective Oversight
Risk Identification and Management
• Risk quantification and reporting
– Risk identification
– Risk analysis and assessment
– Risk rating and ranking
– Risk monitoring and control
– Risk mitigation, escalation and resolution
– Risk reporting
… Enables pro-active decision-making!
© Grant Thornton LLP. All rights reserved.
16
Elements of effective oversight
Key considerations
• Treat the root cause, not just the symptoms.
• Identify pragmatic, actionable solutions to problems.
• Be part of the solution, not just an identifier of problems.
© Grant Thornton LLP. All rights reserved.
17
Case Study
South Carolina State Treasurers Office
Communication
tools
Overview of the State
Treasurers Office NVEST and DEBT
Projects
– Retire the STO's legacy Investment Management System and Debt
Management System
– Implement SAP Treasury Management modules for investments
(NVEST) and debt management (DEBT)
– NVEST and DEBT Project Go Lives Scheduled for September 2015
© Grant Thornton LLP. All rights reserved.
18
Case Study
South Carolina State Treasurers Office
Communication
tools to engage a vendor for
Why did you decide
Independent Verification & Validation services
for your project?
© Grant Thornton LLP. All rights reserved.
19
Case Study
South Carolina State Treasurers Office
Communication
What are sometools
of the barriers to bringing in
a vendor for Independent Verification &
Validation services?
© Grant Thornton LLP. All rights reserved.
20
Case Study
South Carolina State Treasurers Office
Communication tools
What are your expectations of the
Independent Verification & Validation vendor?
© Grant Thornton LLP. All rights reserved.
21
Case Study
South Carolina State Treasurers Office
Communication
What are sometools
of the important considerations
for selecting and managing an Independent
Verification & Validation services vendor?
© Grant Thornton LLP. All rights reserved.
22
Case Study
South Carolina State Treasurers Office
Communication
Can you tell ustools
about the benefits you've seen
from the use of Independent Verification &
Validation services? Challenges?
© Grant Thornton LLP. All rights reserved.
23
Independent Verification & Validation
Ensuring Project Success
Communication tools
What recommendations would you have to
share with us?
© Grant Thornton LLP. All rights reserved.
24
Independent Verification & Validation
Ensuring Project Success
Communication tools
What trends have you seen related to
Independent Verification & Validation
services?
© Grant Thornton LLP. All rights reserved.
25
Questions?
© Grant Thornton LLP. All rights reserved.
26
Contact Information
Cynthia Dannels, Deputy State Treasurer
T 803.734.2655
E Cynthia.Dannels@sto.sc.gov
Anthony Hernandez, Principal
T 850.201.7288
E Anthony.Hernandez@us.gt.com
Julie Batchelor, Director
T 919.881.5930
E Julie.Batchelor@us.gt.com
© Grant Thornton LLP. All rights reserved.
27