Corrections Technology
Association (CTA) 2015
June 1, 2015
Daytona, FL
Launch Transformation
Boosted by a
Strategic Advisor
With You Today
Aaron Garner
Executive Director
Research & Technology
Indiana Department of Correction (IDOC)
Jack Umansky, PMP
Director
KPMG LLP
345 Park Ave New York, NY. 10154
518-951-8199 | JackUmansky@KPMG.com
Keven P. Star
Director
KPMG LLP
500 Capitol Mall, Suite 2100
916-690-6702 | Kstar@kpmg.com
1
Why Projects Need a QA, IV&V or Technical Advisor?
Source: http://onproductmanagement.net/wp-content/uploads/2010/08/treecomicbig.jpg.
2
IV&V – Return on Investment (ROI)
3
The Challenges of IT Projects – Problems Launching
The Project Management Profession Lives in Troubled Times…..
“Only 25 percent of telecom IT projects
succeed, a rate that is worse than IT
projects across all industries, where
28 percent succeed.”
Standish Group
“75% of 1,450 firms surveyed exceeded
their IT deadlines and more than 50% of
them substantially exceeded their budgets.
Key reasons were poor planning,
inadequate studies on how IT projects
relate to the firm’s needs and lack of
management support.”
Why?
“72% of all IT Projects are late, over
budget, lack functionality, or are never
delivered as planned”
Meta Group
KPMG Survey
“…the odds of a FORTUNE 500 software
development project coming in on time
and on budget are unfavorable – only one
in four succeed.”
Standish Group
4
Recent IT Project Failures at State and Local Municipalities

A large US state tried and failed twice to implement a new child support enforcement system at a taxpayer
cost of $1 billion (USD). In addition, the State was sanctioned another $1 billion (USD) in federal penalties
until it successfully implemented a system on its 3rd attempt.

Implementation of a new $55 million payroll system for a large urban school district resulted in $53 million
in overpayments to teachers, months of non-payments, and additional costs of $35 million for repairs and
delays.

A Southern state planned a $68 million construction project of two data centers located on the floodplain of
a dam determined by the Army Corps of Engineers to be at risk of failure.

With the $100 million price tag already spent, a state’s enterprise financial management system project
was indefinitely suspended after an independent assessment determined that the state failed to adhere to
industry best practices for funding, planning, and implementation.

A $171 million five-year project to build and run a state’s Medicaid claims system is behind schedule and
bogged down in litigation, with the state blaming the vendor for unacceptable work product and the vendor
blaming the state for inefficient project management and lack of knowledge about the current system.

Nearly half of eight months’ worth of Medicaid fraud documents were destroyed and scores of prosecutions
were compromised when the outsourcing vendor for one of the largest states failed to follow file backup
procedures.

A large Northeastern city has yet to deploy a new billing system for its half-million water customers after
spending over three years and $18 million, more than twice what it expected to pay.

State auditor (CA) recognizes need for independent assessments as $1Billion spent on failed projects.
5
Strategic, Technical, and Business Transformation Advisory
Phase 1
 Strategic Planning
Phase 2
 Business Process
Analysis and Reengineering
 Requirements
Definition
 Vendor and
Software Selection
Assistance
Phase 3
 Project
Management,
Monitoring, and
Quality Assurance
 Risk Management
 Independent
Verification and
Validation (IV&V)
6
Phase 1
 Business Process Analysis
and Re-engineering
 Why Define Requirements?
7
Requirements Definition
Requirements describe what the system should do or what the system should
allow a user to do, not how it will be done.
What are Requirements?
• A requirement is a “condition or capability needed by a user to solve a
problem or achieve an objective.”
• Requirements are the functions we expect a system to be able to perform
and at what level of performance the functions are desired.
• Requirements are specific, measurable, testable and attainable.
What Requirements are NOT.
• Requirements are not specifications regarding how a function will be
implemented.
• Requirements do not describe how the system or user will do it.
8
Requirements Definition – Examples
The following are sample requirements:
•
The system will allow users to search by Inmate/Offender ID
•
The system must support a minimum of 5,000 concurrent users
The following are NOT requirements:
•
The user will select search criteria using radio buttons and a drop down
menu
•
The system will display the alert message in red
9
Phase 2
 Vendor and Software Selection
Assistance
10
Vendor and Software Selection Assistance Objective
Objective: Assist government clients to prepare a solicitation document for a
Systems Integrator (SI), guide , advise, and assist in the evaluation process, and
support the negotiation and contracting of the SI for the implementation of a new
system.
Item 1
Item 2
Item 3
Pre-Selection Assistance
Develop a “long-list” of potential vendors
Approach:

Pre-Selection Assistance

Initial Analysis

Risk Assessment

Oral Presentations

Final Analysis

Post Selection Assistance
and Contract Negotiations assistance
Initial Analysis
Utilize Proposal Assessment Matrix (PAM)
Risk Assessment
Identify potential risks & questions for the vendor
Oral Presentations
Facilitate planning and develop questions for orals
Final Analysis
Update Proposal Assessment Matrix
Post Selection Assistance
Prepare Final Evaluation Report
11
Selection Assistance Services Background
Selection Assistance Services can assist clients with the evaluation and selection of
systems integrators, solutions, products, or technologies as part of the projects that
may result. Our SAS approach supports:

Design of the selection process

Determination of scoring criteria

Research relevant vendors

Drafting of requests for information/request for proposals based on specific selection
criteria

Performing objective risk assessments of submitted proposals

Scoring vendor responses based on selection criteria

Support with vendor inquiries

Assist agency with oral presentations and software demonstrations

Assistance with the selection including collating vendor responses and facilitating scoring
workshops

Assist agency with contract negotiations, drafting, review including risk assessments and
recommendations
12
Phase 3
 Project Management, Monitoring,
and Quality Assurance
 Risk Management
 Independent Verification and
Validation (IV&V)
13
Representative Tasks, Roles and Responsibilities
Deliverables
Management
Ongoing
Schedule
management,
monitoring
and reporting
Risk, Issue and
Action Item
Management
Your
Trusted
Advisor
Independent
Project
Monitoring /
Assessment
Requirements
Traceability
and Scope
Management

Approach that is tested and
tailored specifically for the
Corrections environment

Approach that includes proven
templates and processes
developed and successfully
implemented for Corrections
initiatives

Based on well-known project
management and quality
disciplines (e.g., IEEE, PMI’s
PMBOK and CMMI)

Risk-based and forward-looking

Drives quality in the project
processes and project
deliverables
14
Typical Tasks
Deliverable Review Management:
•
•
•
•
Review SI Workplan w/ Status Reports and
Help facilitate SI deliverable preparation, report on/provide the following:
• Tasks late and overdue
submission and DOC review process
• Tasks slipping or at risk for missing due dates
Develop and maintain deliverable tracker • Critical Path analysis
Independently verify that deliverables and • Resource challenges
work products meet contractual and DOC • Identify DOC dependencies/responsibilities
expectations and professional standard
• Generate reports to help track project
requirements
schedule, estimates to complete, etc.
Review SI deliverables, provide actionable • Raise concerns and recommendations as
appropriate
comments, and recommendations
Action Item Management
Risk/Issue Management
•
•
Identify, document, present risks and
issues to DOC Management, provide
actionable recommendations and assist
with risk mitigation planning
•
Maintain Action Item List and assist DOC in
resolving action items related to the systems
implementation,
•
Establish regular tracking mechanism and
document actions and decisions made.
Assist with proactive risk mitigation
15
Typical Tasks
Assist with Requirements traceability
• Participate in meetings (JAD, conversion,
design sessions, and others) and review
deliverables for adherence to requirements
• Review and assess requirements
coverage, linkages to artifacts (functional
designs, code, test cases, etc.)
• Perform periodic audits of requirements
coverage at project milestones
• Provide ongoing advice and
recommendations
Attend and participate in project meetings
and assist with the following:
• Provide deep subject matter knowledge and
advice to assist DOC
• Provide integration across work streams
and DOC business process owners and
SMEs
• Review meeting decisions and documented
meeting minutes
• Provide recommendations as appropriate
Conduct Change Request (CR) analysis
• Assist DOC with evaluation of SI CR
Impact Assessments
• Review SI Impact Assessments for
appropriateness
• Review Level of Effort estimation
• Assist DOC with tracking of CR LOE
“debits” and “credits” as needed
16
Why Invest in Project Quality Assurance and/or IV&V?
 Independent assessment of deliverables, activities, and processes
 Subject matter expertise in individual disciplines
 Unbiased technology and project recommendations
 Insight into leading practices from other jurisdictions
 Partnership to provide assessments and recommendations
 The federal government supports and funds third-party QA/IV&V for federally funded
projects (45 CFR Part 95) ((e.g., Federal OCSE for Child Support projects)
 Jurisdictions require QA/IV&V for significant projects (e.g., Indiana, NYC, NASA,
California)
Costs of Project Assurance (IV&V, QA, PMO, others) can include:

Fiscal Costs

Time and Resource Costs to coordinate with Assurance teams

Cultural Impacts

Sponsorship Costs
17
The Quality Assurance Value Proposition

A Titan study conducted at NASA determined:
— The use of an Independent Project Oversight can reduce the total ownership costs of the system
— Nearly two-thirds reduction in Defect Density with full lifecycle Independent Project Oversight

A case study conducted by Digital, Finding Defects Earlier Yields Enormous Savings, indicated that each phase a
defect survives it costs approximately three times as much to fix that defect (http://www.digital.com/solutions/roi-cs2.php)

Another case study, conducted by NASA, A Case Study of IV&V Return on Investment (ROI), indicated that the
ROI for IV&V on a software development project was between 1.25 and 1.82

For another project, the IV&V ROI was 11.8 (Estimating Direct Return on Investment of Independent Verification
and Validation using COCOMO-II, J.B. Dabney, G. Barber, and D. Ohi)

State auditor recognizes need for independent assessments as $1Billion spent on failed projects.
California Bureau State Audit Report - https://www.auditor.ca.gov/pdfs/reports/2014-602.pdf
The Titan Study argues the ROI figures understate the full value of an Independent Project Oversight as a result
of the following factors:
• Watchdog effect – The presence of a QA vendor makes the developer more conscientious and less
likely to cut corners
• Improved maintainability – QA reviews improve the accuracy, readability, and general usability of
system documentation
• Better understanding and response to risks – QA offers impartial evaluations and
recommendations as to how to proceed when there are difficult alternatives
18
What Quality Assurance is NOT!



QA is not a guaranty of project success – issues in or out of a project’s control may
ultimately overtake other best efforts:
•
Incorrect or insufficient staffing
•
Funding issues
•
Lack of timely or appropriate decision making
•
Absence of appropriate tools
•
Absence of senior management or key stakeholder support
QA is not “staff augmentation” or an open checkbook:
•
QA does not perform a client’s job if the client happens to have insufficient staffing
•
QA must perform contracted functions and not “stop-gap” or out-of-scope activities
QA assesses information and provides recommendations based on those
assessments:
•

The objective is not to find fault with staff; it is to help provide potential improvements
in existing processes, alternatives that may not have been recognized or considered,
and independent, unbiased views and suggestions
QA helps “ensure” quality – but does not provide absolute assurance
19
What is Independent Verification & Validation (IV&V)?
Commonly accepted industry definition – IEEE 1012-2008
Verification – focused on whether the
software and its associated products and
processes:
- Conform to requirements
- Satisfy standards, practices, and
conventions during life cycle
processes
- Successfully complete each life cycle
activity and satisfy all the criteria for
initiating succeeding life cycle
activities
Validation – focused on whether the
software and its associated products and
processes:
- Satisfy system requirements at the end
of each life cycle activity
- Solve the right problem
- Satisfy intended use and user needs
Simply put…
Verification
Are you building the system in the
right way?
Validation
Are you building the right system?
20
IV&V versus QA
Key differences can be summarized in three basic areas:
Independence
- IV&V provider must be independent of the project team
- QA provider works at the direction of the project team
Risk Remediation
- IV&V cannot participate in the remediation of identified risks
- QA can advise and assist project management in risk remediation
Participation
- IV&V can be conducted as a periodic assessment or ongoing
- QA is generally involved ongoing throughout the project lifecycle
21
Similarities between IV&V and QA
Typical IV&V Scope that also applies to QA
• Inspect and provide feedback on project management and system
development life cycle (SDLC) plans and practices against key quality
attributes.
• Verify whether project practices conform to industry leading standards and
practices
• Inspect software and system engineering artifacts (e.g., requirements documentation,
system design, technical architecture, system development and configuration, testing, training,
implementation, deployment and system maintenance).
• Provide artifact feedback regarding deficiencies (e.g., when project documentation
does not comply with industry standards or other project-specific references.)
• Validate traceability of project requirements throughout the SDLC
• Provide verification of test results reports via an appropriate method
• Provide risk assessments and mitigation recommendations
22
Qualities of a Good IV&V and QA Function
A Qualified QA or IV&V Function Should:

Be Independent and Objective

Bring an Already Established Approach that is Focused on Risks and Controls

Provide Reasonable and Actionable Recommendations for Identified
Deficiencies

Have Tool Sets Built Around a Structured, Repeatable Process Based on IEEE
Standard 1012-2004/2012 and the Project Management Body of Knowledge
(PMBoK)

Have the Bench Strength to Deploy and Maintain Continuity for a Team
Experienced with the Program/Agency or Domain, IV&V, Project Management,
IEEE, and Relevant Technology Skills that Can Serve as a Trusted Advisor

Articulate an Approach that Builds in Opportunities for Collaboration Without
Compromising Independence and Makes Efficient Use of State DOC and System
Integrator’s Time

Focus on successful implementation of your system
•
IV&V/QA does not get rewarded for the volume of findings they make
23
Recurring Themes
 State
Staff Resources are Insufficient or Not Appropriately Qualified To Manage
Project
 Project
Organization and Lines of Authority are Blurred or dysfunctional
 Insufficient
Quality Assurance and Quality Control
 Lack
of Appropriate, Usable Project Plans
 Lack
of Resource-Loaded Schedules and Use of Critical Path Analysis
 Lack
of Project/QA Metrics or Use Thereof
 Incomplete
 Lack
or Poorly Maintained Requirements Management And Traceability
of Communications Between Project Teams and Stakeholders
 Inadequate
Risk Management, Including Insufficient Risk Definition, And Lack of
Mitigation Strategies and Prioritization
 Poor
Adherence to Project Management Plans, and Contract And Industry
Standards
24
Questions?
25