EAA 2011: Rome, Italy
Improving System
Development Project
Success: How Internal
Auditors Add Value Through
Process Involvement &
Measurement
Glen L. Gray, California State University, Northridge, USA
Anna H. Gold, VU University, The Netherlands
Christopher G. Jones, California State University, Northridge, USA
David W. Miller, California State University, Northridge, USA
Overview
• Background
– SDP failures and the dismal
rate of SDP success
– Control issues
• Research objective
– Internal auditor’s role in
SDP success
• Research questions,
methods, and summary of
findings
2
Many SDP failures…
• December 2002: McDonald’s abandons major project after
two years. Cost: US$170 million
• November 2004: Sainsbury (UK supermarket chain) writes off a £260 million
IT investment in its supply chain
• February 2008: Los Angeles Unified School District’s
faulty US$95 million payroll system goes live. For months
afterward, thousands are overpaid, underpaid, or not paid at
all.
• November 2010: FBI spent $405 million of the $451 million
budgeted for new Sentinel case-management system, but,
as of September, it’s two years behind schedule and $100
million over budget
3
Few SDP Successes…
32%
Successful
24%
Failed
44%
Challenged
Standish Group [2009]
4
Costly Conundrum
• How do failing or challenged projects go
undetected?
• Where were the ‘red flags’?
– Missed, dismissed, or ignored all together?
• Who’s responsible for
monitoring the controls and
raising these red flags?
5
Research Objective
• To explore how internal auditors currently do and
potentially can provide value-added support to
proactively help identify and monitor system
development project controls to either:
– Help get these projects
back on track toward success or
– Stop projects when the
investment in the projects
is still relatively low
6
Post-SOX Changes?
• Pre-SOX: internal auditors usually came into a system
development project after the project was completed to
evaluate the internal controls—bayoneting the wounded
• Post SOX: internal auditors are more frequently active
members of major system development projects, but—
– auditor focuses on controls for the specific processes
being automated, not the system development controls
Gray [2004, 2007]
7
Research Questions
RQ1: When and how should internal auditors
become involved in SDPs?
RQ2: For which factors critical to system
success can internal auditors add the most
value?
RQ3: What metrics should be used to monitor
SDPs?
8
Mixed-mode Research Method
1. Review IS and internal auditing literature
•
CSFs and CFFs
2. Conduct internal auditor focus groups exploring
RQ1 – RQ3.
•
Qualitative
3. Develop CSF taxonomy from an internal auditing
perspective
•
Qualitative
4. Survey a sample of The IIA membership
•
Quantitative
9
Critical Success Factors
• Literately, hundreds of success/failure factors
– However, many different ways to say same things
• From both professional and academic literature
• Mostly opinions/observations vs. rigors analysis
• Mostly not stated as measurable factor/metric
(e.g., adequate user involvement)
• Our next task: reduce factors to manageable
set.
10
Critical Success Factor Taxonomy
Organization
Project
People
Project
Management
Externalities
11
Critical Success Factors
Project Management
1. Systems Development
Methodology
2. Quality Assurance
3. Change Management
4. Monitoring SDP Process
5. Financial Management
6. Tools and Infrastructure
7. Agile Optimization
Project
8. System Requirements
9. Systems Interoperability
People
10. Executive Support
11. Project Personnel
12. Project Management
Expertise
13. Conflict Management
Organization
14. User Involvement
15. Business Alignment
Externalities
16. Vendor Relationship
Management
12
Summary of Findings (1)
RQ 1 Internal Auditor’s Role
– Waiting until post-implementation review is too
late.
30%
25%
20%
15%
10%
5%
0%
Project Selection
Project Plan
Analysis & Design
Implementation
Review Phase
Greenberg & Murphy, 1989
13
Summary of Findings (2)
RQ 1 Internal Auditor’s Role
– It’s OK to invite yourself to the party.
How do auditors get involved?
11.3%
10.0%
IA Initiated
Mgt Initiated
Mandated
Other
39.5%
39.2%
14
Summary of Findings (3)
RQ 2 Where Internal Auditors Add Value
– Some CSFs more critical than others.
• Criticality transforms.
Internal
Auditing
Adds Value
Contributes to
Project
Success
Critical Success Factor
Rank
Mean
Rank
Mean
Quality assurance (PM)
1
4.04
5
4.54
Change management (PM)
2
4.01
6
4.54
Monitoring SDP (PM)
3
3.93
10
4.46
System requirements (P)
4
3.85
1
4.72
Systems development
methodology (PM)
5
3.80
3
4.60
15
Summary of Findings (4)
RQ 3 Monitoring SDP Success
– Metrics abound but dashboards uncommon.
– Conventional wisdom evolving.
Old Conventional
Wisdom
New Conventional
Wisdom
Internal auditing should
primarily focus on
application controls
Internal auditing should
also focus on
SDP controls
16
Internal Auditor Involvement
• Three basic approaches to the auditor’s involvement in
SDPs:
– Auditor approach would be the more traditional auditing
function by monitoring the SDP on a milestone basis to
monitor how the project is progressing on behalf of
management and the board.
– Consultant approach where the internal auditors are
advising the SDP team on an as-needed basis regarding
controls.
– Embedded approach where internal auditors are
integrated in the SDP team functioning as the control
experts.
17
Internal Auditor Involvement
[Large]
Embedded
Internal
Audit
Department
Size
Consultant
Auditor
[Small]
[Audit]
IT Skill Portfolio
[IT]
18
The Final Survey Question
Q: What is the one best way for internal auditors
to improve the success rate of SDPs?
A: “Be included, be involved, and participate
regularly in the process from project
inception.”
19
Questions?
Thank You!
Grazie Mille!
Glen L. Gray [glen.gray@csun.edu]
Anna H. Gold [a.h.gold@vu.nl]
Christopher G. Jones [christopher.jones@csun.edu]
David W. Miller [david.w.miller@csun.edu]