the 22nd meeting of the INTOSAI Working Group on IT Audit
(WGITA)
KPI Project Final Report
— Key Performance Indicators Methodology
for Auditing IT Programs
Amy Young, IT Center
China National Audit Office
April, 2013
Lithuania
Framework
CNAO
1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
2
1. Background
CNAO
•the 19th of the INTOSAI Working Group on IT
Audit , April 2010
– three proposed projects:
• Development of IT Performance Indicators
• Performance measures of IT Solutions implemented in
government organizations
• Index System about IT Performance Audit
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
3
1. Background
CNAO
• the 19th of the INTOSAI Working Group on IT
Audit , April 2010
– The above three projects were put into a new one.
“Key Performance Indicator Methodology for
Auditing IT Programs”.
– SAI China volunteered to be the team leader.
– Team members : SAI Bhutan, China, Ecuador,
Japan, Kuwait, Malaysia, Pakistan, Poland, Russia
and USA.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
4
Framework
CNAO
1. Background
2. Project objectives
3. Project progress
4. Project products
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
5
2. Project objectives
CNAO
• two principal products
– a set of specific and measurable IT-related indicators
– guideline
•the ideas of KPI application
•feasible KPI evaluation methodology
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
6
2. Project objectives
CNAO
• Analysis, conclusions and decision making
based on results of performance measurement
•exchange views, know-how, and information
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
7
Framework
CNAO
• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
8
3. Project progress
CNAO
• Milestones
– By the end of 2011, the beta database of KPI
indicators.
– By the end of 2012, the draft Guideline.
– In 2013, the final Guideline and KPI Database.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
9
3. Project progress
CNAO
• the first team meeting on Apr 16, 2010 in
Beijing.
• a new name for the project, “Key Performance
Indicators Methodology for Auditing IT
Programs”.
• the minutes of the first team meeting.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
10
3. Project progress
CNAO
• SAI Russia, Japan , USA , Kuwait and Ecuador
explained their own understanding about the
project from April to June, 2010.
• The project initiation document to Chair India
in July 2010.
• SAI Kuwait submitted the beta database of KPI
indicators on Dec 29, 2010.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
11
3. Project progress
CNAO
• The original framework based on SAI Kuwait’s
and China’s PKI indicators in Feb, 2011.
• SAI Kuwait, Bhutan and Japan gave the
comments in Feb and Mar, 2011.
• the Second team meeting on Apr 16, 2011 in
South Africa.
• the minutes of the second team meeting.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
12
3. Project progress
CNAO
• SAI Kuwait submitted the categorized KPI beta
database on August 11, 2011.
• SAI Bhutan gave comments on August 18 ,2011.
• SAI China composed the beta indicators and send it
to the team members for comments on November
30, 2011.
• SAI Japan and Kuwait gave the comments in
December,2011.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
13
3. Project progress
CNAO
• In the 21st meeting in Kuala Lumpur, Malaysia in
January 2012, the team made the report about the
KPI project.
• SAI China sent the content framework of guideline
to the team members for comments on April 9, 2012.
• In April and May, SAI Kuwait and Bhutan gave the
comments and SAI Kuwait applied to compose Part 1
and Part 2.
• SAI China began to compose the draft guideline in
June, 2012.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
14
3. Project progress
CNAO
• SAI China delivered the draft guideline for comments
among the team members on December 30, 2012.
• SAI Japan, Kuwait and USA feedback in Jan and Feb
2013.
• In early March 2013, SAI China send the final
guideline v1 to Chair India for comments among all
the WGITA members.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
15
3. Project progress
CNAO
• SAI Bangladesh, Brazil, Iraq and Russia gave
the comments.
• Based on the comments, SAI China made the
final guideline v1.1.
• Comments from SAI Malaysia, Lithuanian and
Qatar.
• In Apr 2013, the final guideline V1.2 was
delivered for approval in the 22nd meeting of
WGITA .
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
16
3. Project progress
CNAO
• Communication Management Strategy
– mainly based on email through Internet
– email contact list for the team members
– sometimes discuss trough telephone
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
17
Framework
CNAO
• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
18
4. Project products
CNAO
• Criteria for the indicator
– The indicators: be widely accepted and measurable
– the data source of the evaluating indicators : be
accessible and analyzable
– the calculation methods of the evaluating indicators:
be relatively simple
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
19
4.1- KPI database
CNAO
• 11 parts, 367 indicators with three-level.
• 1. Decision (15 indicators)
– Compliance with the laws
– Feasibility study
– Participation in decision
• 2. Requirement analysis (13 indicators)
– Organization target
– Core business coverage
– Response/change
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
20
4.1- KPI database
CNAO
• 3. Design / Planning (45 indicators)
– Requirement coverage
– Time limit
– Capacity Planning and Resource Provisioning
– Cost estimation
– IT risks
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
21
4.1- KPI database
CNAO
• 4. Procurement/Development (48 indicators)
–
–
–
–
–
–
–
–
–
Selection for partner or supplier
Cost control
Process Control
Code control
Outsourcing
Quality Control
Testing
Training
Upgrading
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
22
4.1- KPI database
CNAO
• 5. Product (26 indicators)
– User satisfaction
– Price
– Delivery
– Performance
– Integration
– Technology applicability
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
23
4.1- KPI database
CNAO
• 6. Maintenance (125 indicators)
–
–
–
–
–
–
–
–
–
Follow the management rules
Incident management
System Usability
Availability
Maintenance cost
Website
Monitoring
Change Management
Data Center
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
24
4.1- KPI database
CNAO
• 7. Security (26 indicators)
– Security plan
– Identity management
– User account management
– Security testing monitoring
– Security incident definition
– Malicious software prevention
– Network security
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
25
4.1- KPI database
CNAO
• 8. Backup and Disaster Recovery (14
indicators)
– Backup plan
– Recovery plan
– Backup operation management
– Recovery operation management
• 9. Service (32 indicators)
– Service request
– Service Response
– Service Satisfaction
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
26
4.1- KPI database
CNAO
• 10. Effectiveness (19 indicators)
– Coverage of the core business
– Benefit
– Internal Management optimizing
– Public Service
• 11. Others (4 indicators)
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
27
4.2 - Guideline
•
•
•
•
CNAO
Part 1. Preface
Part 2. Audit Plan
Part 3. Audit Implementation
Part 4. Audit report
• Reference : INTOSAI training materials, ISACA,
COBIT, ITIL and others.
the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA)
28
the 22nd meeting of the INTOSAI Working Group on IT Audit
(WGITA)
Thanks
for your attention!
China National Audit Office
April, 2013
Lithuania