Performance Auditing
Alta Fölscher
CABRI/WBI PFM Training Seminar
18-20 June 2007
What is a Performance Audit
• Differs from regularity audit
– Different approach, more ad hoc, less rigid, different skills
– But same insofar an independent, objective assessment
• Audit of 3 Es
– Economy
– Efficiency
– Effectiveness
•
•
•
•
All three or any combination of three
Of wide range of issues
Aim to lead to improvements
Two dominant approaches
– Performance oriented
– Problem oriented
An example of a performance audit
• Does the Drug Abuse Resistance Education (DARE)
Program Work?
– DARE is a drug abuse prevention program that aims to reduce
drug use among school-age children. The performance audit of
the program measured the extent to which it was achieving its
goals. The audit used an experimental design, comparing
juvenile arrest rates for youth in the program (the intervention
group) with those who did not (the control group). The
demographic profiles (ethnicity, income levels, age) were
identical. The audit found that students who participated in the
program were arrested more frequently than the control group,
for both drug-related and non-drug-related offences. The design
of this audit was heavily dependent on the existence of sufficient
reliable data for determining student involvement in the program
and identifying their arrest information in the local juvenile
correctional system. These conditions are often difficult to meet,
unless such comparisons have been planned from the start.
Source: Location fictionalized from the 1994 audit of the Austin DARE Program, Office of the City Auditor, Austin, Texas as reported in
Waring C and Morgan S, 2007. Public Sector Performance Auditing in Shah, A (ed): Performance Accountability and Combating
Corruption, WBI, Washington DC
Why is it important?
• Based on need for public accountability
– Insight into activities of public agency on behalf of
stakeholders
• Reliable independent information on
performance strengthens legitimacy and trust
• Basis for future decisions
• Incentives for change
• Better government spending, better services,
better public accountability and better public
management
• Are things done in the right way; are the right
things being done?
The 3 Es
• For same level of quality
• Economy
– Is about keeping the cost low
– Minimising the cost of resources used or required
• Are means chosen most economical use of resources?
• Have human, financial and material resources been used
economically?
• Is management of programme in line with sound admin and
management?
– Spending less
• Efficiency
– Relationship between outputs and the goods, services and
resources used to produce them
– Can I produce more from given resources, can I use less
resources for a level of outputs?
– In practice often difficult to distinguish from economy
– Spending well
3 Es
• Effectiveness
– Relationship between intended and achieved
results of public spending
• Are aims being met by the means employed,
outputs produced and impacts?
• Are the impacts really result of policy rather than
other circumstances?
– Spending wisely
3Es
•
•
•
•
•
Assessment always relative to audit criteria (standards against which actual
performance measured)
Important choice in audit design
Standards used to determine whether programme meets or exceeds
expectations
Audit scope and design interprets economy, efficiency and effectiveness
concretely
Criteria should be relevant, reasonable, attainable eg
– Historical; Similar programmes or other locations; Legislation; Accepted
standard; Professional opinion
•
Assessment of effectiveness particularly tough
–
–
–
–
Starts with programme design and clear, concrete goals
Ideally measurement before and after + control group
In practice difficult
Less ambitious
• Assess plausibility of assumptions in programme design and achievement of goals as
set down (goal achievement analysis)
• Have objectives been reached, target groups reached, what is level of performance?
Mandate of Performance Auditor
•
•
•
•
Important underpinning for work
Mandate and goals should be defined in legal framework
Specific regulations
Auditor must be free to select audit areas within mandate
and refuse commissions
• Professional quality of work important: sufficient
resources to acquire skills
• Differs across countries
– Coverage of mandate
– Scope of audits
– Approach
Input Output chain
Programme dimension
Commitment
Input
Action/production
Output
Outcome
Purpose
defined
Resources
assigned
Activity undertaken
Goods and
services
provided
Objectives
met
Auditor
should be
careful not to
stray into
political
terrain
But, audit
findings can
include
reflections on
policies
Source: Waring C and
Morgan S, 2007.
Public Sector
Performance Auditing
in Shah, A (ed):
Performance
Accountability and
Combating Corruption,
Performance Auditor’s domain
Performance dimension
Economy
Efficiency
•Physical
and
financial
•Amount
•Timing
•Quantity
•Quality
Productivity
Unit cost
Operating ratios
Output
Outcome
effectiveness effectiveness
Level
Quantity
Timeliness
Quality
Price Cost
Customer
satisfaction
Mission and
goal
achievement
Financial
viability
Cost benefit
Cost
effectiveness
Cross-cutting performance aspects
Compliance with laws and regulations; validity, reliability and availability
of information; maintaining underlying values; continuous improvement
What to audit?
•
•
•
•
Determine selection criteria country by country
Added value important
Important problem / problem area
Risk
–
–
–
–
–
Substantial budgets, sudden increase in budgets
Traditional risk areas
New, urgent activities
Complex management arrangements
Performance information not updated
Standards for Performance Auditing
• Well thought out plan very important
• Pre-study
• Audit proposal
– Identify issue and audit objectives (what will be audited and why)
– Develop scope and design of audit
•
•
•
•
•
What issues, what are hypotheses, what type of study?
Is issue auditable and worth auditing?
Understanding audited programme
Defining audit criteria
Methodological planning
– Quality assurance, timetable, audit team and resources
• Field standards
– Communicative and analytical process
– Sufficiency, validity, reliability, relevance, reasonableness of
evidence
– Scepticism
– Record-keeping and oversight
• Accessible, comprehensive reporting
Performance auditing chain
Audit cause – the programme (intervention)
Audit criteria – what should be
Audit evidence (condition) – what is
Audit finding – what is compared with what should be
Determine the causes and effects of the finding
Develop audit conclusions and recommendations
Estimate likely impact of recommendations
Examples of audit findings
Type
Elements
Example
Descriptive Condition only
Annual cost to incarcerate a prisoner was $67,800
in 2005.
Normative
Annual cost to incarcerate a prisoner was $67,800
in 2005, compared with $52,000 at comparable
prison
Condition and
criteria
Traditional/ Criteria,
causal
condition,
cause, and
effect
Impact
Condition with
cause
compared with
condition
without cause
Annual cost to incarcerate a prisoner was $67,800
in 2005. Budget authorized $58,000, resulting in a
deficit of $17.8 million. The additional costs were
caused primarily by increase in labour and benefit
costs following the May 2005 union contract.
Recidivism rates among alcohol-dependent
inmates who participated in the alcohol treatment
before release were significantly lower than rates
among alcohol-dependent inmates who did not
receive the treatment.
Source: Waring C and Morgan S, 2007. Public Sector Performance Auditing in Shah, A (ed): Performance
Accountability and Combating Corruption, WBI, Washington DC
Performance auditing in SubSaharan African
• Performance auditing can help build foundations for
performance system
• But
– Keep it simple at start
– Basics should be in place (rule of law, clearly defined institutions
with understood roles and responsibilities, budget systems and
accounting systems)
– Financial audit function in place first
– Organisational independence, legal mandate, sufficient funding,
strong leadership, ability to recruit and retain skilled staff,
stakeholder support and professional audit standards
– Credibility
– Support at highest levels of government