Get Results From Your
Performance Audits
Session Objectives
This session is intended to help you:
1.
2.
Identify and select high impact
performance audits.
Perform preliminary reviews that
enhance audit effectiveness and efficiency
with:
◦ The audit team having an understanding of the
program or entity being audited.
◦ A prioritized assessment of audit risk and audit
opportunity relative to the program or entity.
◦ A basis for developing audit objectives or
terminating the audit.
Session Objectives
This session is intended to help you:
Prepare preliminary review documents
that present a persuasive rationale for
proceeding with or terminating a
performance audit.
4. Develop well-defined audit objectives to
achieve a focused audit effort for
maximum audit efficiency.
5. Understand how group activity techniques
add value to performance auditing.
3.
What Is a Performance Audit?
A Performance Audit is an analysis that
answers a question.
The question is:
“What Is?”
Which is then compared to:
“What Should Be?”
Yellow Book Description
Performance audits are defined as audits that provide
findings or conclusions based on an evaluation of
sufficient, appropriate evidence against criteria.
Performance audits provide objective analysis to assist
management and those charged with governance and
oversight in using the information to improve program
performance and operations, reduce costs, facilitate
decision making by parties with responsibility to oversee
or initiate corrective action, and contribute to public
accountability. (2.10)
Examples of High Impact
Performance Audit Findings
Unemployment Insurance Agency
 Business Application Modernization
(BAM) Project
 Pharmaceutical Costs

Unemployment Insurance Benefit
Overpayments and Nonmonetary
Eligibility Determinations
We estimated that UIA failed to identify and pursue recovery of
overpayments of up to $72.5 million:
Overpayments  Claimant wage and benefit cross match process = $17.9 million
 Cross-program benefit offset process = $12.1 million
 Verification of claimant identity = $8.2 million
 Deceased claimants = $0.4 million
 Untimely nonmonetary eligibility determinations = $26.0 million
 Ineligible alien claimants = $7.9 million
 Total = $72.5 million
Unemployment Insurance Benefit
Overpayments and Nonmonetary
Eligibility Determinations
We estimated that UIA did not assess fraud-related penalties of
between $120.0 million and $236.6 million:
Unassessed Penalties  Classification of claimants’ misrepresentations = $81.5 - $191.8
million
 Claimant wage and benefit cross match process = $37.8 – $43.4
million
 Deceased claimants = $0.7 – $1.4 million
 Total = $120.0 - $236.6 million
Business Application Modernization
(BAM) Project
Enforcement of Contract Provisions The Department should have collected liquidated damages resulting
from the development contractor failing to meet the agreed-upon
implementation dates for BAM. The development contract specified that
liquidated damages would be assessed at $200,000 per month if the
contractor failed to meet the agreed-upon implementation dates for
Phases 3A, 3B, and 3C.
The development contractor missed the initial implementation date of
August 2007 for Phase 3A (renamed as Release 1).
As a result, we calculated that the development contractor owed the
State liquidated damages of $12.5 million.
Prisoner Pharmaceutical Costs
Potential Savings Based on Seroquel and Risperdal Prescription Data
January through July 2010
Percentage reduction in Seroquel
Average monthly number of Seroquel
scripts
Potential reduction in number of Seroquel
scripts
10%
25%
33%
50%
75%
100%
2,386
2,386
2,386
2,386
2,386
2,386
239
597
787
1,193
1,790
2,386
Average monthly cost of Seroquel script
$
335
$
335
$
335
$
335
$
335
$
335
Projected reduction in Seroquel cost
$ 80,065
$ 199,995
$ 263,645
$ 399,655
$ 599,650
$ 799,310
Average monthly cost of Risperdal script
$
38
$
38
$
38
$
38
$
38
$
38
Projected increase in Risperdal cost
$
9,082
$
22,686
$
29,906
$
45,334
$
68,020
$
90,668
Average monthly reduction in cost
$ 70,983
$ 177,309
$ 233,739
$ 354,321
$ 531,630
$ 708,642
Annualized potential savings
$851,796
$2,127,708
$2,804,868
$4,251,852
$6,379,560
$8,503,704
Prisoner Pharmaceutical Costs
PHARMACEUTICAL COSTS
Department of Corrections (DOC)
Comparison of Michigan's and Other States' Average Atypical Antipsychotic Pharmaceutical
Costs
Per Prisoner Per Month
For the Period February 2010 through July 2010
$25
$22.34
$20
$15
$10
$4.16
$5
$1.19
$1.33
$1.47
$4.54
$4.67
$6.36
$7.11
$2.39
$0
Average Cost Per Prisoner Per Month
Note: The chart includes 10 states whose departments of corrections contract with PharmaCorr, LLC, for statewide
pharmacy services.
As noted in Finding 1, about 80% of Michgan's atypical antipsychotic medications prescribed for DOC prisoners are
written by the Department of Community Health and contracted psychiatrists.
How to Identify and Select High Impact
Performance Audits?
Start with a Risk Ranking
Qualitative
Quantitative
Program/System
Qualitative Risk Ranking
Is there a risk to:
 Public
Health, Safety,
Protection
 Service Delivery
 State Security
 Information Security
& Privacy
 Economic
Growth
 Education
 Conservation
&
Protection of
Natural Resources
 Commercial or
Public
Transportation
Qualitative Risk Ranking
Things to consider:
Public Health, Safety,
&Protection
Injury or loss to life, care & protection
of vulnerable citizens, privacy or
citizen rights, food safety, agriculture
State Security
Homeland security, emergency
preparedness (state & citizen),
customs & borders, national
guard, escape, recurring
offender, parole violations, and
MSP.
Service Delivery
Services not factored into other
categories (examples, state park
reservation system, hunting
permits, licensing), Medicaid
payments provided to doctors,
eligibility determination, family
reunification services, mental
health services, etc.
Information Security &
Privacy
Protection of IT resources,
data security
Qualitative Risk Ranking
Things to consider:
Economic Growth
Michigan economy growth, job
increase, business increase
Education
K - 12, higher-ed, adult ed, early
childhood, special ed, laws &
regulations, school aid, school
improvement grants
Conservation and Protection
of Natural Resources
Air, land, waste, water, forests,
wildlife
Commercial or Public
Transportation
Roads, rail, public
transportation, bridges, ferries,
aeronautics
Qualitative Risk Ranking
0 = Not Relevant
1 = Low Relevance
2 = Moderate Relevance
3 = High Relevance
Quantitative Risk Ranking
Is there a risk of:
State Revenue Sources Not Being Realized
or Collected
 Ineffective and/or Inefficient Use of State
Resources
 Improper Payments (State or Federal)
 Contingencies of Potential Liabilities
 Value/Condition of Major State Assets

Quantitative Risk Ranking
Things to consider:
Value of Major Assets
Need for replacement, need
for repairs, and need for
maintenance using State
resources
Examples of Assets:
infrastructure, bridges, roads,
property, IT equipment
State Revenue
Sources Not Being
Realized or
Collected
Failure to collect tax
revenues or fees,
incorrect federal match,
underreporting of
accounts receivable
Contingencies or Potential Liabilities
Financial obligation, debt, claim
Quantitative Risk Ranking
Things to consider:
Ineffective and/or
Inefficient Use of State
Resources
Not seeking federal grants,
higher priced contracts, use
of overtime, staffing
allocations, potential
privatization, combined
departmental operations,
program does not function
as intended, program
provides no value to State
or stakeholders, ineffective
program
Improper Payments
(State and Federal)
Payments to ineligible
recipient, payments for
ineligible service, duplicate
payments, payments for
services not received,
wasted payments that
should not have been made,
payments of incorrect
amount (overpayment &
underpayment)
Quantitative Risk Ranking
0 = Not Relevant
1 = Less than $5 million
2 = $5 million - $10 million
3 = $10 million - $50 million
4 = Over $50 million
Program/System Level Risk Ranking
Is there a risk of:
Unauthorized Access of
Loss of Sensitive
Information
 Persons/Clients at Risk
if Program Fails to
Provide Services
 Internal
Monitoring/Oversight

Reduced Confidence in
Government
 Legislative/Media
Interest
 Material Control
Weaknesses
 Reliability of External
Data

Program/System Level Risk Ranking
Things to consider:
Reliability of
External Data
For example - daycare
billings from providers,
data from school districts,
Medicaid provider billings.
Lack of oversight of third
party administrators.
Program Environment
Unreliable decision-making
data, competent staff,
management oversight,
tone at the top, high staff
turnover, limited resources,
segregation of duties, limited
evaluation of program data
Internal Monitoring/Oversight
Functioning internal audit
Program/System Level Risk Ranking
Things to consider:
Persons/Clients at risk of
program fails to provide
services
Clients not receiving
assistance, medical treatment,
or needed services; persons
harmed, neglected, or killed
because of failure to keep
safe or protect
Unauthorized access
or loss of sensitive
information
Confidentiality of
information, potential
impact to organizational
operations, assets, or
individuals
Reduced confidence in government
Past program failure
Program/System Level Risk Ranking
Things to consider:
Legislative/Media
Interest
Recent or current legislative
interest
Other
Unusual risks such as
fraud, significant safety
risks, other extreme high
risks not fully measured
elsewhere in model
Material Control
Weaknesses
Lack of prior audit finding
correction, demonstration
to resolve material
control weaknesses,
progress to strengthen
controls to address
material control
weaknesses
Program/System Level Risk Ranking
0 = Not Applicable
1 = Low Risk
2 = Medium Risk
3 = High Risk
Audit Risk Ranges
High Risk Audit = 77 and Up
Moderate Risk Audit = 39-76
Low Risk Audit = 0-38
How do I begin a Performance Audit?
To determine the SCOPE of your performance audit,
start by gathering INFORMATION and obtaining an
understanding of the program or entity.
INFORMATION
How Do You Get It?
NLPES
NASACT
SURVEY
Exercise 1
As auditors, we are used to being observant. But just
How Observant Are You?
On a standard traffic light, is green on the top or bottom?
2. When you walk, does your left arm swing with your right or
left leg?
3. On the U.S. flag, is the top stripe red or white?
4. Which way does water go down the drain: clockwise or
counter-clockwise?
5. Which way do fans rotate?
6. Whose face is on a dime?
7. Do books have even numbered pages on the right or left
side?
8. How many curves are there in a standard paper clip?
9. How many lug nuts are on a standard car wheel?
1.
Great Job!!
Determine the Scope and Gather
Information During the Preliminary Review

The primary goal during the preliminary
review is to gain the best understanding of the
agency/program and its processes. This will
allow the audit team to identify deficiencies
and risks, and plan the audit scope and
methodology.
Preliminary Review Process
Obtain an overall understanding of the
audited activity.
 Quickly gather information without
detailed verification.
 Gain an understanding of internal control.
 Identify audit risk and audit opportunity.

Preliminary Review Process
Audit

Risk
The degree to which activities are
exposed to the potential for ineffective or
inefficient use of resources; the
inappropriate disclosure of data; or to
forms of fraud, waste, and abuse which
could result from inadequate controls.
Preliminary Review Process
Audit

Opportunity
The potential for developing audit findings
related to effectiveness and efficiency of a
program.
Preliminary Review Process
 Preliminary








Review Considerations
What are the program’s mandates, responsibilities, functions and
activities?
Which functions are most critical to the overall operation of the
program?
What are the risks related to the program?
What responsibilities involve/create the most problems?
What changes would make your program more effective in
accomplishing its designated responsibilities?
Who are the program’s clients/customers?
Does program management have any concerns that should be
considered?
Have recent reorganizations occurred (downsizing)?
Preliminary Review Process

Preliminary Review Procedures/Methodologies

Interview program personnel.
Perform analytical reviews of expenditures, revenue, and data pertinent
to the program’s mission.
Analyze prior audit results of audits of similar programs in other
states.
Review other external or internal audits.
Review program mission, goals, objectives, policies, & procedures.
Obtain organization charts and review staffing levels, including the
number and type of employees.
Research professional literature.
Tour facilities, including off-site locations.
Review other states’ similar programs and their mission, goals,
objectives, and requirements.
Obtain norms and standards pertinent to the program’s operations.









Preliminary Review Summary
Preliminary Reviews enhance audit
effectiveness and efficiency with:
The audit team having an understanding of the
program or entity being audited.
A prioritized assessment of audit risk and audit
opportunity relative to the program being audited.
A basis for developing audit objectives.
Achievable audit budgets.
Developing Audit Objectives
What is the audit going to
accomplish?
Specific audit objectives are critical to the process of
auditing smarter and focusing our audit efforts.
Without precisely stated objectives, the risk is that the
audit work will not produce the desired results.
Therefore, it is important to define the issue, problem, or
concern that the audit is to examine.
Types of Objectives
Effectiveness
◦ Success in achieving mission and goals
Efficiency
◦ Achieving the most outputs and outcomes
practical with the minimum amount of
resources.
Audit Objective Characteristics
Clarity
◦ Objectives should be stated clearly to inform
the reader what aspects of the program,
activity, or function are being assessed.
Neutrality
◦ Objectives should be stated in neutral terms
so that the reader understands that we
gathered and analyzed data without bias.
Writing Solid Audit Objectives

Objectives should be:
◦ Specific
◦ Focused on one topic

Objectives should not be:
◦ Broad
◦ Compound
Writing Solid Audit Objectives
Ask Yourself:
◦ Will I be able to conclude on this objective?
◦ Will I have multiple conclusions ?
 Effective at one thing; not effective at another thing.
◦ Will my conclusion make sense?
◦ Is this something we even want to conclude
on?
 We may not want to conclude that tax collections
were accurate, but we would conclude that their
efforts to collect taxes were effective.
Audit Objectives - Example 1
To assess the effectiveness of the
Department's efforts to establish
performance measures, identify best
practices, and evaluate services provided to
adults transitioning from foster care to selfsufficiency.
Audit Objectives - Example 1
Final Report Version

To assess the effectiveness of DHS's efforts to
evaluate services provided to youth transitioning
from foster care to self-sufficiency.

To assess the effectiveness of DHS's efforts to ensure
the propriety and reasonableness of discretionary
payments and post-secondary educational assistance
payments made on behalf of youth transitioning from
foster care to self-sufficiency.

To assess the effectiveness of DHS's efforts to ensure
that training for caseworkers is tailored to address
youth transitioning from foster care to self-sufficiency.
Audit Objectives - Example 2
To assess the effectiveness of the
Correctional Facility's efforts to comply
with policies and procedures.
Audit Objectives - Example 2
Final Report Version
To assess the effectiveness of DOC's efforts
to comply with selected policies and
procedures related to safety and security at
the St. Louis Correctional Facility.
Audit Objectives – Example 3
To assess the effectiveness of the Bridges
information system.
Audit Objectives – Example 3
Final Report Version
• To assess the effectiveness of DHS and DTMB's
efforts to implement change controls over the
Bridges application and data.
• To assess the effectiveness of DHS and DTMB's
efforts to implement controls to ensure the
accuracy, completeness, and timeliness of Bridges
interfaces.
Audit Objective Summary

Well-defined audit objectives achieve a
focused audit effort and maximum audit
efficiency.
Referent Group Meetings
•
•
•
Objectives
Documents to Bring
Topics Discussed
Referent Group Meetings
Help decide the scope of the audit.
Objectives of a referent group meeting:





Use a risk-based approach to select activities &
programs for audit
Agree upon audit focus, scope, and methodology
Bring people together
Maximize the use of limited resources
Produce significant savings in audit hours
Referent Group Meeting
Who Attends:
•
•
•
•
•
•
•
Audit Supervisor
Audit Team
Primary Audit Division Administrator and/or Audit Manager
Concurring Audit Division Administrator and/or Audit Manager
Performance Audit Coordinator
Deputy Auditor General
State Relations Officer
Referent Group Meeting
What happens during the meeting?
• Bring several people together (brainstorming).
• Decide on focus and scope of audit.





Discuss audit methodology.
Discuss identified risks and probable audit findings.
Review the wording of the audit objectives.
Strive to include only high impact areas.
Review budget proposals.
• Alleviate fears of missing an issue.
• Discuss fraud risk factors.
• Produce significant savings in terms of audit hours.
Documents Used at the Referent Group Meeting
Narrative document that includes
 Description of Agency/Program
 Preliminary Objectives
 Primary Questions that Objectives Will
Answer
Audit Budget
• Proposed Budget Hours
• Estimated Fieldwork Completion
Date
• Estimated Staff Release Dates
Documents Used at the Referent Group Meeting
Audit Approach Matrix
 Preliminary Audit Objectives
 Known Findings
 Potential Findings
 Preliminary Testing Performed
 Proposed Audit Methodology
Audit Objectives Memo
Other Topics Discussed at the Referent Meeting
Efficiency Aspects of the Audit
 Identify “efficiency” related issues (dollar
savings)
 Develop “efficiency” objective (if warranted)
Legislative Interests/Concerns
 Audit-related interests/concerns
communicated by legislators and/or their staff
Other Topics Discussed at the Referent Meeting
IT Systems
Are there any significant information systems related
to the audit objectives?
 Do we need an objective regarding the reliability and
integrity of data within significant systems?
 Need for IT auditor assistance?
 Potential for “Data Mining” (downloads, extractions,
and analyses)?

Fraud Risk Factors (SAS 99 Compliance)
 Discuss significant results from “Fraud
Brainstorming”
Yellow Book Changes Related to Performance Audits
(from 2011 Government Auditing Standards)

The discussion of validity as an aspect of the quality of evidence has
been revised to indicate that it is the extent to which evidence is a
meaningful or reasonable basis for measuring what is being evaluated. In
other words, validity refers to the extent to which evidence represents
what it is purported to represent. (6.60b)

The discussion of the sufficiency and appropriateness of computerprocessed information now indicates that the assessment of the
sufficiency and appropriateness of computer-processed information
includes considerations regarding the completeness and accuracy
of the data for the intended purposes. (6.66)
Yellow Book Changes Related to Performance Audits
(from 2011 Government Auditing Standards)

The auditor’s responsibilities for communicating identified internal
control deficiencies that are not significant to the audit have been
clarified. Related documentation requirements and those related to
noncompliance with provisions of contracts or grant agreements
or abuse that are not significant to the audit have been removed.
(7.19, 7.22)

The fraud reporting requirement is now limited to occurrences
that are significant within the context of the audit objectives (7.21),
with a requirement to communicate in writing other instances of
fraud that warrant the attention of those charged with governance.
(7.22)
Yellow Book Changes Related to Performance Audits
(from 2011 Government Auditing Standards)

The requirement that audit organizations develop policies to
address requests by outside parties to obtain access to audit
documentation has been removed. (2007 GAGAS, 7.84)

Early communication of deficiencies has been added as a
consideration auditors may follow in the course of the
performance audit. (6.78)
You Too Can Get Results From Your
Performance Audits by –

Conducting a risk assessment to select
high impact audits

Conducting a preliminary review to
establish scope

Developing clear audit objectives

Brainstorming with a team