Inspection, Evaluation, and
Performance Auditing
Presented By
George F. Grob
Center for Public Program Evaluation
To the Association of Inspectors General
New Orleans
November 2013
2
Essentials of Evaluations and
Inspections
•
•
•
•
•
•
•
•
What are they?
Comparison to Audits
Other Related Professional Disciplines
Evaluation Types and Approaches
Analytic Frameworks
Possible Organizational Locations
Managing an Evaluation Office
Quality Standards
Center for Public Program
Evaluation
3
Center for Public Program
Evaluation
What Is an Evaluation
Center for Public Program
Evaluation
4
Evaluation—What Is That?
Evaluation involves assessing the strengths and
weaknesses of programs, policies, personnel,
products, and organizations to improve their
effectiveness.
American Evaluation Association
Center for Public Program
Evaluation
5
Examples of Government Sponsored
Evaluations
• Early implementation of emergency response
systems after 9-11
• Effects of programmatic earmarks for international
HIV/AIDS relief programs
• Case study of Freddie Mac’s loss of $1.2 billion from
Lehman Brothers bankruptcy
• Frequency of and criteria for home visits to foster
care families
• Performance standards for homeless youth centers
Center for Public Program
Evaluation
6
More Examples
• Unlicensed fire sprinkler contractors performing
inspections in Chicago
• Potential use of a fiscal intermediary by the Chicago
department of public health
• Assessment of New Orleans' System of City Courts and
Performance Review of New Orleans Traffic Court
• Evaluation of the City of New Orleans Delinquent
Property Tax Collection Program
• Promising approaches for State and local government
enforcement of prohibition on sale of cigarettes to youth
• Pharmacies with extreme outlier filling of prescriptions for
dangerous, addictive drugs
Center for Public Program
Evaluation
7
Websites of OIG Evaluation Reports
Federal Agencies
•
•
•
•
•
•
U.S. Congress
Department of Health and Human Services
OIG Office of Evaluation and Inspections
Department of Commerce
OIG Audits and Evaluations
Department of State
OIG Audits and Program Evaluations
OIG Inspections
Department of Veterans Affairs
OIG Office of Audits and Evaluations Oversight
Reports
Department of Housing and Urban Development
OIG Evaluations and Inspections
Federal Housing Finance Agency
OIG Audits and Evaluations
•
Government Accountability Office (GAO)
Reports and Testimonies
Audits, Inspections, and Evaluations
City Government
•
•
City of Chicago
Audits, Reviews, Advisories
City of New Orleans
Audits, Inspections, and Evaluations
Center for Public Program
Evaluation
8
Center for Public Program
Evaluation
Evaluation Standards
Center for Public Program
Evaluation
9
Government and Professional Standards for
Evaluations and Inspections
• Association of Inspectors General
• Council of the Inspectors General on Integrity
and Efficiency
• Government Accountability Office
• Institute for Internal Auditors
• American Evaluation Association
Center for Public Program
Evaluation
10
Association of Inspectors General
QUALITY STANDARDS FOR INSPECTIONS, EVALUATIONS, AND REVIEWS
• General Standards
 Staff Qualifications
 Independence
 Due Professional Care

Qualitative Standards

Quality Control
Planning
Data Collection and Analysis
Evidence
Timeliness
Fraud and Other Illegal Acts
Reporting
Confidentiality
Follow-Up








http://inspectorsgeneral.org/files/2012/06/IGStandards_revised_july2012.pdf
Center for Public Program
Evaluation
11
CIGIE Standards for Inspection and
Evaluation
•
•
•
•
•
•
•
Competency
Independence
Quality control
Planning
Data Collection and Analysis
Evidence
Records maintenance
•
•
•
•
•
•
Timeliness
Fraud, Illegal Acts, Abuse
Reporting
Follow-Up
Performance Measurement
Working Relationships and
Communications
http://www.ignet.gov/pande/standards1.html
Center for Public Program
Evaluation
12
GAO Standards for Performance Audits
• Chapter 3: General Standards
▫
▫
▫
▫
Independence
Professional Judgment
Competence
Quality Control
• Chapter 7: Field Work
▫
▫
▫
▫
Planning
Supervision
Evidence
Documentation
• Chapter 8: Reporting
▫ Form
▫ Contents
▫ Quality Elements
▫ Report Issuance and
Distribution
http://www.gao.gov/govaud/yb2003.pdf
Center for Public Program
Evaluation
13
AEA Guiding Principles for Evaluators
•
•
•
•
•
Systematic Inquiry
Competence
Integrity/Honesty
Respect for People
Responsibilities for General & Public Welfare
http://www.eval.org/p/cm/ld/fid=51
Center for Public Program
Evaluation
14
Institute of Internal Auditors
• Standards for the Professional Practice of
Internal Auditing
Discusses evaluation of risk and controls,
and requires audit results to be based on
analysis and evaluation.
No “Evaluation” standard as such
https://na.theiia.org/standards-guidance/mandatoryguidance/Pages/Standards.aspx
Center for Public Program
Evaluation
15
Inspections, Evaluations, and
Reviews—AIG Definition
• OIGs may have responsibilities that go beyond audits and investigations.
• This additional work includes inspecting, evaluating, reviewing, studying,
and/or analyzing government operations and programs for the purposes of
providing information for decision-making, and of making
recommendations to improve programs, policies, or procedures.
• The objectives of these processes include: providing a source of factual and
analytical information, monitoring compliance, measuring performance,
and assessing the efficiency and effectiveness of operations.
Source: Association of Inspectors General
Center for Public Program
Evaluation
16
Inspections and Evaluations—Federal
IG Definition
Systematic and independent assessments of the design,
implementation, and/or results of an Agency’s operations,
programs, or policies.
They can be used to determine efficiency, effectiveness,
impact, and/or sustainability of agency operations, programs,
or polices.
Source: Council of the Inspectors General
on Integrity and Efficiency
Center for Public Program
Evaluation
17
Inspections and Evaluations—Federal
IG Definition
• They provide information that is timely, credible, and useful for
agency managers, policymakers, and others.
▫
▫
▫
▫
▫
▫
providing factual and analytical information;
measuring performance;
determining compliance with applicable law, regulation, and/or policy
identifying savings and funds put to better use;
sharing best practices or promising approaches;
assessing allegations of fraud, waste, abuse, and mismanagement.
Source: Council of the Inspectors General
on Integrity and Efficiency
Center for Public Program
Evaluation
18
Center for Public Program
Evaluation
The Proving Professions
Center for Public Program
Evaluation
19
Auditors and Evaluators
Audit
Evaluation
Center for Public Program
Evaluation
20
Auditors and Evaluators
Audit
Evaluation
Center for Public Program
Evaluation
21
Auditors and Evaluators
Audit
Evaluation
Issue Analysis
Financial Audits
Attestation Audits
Non-Audit Services
Inspections
Performance
Audit
Performance
Evaluation
Center for Public Program
Evaluation
Reviews
A Multitude of Analysis
Types
22
Other Professional Disciplines
Evaluators/Inspectors Come From Many Professions
•
•
•
•
•
•
Evaluation
Inspection
Audit
Social Science
Economics
Operations Research
Analysis
• Policy Analysis
• Public Administration
• Program Specialties
▫
▫
▫
▫
▫
Health Care
Housing
Defense
Trade
International Assistance
Center for Public Program
Evaluation
23
Legal
Analysis
Public
Administration
Applied
Social
Research
Inspections
Evaluation
Statistics
Economics
Operations
Research
Psychology
Center for Public Program
Evaluation
24
Public
Applied
Legal Administration
Social
Analysis
Research
Inspections
Economics
Evaluation
Operations
Research
Statistics
Psychology
Center for Public Program
Evaluation
25
What About Investigators?
Evaluators
?
Investigators
Center for Public Program
Evaluation
26
Useful Interplay
Investigators
Fraud
leads
Fraud
trends
Evaluators
Center for Public Program
Evaluation
Tech
assist
27
Center for Public Program
Evaluation
Evaluation Types and
Approaches
Center for Public Program
Evaluation
28
Evaluation Types and Approaches
• There are many legitimate and useful forms of
evaluations
• Each has its purpose. The methodological
approach must be adapted to it
• Generally accepted professional guidelines are
available to guide the work.
Center for Public Program
Evaluation
29
Center for Public Program
Evaluation
As A Program Matures
The Evaluation Questions Change
How is the
implementation
going?
Any Issues?
Are service
providers in
compliance with
program rules?
Are some service providers
doing better than others?
How so?
Are the required
services being
delivered?
Have conditions
improved as
intended?
Are impacted citizens
or businesses satisfied
with program results?
How do the services stack up to goals and standards?
Center for Public Program
Evaluation
30
Center for Public Program
Evaluation
And So Do the Analytic
Approaches and Methods Used
Some inspections and evaluations need to be
done fast
Others need more time
Some require more resources than others
All must be done professionally
Following are some evaluation and inspections
types commonly used today.
Center for Public Program
Evaluation
31
Center for Public Program
Evaluation
Inspection/Evaluation Types
•
•
•
•
•
•
•
Compliance Review
Performance Review
Landscape Survey
Early Implementation Reviews
Impact Assessment
Report Card
Process Review
Center for Public Program
Evaluation
32
Center for Public Program
Evaluation
More Inspection/Evaluation Types
•
•
•
•
•
•
•
“Just the Facts” Review
Report from “On the Ground”
“Try it out” Reviews
Client Satisfaction Surveys
Before and After Assessment
Control System Survey
Program Logic Model
Center for Public Program
Evaluation
33
Center for Public Program
Evaluation
Analytic Frameworks
Center for Public Program
Evaluation
34
Center for Public Program
Evaluation
The Standard IG Analytic Model
• FINDINGS
-- Criteria
-- Condition
-- Effect
-- Cause
• RECOMMENDATIONS
Center for Public Program
Evaluation
35
Center for Public Program
Evaluation
What the Federal IG Blue Book Says
“Findings often have been regarded as containing the elements of
criteria, condition, effect, and, when problems are found, cause.
However, the elements needed for a finding depend entirely on the
objectives of the inspection. Thus, a finding or set of findings is
complete to the extent that the inspection objectives are satisfied and
the report clearly relates those objectives to the applicable elements of
a finding.”
Center for Public Program
Evaluation
36
Two Analytic Frameworks
Performance Audit
Performance Evaluation
• Findings
▫
▫
▫
▫
• Objective
Criteria
Condition
Effect
Cause
• Questions to Be
Answered
• Analytic Approach
▫ Mixed Methods
▫ Qualitative
▫ Quantitative
• Evidence
▫
▫
▫
▫
Physical
Documentary
Testimonial
Analytic
• Findings
• Recommendations
• Recommendations
Center for Public Program
Evaluation
37
Center for Public Program
Evaluation
METHODOLOGIES
Center for Public Program
Evaluation
38
Center for Public Program
Evaluation
The Fundamentals
• The Correct Methods to Use Are the
Ones Most Compatible With the Purpose
of the Study
• Always Use More Than One Method
▫ At least one quantitative
▫ At least one qualitative
Center for Public Program
Evaluation
39
Center for Public Program
Evaluation
Quantitative and Qualitative Methods
• Quantitative methods involve things you can count
▫ Even if they are subjective answers to a closed ended questionnaire
• Qualitative methods are applied to the interpretation of statements
▫ Especially from interviews
▫ But also from observations and in documents
• Reviews are far more compelling if both methods are used
▫ Otherwise people will say:
“Numbers always lie”
or
“They always say that”
Center for Public Program
Evaluation
40
Center for Public Program
Evaluation
The Most Common Methods Used
by Evaluators
•
•
•
•
•
•
•
Background Search
On-Site Inspections
Review of Administrative Files
Sampling
Data Collection Instruments
Surveys
Focus Groups
Center for Public Program
Evaluation
41
Center for Public Program
Evaluation
Some Exercises
Center for Public Program
Evaluation
42
You Be the Evaluator
• Early implementation of emergency response
systems after 9-11
• Frequency of and criteria for home visits to
foster care families
• Pharmacies with extreme outlier filling of
prescriptions for dangerous, addictive drugs
Center for Public Program
Evaluation
43
Center for Public Program
Evaluation
The Quick Start
1. State the purpose
2. Ask two or three key questions
3. Choose the methods for each question

Quantitative

Qualitative
4. Conduct the Field Work
5. Analyze the data
6. Construct Findings and Recommendations
7. Draft the report
Center for Public Program
Evaluation
44
Discussion
1. Would your approach work as either an audit or
evaluation?
2. Who will use the findings and recommendations?
3. How will you know that the recommendations
have been acted upon?
4. What can you expect will be the result?
Center for Public Program
Evaluation
45
Center for Public Program
Evaluation
Organizing and Managing
the Evaluation Function
Center for Public Program
Evaluation
46
Possible Organizational Location
Inspector General
Audit
Evaluation
Investigation
Evaluation
Center for Public Program
Evaluation
Evaluation
Evaluation
47
Managing an Evaluation Office
Work Planning
Operating Procedures
Quality Control
Quality Assurance
Professional Development
These are
generally
similar to
audit
practices
Take Them Seriously !
Center for Public Program
Evaluation
48
Training Priorities for Evaluators
Highest Priority
Other Worthwhile Skills
• General Evaluation Concepts
• Writing to the Point
• Focus Groups
• Data Base Development
• Randomized Control Trials
•
•
•
•
•
•
Survey Design
Interviewing
Descriptive Statistics
Sampling
Spread Sheets
Graphs
• Team Leadership
• Programmatic Knowledge
Center for Public Program
Evaluation
49
Work Planning
• Combine Audit and Evaluation
• Focus on risk and return on investment
• Use a Tactical Approach
▫ To systematically fill gaps
▫ To develop staff capacity
▫ To keep up the attention on important areas
Center for Public Program
Evaluation
50
Center for Public Program
Evaluation
A strategic plan might
consist of several families
of program studies, some
in the early stages, some
further along, even in final
stages
Program
Area A
Program
Area B
Program
Area C
Program
Area D
Center for Public Program
Evaluation
51
Public
Applied
Legal Administration
Social
Analysis
Research
Listen to What
They Have to Say
Inspections
Economics
Evaluation
And
Audit
Operations
Research
Statistics
Psychology
Center for Public Program
Evaluation
52
Good Luck!
Center for Public Program
Evaluation
53
George F. Grob
Center for Public Program Evaluation
540-454-2888
GeorgefGrob@CS.com
Center for Public Program
Evaluation