1
Find Strength in Numbers: Sampling Techniques to Improve
Financial Audit, Control, & Program Performance
April 18, 2012
AGA Training Audio Conference
Albert J. Lee, Ph.D., CEO, Summit Consulting LLC
Denise Wu, CGFM, CPA, Partner, CliftonLarsonAllen LLP
About CliftonLarsonAllen
2





CliftonLarsonAllen (CLA) is the union of Clifton Gunderson and
LarsonAllen
This merger brings over 100 years of combined experience
The nation’s newest top ten public accounting and consulting firm
More than 3,600 professionals operating from more than 90
offices
CLA’s Government niche (Federal, state and local governments)
is a significant part of the company’s firm-wide public sector
practice
About CliftonLarsonAllen (cont’d)
3

Assurance and related IT services include a sampling of
the following:
CFO and Non-CFO Act audits and advisory services
 State and Local Government Financial and Compliance Audits
 Internal control/FMFIA reviews/other financial management
advisory services
 Statement on Auditing Standards SSAE-16 examinations
 Performance Audits
 Federal Information Security Management Act services
 Information system security reviews

About Summit Consulting LLC
4




Founded in 2003, specializes in cutting-edge quantitative
techniques, including statistical sampling, financial modeling, and
program evaluation
More than 60% of staff professionals have advanced degrees in
economics, statistics, finance, and other quantitative disciplines
Summit has provided sophisticated consulting services for many
federal agencies, including the Treasury, HUD, USDA, VA, DOT,
DOE, and DOL
Affiliated experts and staff members testified at courts of law and
other judicial settings for matters involving economics and statistics
Presentation Objectives
5
1.
2.
3.
4.
5.
Define audit sampling and distinguish attribute vs.
variables sampling, and statistical vs. non-statistical
sampling.
Discuss relevant activities of a sampling project.
Define the project team members’ roles and
responsibilities.
Provide real life case studies to highlight the use of
sampling and sampling best practices.
Provide sampling best practices for performing audit,
internal control, and program evaluation sampling
projects.
Presentation Outline
6







Why sampling?
What comprises a sampling project?
Who does what during a sampling project?
Types of sampling designs
Real life examples of sampling design
What are some of sampling’s best practices?
How should sampling results be used?
Why Sampling?
7


Primary objective of sampling is to create a subset of
elements from a population that is as similar to the
population as possible to allow users to evaluate
certain characteristics of the population
Benefits of sampling:
Reduces costs associated with evaluation
 Increases efficiency of population evaluation
 Provides the ability to quantify variability (i.e., range of
values) in the population
 Allows measurement of the sufficiency of the evidence
obtained


Attribute sampling (test for deviations) vs. variables
sampling (test for misstatements)
Why Sampling? (cont’d)
8


Statistical (projectable) sampling vs. non-statistical
(judgmental) sampling
Based on a statistical sample, important projections
can be made about the population:
Averages, totals, ratios
 Data validation and quality
 Trends identification
 Projection of errors and identification of misstatements
 Rate of errors/deviations
 Sampling risk can be quantified

What Comprises a Sampling Project?
9
Planning
Sampling Frame
Formation and
Population
Analysis
Sample Design
Implementation
Sample Size
Determination
Sample
Selection
Sample Testing
and Evaluation
Conclusion
& Reporting
Extrapolation,
Conclusion, and
Reporting
What Are the Planning Activities of a
Sampling Project?
10

Sampling Frame Formation and Population Analysis
Designed to ensure the dataset used for selecting the
sample corresponds to the scope of the project and
adequately addresses risk areas.
 Clearly define the objective of the project and the
population subject to testing.
 Determine completeness of the dataset and validity of
data.
 Analyze the population characteristics and assign risks to
the various characteristics/strata.
 Deploy data checks to eliminate duplicates and reversals.

What Are the Planning Activities of a
Sampling Project? (cont’d)
11

Sample Design
Dictated by the project’s objectives with consideration for
the project’s constraints.
 Two broad categories of statistical sampling:
 Attribute sampling – used primarily in tests of controls
 Variable sampling – used to estimate/project a quantity
and often used to test the monetary value of account
balances
 Clearly define the sampling unit.
 Create subgroups of observations with similar characteristics
that correlate with audit risk.

What Are the Planning Activities of a
Sampling Project? (cont’d)
12

Sample Size Determination
Predicated on the sample design, risk assessment, and
precision requirements.
 Estimate the expected misstatement/expected deviation
rate to achieve a prescribed level of precision.
 Tailor sample size to address emerging risks or issues.
 Balance precision with team resources (i.e., time, people,
and money).

What Are the Implementation Activities
of a Sampling Project?
13

Sample Selection
 Select the items from the sampling frame that will
be tested.
 Sample items should be selected in such a way that
the sample is representative of the population.
 Sample items should have a fixed, known and
positive probability of selection.
What Are the Implementation Activities
of a Sampling Project? (cont’d)
14

Sample Testing and Evaluation
 Test the sampled items and evaluate sample results
mathematically.
 Determine whether unexpected test results could
skew conclusions.
 Determine whether initial assessments should be
revised and the sample expanded.
What Are the Conclusion & Reporting
Activities of a Sampling Project?
15

Extrapolation and Conclusion
Project the sample findings to the population.
 Conclude on the rate of error of attribute sampling or dollar
misstatement in variables sampling.


Reporting

Document the sampling plan, including evaluation
considerations, calculation of the point estimates (best guess
value) along with the confidence intervals, and conclusions
reached regarding the population.
Who Does What During a Sampling
Project?
16
Planning
Sampling Frame
Formation and
Population
Analysis
• Project Team and
Statisticians
Sample Design
• Statisticians and
Project Team
Implementation
Sample Size
Determination
• Statisticians and
Project Team
Sample
Selection
• Statisticians
Sample Testing
and Evaluation
• Project Team
Conclusion
& Reporting
Extrapolation,
Conclusion, and
Reporting
• Project Team and
Statisticians
Who Wants to be a Millionaire?
17



Background: Agency uses statistical sampling to
collect improper payments. By law, the agency can
only collect the lower bound of the improper payment
estimate.
Objective: To collect as close to the true improper
payment amount as possible.
Challenge: Historically, using a simple random sample
design, the agency under collects due to the lack of
precision.

The point estimate (e.g., $15 million) is much greater than
the lower bound (e.g., $300K).
Who Wants to be a Millionaire?
(cont’d)
18

Solution: Stratify the population according to risk
characteristics. Using the same sample size as the
simple random sample design, precision improves and
the lower bound gets closer to the point estimate.

Point estimate remains $15 million, but the lower bound
increases (e.g., to $3 million).
Frequent Flyer Miles, Anyone?
19



Background: Agency uses statistical sampling to
identify assets that are misstated on the general
ledger.
Objective: To determine the size and extent of
misstatement of the agency’s recorded asset balance.
Challenge: Assets are located in all fifty states and
six territories. Using a simple random sample it is
likely to select as many locations as the sample size
(i.e., a great deal of traveling).
Frequent Flyer Miles, Anyone? (cont’d)
20


Solution: Implement a multistage sampling design in
which locations are selected in the first stage and
assets to be tested are selected in the second stage
within the selected locations. The sample controls the
number of locations visited and the number of assets
evaluated.
Benefit: Substantially reduce travel costs and
significantly enhance audit efficiency and
effectiveness.
Should I Stop or Should I Go
21



Background: Agency uses statistical sampling to
determine the extent of misstatement in the UDO
balance at year end.
Objective: To achieve precision by evaluating as small
a sample size as possible.
Challenge: No prior history to estimate the sample
size for a given precision. It is 9/30 and only two
weeks remain to finish the audit.
Should I Stop or Should I Go (cont’d)
22


Solution: Prepare a sample with a randomized sortorder and evaluate sample batches sequentially to
determine if the prescribed precision requirements
have been achieved. If yes, STOP. If no, evaluate the
next batch according to the random order.
Benefit: Performing the least amount of work to
achieve the prescribed precision requirements.
Sampling Best Practices - Audit
23
1.
2.
3.
4.
5.
6.
Sampling frame corresponds to the target population.
Every transaction in the frame is substantive (i.e., net out all
the reversals and eliminate phantom transactions).
Use stratification to test all large transactions with a dollar
amount above a certain threshold.
Define methodology for determining the point estimate:
Mean, proportion, totals, ratios.
Create a unique sort, and set random seeds to ensure sample
can be reproduced.
Careful documentation of sampling design, sample size,
expected precisions, point estimates, and confidence
intervals.
Sampling Best Practices – Program
Performance Evaluation
24
1.
2.
3.
4.
5.
6.
Sampling frame corresponds to the target population
Every subject in the frame is substantive, i.e., eliminate
phantom subjects
Select sample designs according to data structure and
availability
Define performance metrics and all the components
Create a unique sort, set random seeds to ensure sample
reproducibility
Careful documentation, sampling design, sample size,
expected precisions, point estimates, and confident intervals
How Should Sampling Results be Used?
25
1.
2.
3.
4.
Point estimate from a statistical sample is the unbiased and
best guess of a population’s true value
Precision and confidence interval quantify how point
estimates are likely to change from one sample to the next
 Lower (upper) bound of a confidence interval is a
threshold value below (above) which a population value is
unlikely
These statistics determine the likelihood of an estimated
population value (e.g., population error rate)
Based on these likelihoods, certain population values could
be rejected for being extremely unlikely
Contact Information
26
Albert J. Lee, Ph.D., CEO, Summit Consulting LLC
albert.lee@summitllc.us
202-407-8300
Denise Wu, CGFM, CPA, Partner, CliftonLarsonAllen LLP
Denise.Wu@cliftonlarsonallen.com
301-931-2050
Questions?
27