Subrecipient Risk Assessment
Robert Prentiss
Sr. Grants and Contracts Specialist,
The University of Texas at Austin
Susan Wyatt (Sedwick) Linehan
Consulting Associate, Attain LLC
Anita Mills
Solutions Consultant, Evisions
Accountability Timeline
2009 False Claims Act and Fraud Enforcement and Recovery Act (FERA)
2006
FFATA
Signed
2007
2009 ARRA
2010
FFATA
Reporting
2014
Digital Accountability and Transparency Act of 2014
Omni
Circular
Recent Finding
Criminal
Charges
Fraud
Happened
Institution
Allegations
2014
2008-2009
Morgan State
NSF Grant – Paid personal expenses and
work never completed - Facing 20 year
2013
2006-2009
University of
Maryland
P-card Fraud and paying Fake Companies
2011
1999-2009
Florida State
University
Fraudulent Contracts on NSF grant
2006
2001-2002
UMass and Yale
Overstate cost on subcontracts
UG Sub Risk Assessment Requirements
• §200.331 requires an evaluation of each
sub’s risk of noncompliance
– Prior experience
– Results of previous audits including project
specific
– New personnel or substantially revised systems
– Existence of other Federal awards
– Other considerations are allowed
§200.110 Effective/Applicability
• For awards with incremental funding under
UG, subaward flow down will need to be
addressed.
• Agencies have the option of considering
incremental funding actions under the UG on
previously issued awards to be opportunities
to change award terms and conditions with
the first increment issued on or after
12/26/14.
Subrecipient Risk Assessment
The FDP has been developing a questionnaire
to assess subrecipient risk.
UT-Austin has been using draft versions of the
questionnaire for over a year.
We’ve got enough data to start drawing some
conclusions about how it works in practice.
The FDP RAQ
The Risk Assessment Questionnaire (RAQ)
consists of fifteen unscored (yes/no) questions,
and thirteen scored questions.
Of the scored questions, six are related to the
institution, and seven are project-specific.
The FDP RAQ
There are some ambiguities in the questions,
but the questionnaire is intended to be fairly
straightforward and easy to complete.
The more difficult questions are:
 What does the score mean?
 What should be done with the score?
Institutional Questions
Maximum Score
1) Location
9
2) Type of organization
6
3) Negotiated IDC rate agreement
3
4) Audit results
6
5) Maturity
6
6) Conflict of interest experience
6
36
Project Questions
Maximum Score
1) Prime sponsor type
8
2) Prime award type
6
3) Subaward amount
3
4) Percent of prime award
9
5) Human or animal subjects
9
6) Scope of work and deliverables
6
7) Place of performance
6
47
Subaward A
Subrecipient is a large public university in
Texas.
Prime award is a research grant from a large
federal agency.
Subaward A
Institutional Questions
1)
2)
3)
4)
5)
U.S. based institution
University
Negotiated IDC rate
Satisfactory A-133 audit
10+ years subrecipient experience
6) Conflict of interest experience
Score
0
0
0
0
0
0
0
Subaward A
Project Questions
1)
2)
3)
4)
5)
6)
Routine granting agency
Grant
Outgoing funds > $650,000
25%-49% of prime award
No human/animal subjects
Responsible for tangible products
7) All work at subrecipient institution
Score
0
0
3
3
0
1
0
7
Subaward A
Institutional Score
0
Project Score
Total Score
7
7
Subaward B
Subrecipient is a non-profit research
organization.
Prime award is a cooperative agreement from
a large federal agency.
Subaward B
Institutional Questions
1)
2)
3)
4)
5)
U.S. based institution
Non-profit
Negotiated IDC rate
Satisfactory A-133 audit
10+ years subrecipient experience
6) Conflict of interest experience
Score
0
4
0
0
0
0
4
Subaward B
Project Questions
1)
2)
3)
4)
5)
6)
More stringent federal sponsor
Cooperative agreement
Outgoing funds > $650,000
0-24% of prime award
No human/animal subjects
Continuation funding tied to sub performance
7) Some work at prime institution and in the field
Score
4
2
3
0
0
6
4
19
Subaward B
Institutional Score
4
Project Score
Total Score
19
23
Subaward C
Subrecipient is a small company.
Prime award is a contract from a state agency.
Subaward C
Institutional Questions
1)
2)
3)
4)
5)
Score
U.S. based institution
Industry
No IDC requested
No audit
5-9 years subrecipient experience
0
6
0
6
2
6) No conflict of interest experience
6
20
Subaward C
Project Questions
1)
2)
3)
4)
5)
6)
State sponsor
Contract
Outgoing funds $150,000 - $649,999
Over 50% of prime award
No human/animal subjects
Responsible for tangible products
7) All work at subrecipient institution
Score
4
6
2
9
0
1
0
22
Subaward C
Institutional Score
20
Project Score
Total Score
22
42
Quantifying Risk
The FDP questionnaire was designed with the
assumption that risk of subrecipient noncompliance is always multifaceted.
No single factor would cause a subaward to
be considered high risk.
Quantifying Risk
Like most quantifications of complex ideas, the
score can be accurate, but not precise.
Accuracy vs. Precision
An example: walkscore.com
2900 Guadalupe Street
4300 Avenue H
The Approach
Subawards will be grouped into pools of low,
moderate, and high risk.
The Approach
Moderate and high risk subawards will receive
an additional, comprehensive review at the
pre-award stage. Non-quantifiable risks can be
assessed and considered at this point.
High risk subawards will be the focus of our
post-award desk review program.
Subaward scores at UT-Austin
Institutional Score
Project Score
Total Score
Average
4
8
12
Average
Deviation
4
5
8
Number of subawards in each
scoring range (total score)
Institutional Score Correlations
The maturity of the subrecipient correlates
moderately with the type, indirect cost rate
agreement status, and A-133 audit status.
Less mature subrecipients tend to:
a) be in industry
b) not have a negotiated IDC rate agreement
c) not have an A-133 audit
Project Score Correlations
Prime sponsor type correlates moderately
with prime award type.
Federal sponsors tend to award grants, while
industry sponsors are more likely to award
contracts.
Correlations: a problem?
We have much more quantifiable information
about the project then we do about the
institution.
Domestic colleges and universities with clean
A-133 audits almost always have an
institutional score of zero.
Subaward scores at UT-Austin
Average
Average
Deviation
Average + Average + 2 x
Average Dev.
Ave. Dev.
Institutional Score
4
4
8
12
Project Score
8
5
13
18
Total Score
12
8
20
28
Subaward scores at UT-Austin
Average
Average
Deviation
Average + Average + 2 x
Average Dev.
Ave. Dev.
Institutional Score
4
4
8
12
Project Score
8
5
13
18
Total Score
12
8
20
28
Review of subawards A, B, and C
Subaward A
Subaward B
Subaward C
Institutional Score
0
4
20
Project Score
7
19
22
Total Score
7
23
42
Review of subawards A, B, and C
Subaward A
Subaward B
Subaward C
Institutional Score
0
4
20
Project Score
7
19
22
Total Score
7
23
42
The Plan
Moderate risk threshold will be set by the
institutional and project scores.
Moderate risk begins when the score equals
the average plus the average deviation.
High risk threshold will be set by the total
score.
High risk begins when the score equals the
average plus twice the average deviation.
Subaward scores at UT-Austin
Average
Average
Deviation
Average + Average + 2 x
Average Dev.
Ave. Dev.
Institutional Score
4
4
8
12
Project Score
8
5
13
18
Total Score
12
8
20
28
Risk Pools
Risk Pools
Institutional Risk
How much do scores vary over time?
What will the institutional risk profile look like
in 5 years? 10 years?
How much follow-up work will desk reviews
require?
When will site visits be necessary?
Institutional Risk
Institutional Risk
In order to mitigate institutional risk,
thresholds will be reset every six months,
based on the scores of the previous twelve
months.
Institutional Risk
Current thresholds were set on
January 1st:
If they were reset today, they
would be:
Moderate
Risk
High Risk
Moderate
Risk
High Risk
Institutional Score
8
Institutional Score
8
Project Score
13
Project Score
12
Total Score
28
Total Score
27
Other Approaches
Three institutions had responses to UTAustin’s methodology.
We’ll refer to them as institutions X,Y, and Z.
Institution X
“Have you lost your mind?”
Institution X
We’ll take a much more straightforward
approach.
We’ll use a shorter, simplified version of the
FDP questionnaire.
We’ll set fixed thresholds based on our
collective judgment and experience.
Institution Y
The total score should set the moderate risk
threshold.
The institutional and project scores should set
the high risk threshold.
Institution Z
There should only be two risk pools: low and
high.
Total score will be the sole determinant.
Additional questions will be added to better
assess the risk of subawards issued from state
prime awards.
Institution Z
Statistician:
 Nothing about the FDP questionnaire or UTAustin’s methodology is scientific.
UT-Austin:
 We agree! It couldn’t possibly be at this point.
But that doesn’t mean it’s not valuable.
FDP/COGR
From the June 2014 White Paper on the
Uniform Guidance:
 A major issue of interest to Universities is the
unnecessary and unproductive duplication of audit
reviews and a pass-through entity’s inability to rely
on auditor and federal agency management
decisions for entities already subject to the A-133
process.
The Problem
No individual institution looks forward to a
desk review that reveals significant problems
with a subrecipient.
But it’s in our collective interest to avoid
unnecessary desk reviews of compliant
institutions.
The Solution
Therefore our tools in assessing risk should
be sharp, so that they may be used sparingly.
The only way to improve them is to collect
and analyze data about how our subrecipient
risk assessments correspond with the results
of our desk reviews.
Questions
How will your institution be conducting
subrecipient risk assessments?
What are your thoughts on the FDP
questionnaire and UT-Austin’s implementation
of it?
Contacts
Robert Prentiss
rprentiss@austin.utexas.edu
Susan Wyatt (Sedwick) Linehan
ssedwick@attain.com
Anita Mills
Anita.Mills@Evisions.com