AFP and Telework
Program Review
Jeremy Buzzell
and
Robert Groenendaal
Requirements for Program Review
The Handbook for the Discretionary Grant Process
Section 5.3.2 Monitoring Policy
It is ED’s policy to monitor active discretionary
grants with a focus on technical assistance,
continuous improvement, and attaining promised
results. Monitoring a grantee shall continue for as
long as ED retains a residual financial interest in the
project, whether or not ED is providing active grant
support.
2
Requirements for Program Review
Section 5.3.2 Monitoring Policy
ED staff is to monitor each grantee to the
extent appropriate so as to achieve expected
results under approved performance measures,
while assuring compliance with grant
requirements. Existing requirements in
EDGAR extend equally to all grantees and
their partners.
3
Requirements for Program Review
Section 5.3.3 Monitoring Purpose
Systematic and regular monitoring
by ED staff of a grantee’s activities
measures the project quality and
progress, including strengths and
weaknesses.
4
Requirements for Program Review
1. A grantee’s project must:
a) Conform to the grantee’s approved application and
any approved revisions, as well as the effectiveness and
quality of the project (Program Management);
b) Progress against previously established performance
measures (Performance Measurement);
c) Adhere to laws, regulations, conditions of the grant,
certifications and assurances (Compliance); and
d) Manage Federal funds according to Federal cash
management requirements, including expenditures of
funds for authorized purposes (Fiscal Accountability).
5
Requirements for Program Review
2. Regular monitoring enables ED staff
to provide customized technical
assistance, appropriate feedback, and
follow-up to help grantees to:
• Improve areas of need
• Identify project strengths
• Recognize significant achievements
6
Development of Program Review
•
•
•
•
•
•
•
•
Open forum in Denver
Review of existing protocols
ED training in auditing
Developed proposals
NATTAP Blog
Teleconference of directors in September 2007
Refined proposals
Internal review and approval of proposals
7
What we learned from the process . . .
•
•
•
•
•
Proof of compliance should not be based on
data alone.
Every grantee should be assessed the same
way but can show compliance in its own way.
Grantees should not be compared to each
other.
Compliance should not be black or white.
Look for the positive as well as the negative.
8
The result . . .
Program Reviews allow for a holistic
explanation of the program (through
documents, data, and discussion).
• This explanation is done for peer reviewers
and RSA.
• The peer reviewers and RSA answer a series
of questions about the program to determine
the extent of compliance.
•
9
The effect . . .
Pros
• Context is considered
• Focuses on what you can control
• Appropriate to size and funding of program
Cons
• Time-intensive (for all)
• Highly subjective
• Learning curve for RSA and grantees
10
Why this approach?
•
•
•
•
•
•
On-site reviews are not feasible.
RSA staff are limited and do not come from
the field.
Data is not reliable.
Lack of basis for self-assessment.
Program continues to evolve.
One size fits all.
11
Next Steps with Program Review
•
•
•
•
•
•
•
Final drafts cleared by RSA.
Final drafts reviewed by Risk Management.
Official materials shared with AFP/Telework
Programs.
Choose and notify first round of programs.
AFP/Telework Programs submit FY 2008 data.
Train third-party (peer) reviewers.
First peer review in January 2009.
12
Role of Lead Agencies and
Community Based Organizations in
Program Reviews
•
•
•
•
The LEAD AGENCY is being reviewed.
Participation from BOTH the Lead Agency and
CBO is expected.
Participation of all personnel being paid using
grant funds is required (see exception in the next
bullet).
Participation of the Certifying Representative is
encouraged but not required.
13
How are AFP/Telework Programs
selected for review?
Random selection of an equal number of
grantees in two categories:
(1) Title III AFP/Telework grantees claimed
by a Statewide AT Program as a state
financing activity;
OR
(2) Title III AFP/Telework grantees not
claimed by a Statewide AT Program as a state
financing activity.
14
How are AFP/Telework Programs
selected for review?
Exceptions to random selection:
• Recent major change in the
AFP/Telework Program or State Plan
for AT
• Recent major disaster
• RSA is concerned
15
How are AFP/Telework Programs
selected for review?
6 grantees per cycle (3 x 3)
• A “cycle” is a fiscal year (Oct. 1 to Sept. 30)
• 9 weeks per grantee (some program reviews
may overlap)
• First year of program reviews:
• Jan. 1 to Sept. 30
• Only 3-4 reviews
•
16
What happens when an
AFP/Telework Program is selected?
RSA notifies grantee in spring prior to
next cycle.
• RSA and grantee negotiate 9-week
window within next cycle.
• RSA recruits third-party (peer)
reviewers.
• Grantee begins preparing for review.
•
17
How does a grantee prepare for
the review?
A review assesses two “Core
Components” of grant implementation,
each in a slightly different way.
• Core Component 1 – Program
Management
• Core Component 2 – Program
Performance
18
Core Components of
Program Review
Elements of Program
Management
• Fiscal Management
• Contract Oversight
• Personnel Management
Elements of Program
Performance
• Emphasis and Expansion
of Consumer Choice and
Control
• Timely and Equitable
Access to Financial
Assistance
• Capacity to Continue on a
Permanent Basis
19
Program Management
The review of Program Management verifies
that a grant is managed efficiently and
appropriately in accordance with title III, the
NFP, EDGAR, and OMB Circulars, as
applicable. This includes:
• Fiscal Management
• Contract Oversight
• Personnel Management
20
Fiscal Management
EDGAR requirements:
• Fiscal control and accounting procedures to insure proper
disbursement of and accounting for Federal funds; controls and
procedures adequate to establish that such funds have not been
used in violation of applicable statutes.
• Records that fully show the amount of funds under the grant,
how the grantee uses the funds, the total cost of the project, the
share of the cost provided by other sources, and other records to
facilitate an effective audit.
• Audits in accordance with OMB A-133.
21
Fiscal Management
Title III or NFP requirements:
• Assurance that the grantee will provide the non-federal share
of the cost of the program in cash, from state, local, or private
sources.
• Assurance that the grant funds will be used to supplement and
not supplant other federal, state, and local public funds.
• Assurance that the grant funds will be placed in a permanent
separate account and identified and accounted for separately
from any other fund.
• Assurance that the funds are administered, expended, and
invested in accordance with title III or the NFP.
22
How to prepare for a review of
Fiscal Management
Submit to RSA:
• Expenditure report.
• Proof of title III or NFP fiscal management assurances.
• Approved indirect cost rate agreement/cost allocation plan
or questionnaire provided by RSA (applies to both Lead
Agencies and Community Based Organizations.)
• Copies of most recent audit reports or questionnaire
provided by RSA (applies to both Lead Agencies and CBOs).
(In addition, RSA will look at ED 524Bs and GAPS records.)
23
Expenditure Report
1 for Lead Agency to administer the grant, 1 for CBO to
implement the program.
• From the most recent federal fiscal year.
• There is no “format”, but RSA must be able to understand the
reports.
• Show use of all federal and match funds as well as income from
interest, investments, loan repayments, and other sources.
• Show use of funds used for administration of the program
separately from the amount used for loans and other program
purposes.
• Show the amount of funds used for indirect costs.
•
24
Expenditure Report
What RSA is looking for:
• Existence, quality, and legitimacy
of report
• Permanent separate account
• Compliance with indirect cost
limits
25
Proof of Fiscal Management Assurances
What RSA is looking for:
The sources, amounts, and appropriateness of
match provided for all grants for the AFP/Telework
Program.
• The Lead Agency’s and CBO’s internal controls
that ensure all principal and interest from the
permanent separate account (a) support only
AFP/Telework activities and (b) supplement and do
not supplant other federal, state, and local public
funds expended to provide financial loans.
•
26
Indirect Cost Rates
What RSA is looking for:
Existence of agreement
• Appropriate kind of agreement
• Handling of indirect costs between Lead
Agency and CBO
• Amount of indirect will be seen in
expenditure reports
•
27
Audits
What RSA is looking for:
•
•
•
An audit was done (if required)
Findings of the audit
Resolution of findings
28
ED 524B Forms and GAPS Records
What RSA is looking for:
•
Completeness and accuracy of
financial reports
29
Contract Oversight
EDGAR requires grantees to:
• Directly administer or supervise the administration
of each project.
• Have procedures for providing TA, evaluation and
for performing other administrative
responsibilities necessary to ensure compliance.
• Monitor activities to assure compliance with
applicable Federal requirements and that
performance goals are being achieved.
30
Contract Oversight
Grantees are required to enter into a contract with a CBO or
group of CBOs that has individuals with disabilities
involved in the organizational decision-making to
administer the AFP/Telework Program. The contract must:
• Include a provision requiring funds be administered
consistent with title III or the NFP.
• Include any provision the Secretary requires concerning
oversight and evaluation of federal financial interests.
• Require the CBO(s) enter into a contract with commercial
lending institutions or organizations or state financing
agencies.
31
How to prepare for a review of
Contract Oversight
Submit to RSA:
• A dated copy of the grantee’s most recent
contract with the CBO
32
How to prepare for a review of
Contract Oversight
What RSA is looking for:
• Appropriate oversight to protect Federal
interests and ensure implementation
• Relevant to title III or the NFP
33
Personnel Management
OMB Circulars:
•
•
•
•
Compensation to personnel must be reasonable.
Charges to Federal awards for salaries and wages are based
on documented payrolls.
Charges for salaries and wages of employees who work
solely on a single Federal award are supported by periodic
certifications.
Charges for salaries and wages of employees who work on
multiple activities are supported by personnel activity
reports.
34
How to prepare for a review of
Personnel Management
Submit to RSA either:
(A) Position descriptions of record for those
being paid with grant funds and personnel
activity reports for them; OR
(B) Synopsis for key personnel describing his
or her role and responsibilities associated
with the grant.
35
Personnel Management
What RSA is looking for:
• Appropriate number of personnel
• Appropriate time commitments of
personnel
• Appropriate duties for personnel
36
How to prepare for a review of
Program Management
•
•
•
•
Submit all the previously described documents.
RSA reviews the documents.
RSA discusses the documents via conference call
(the grantee is responsible for bringing together
the appropriate parties).
RSA uses this information to answer questions
about program management.
• Yes/no with explanation.
37
Questions about Program Management
1. Has the grantee obtained its full matching funds from
appropriate sources? If not, does the grantee have a plan for
obtaining the remaining match funds?
2. Does the grantee account for grant funds accurately and
completely in accordance with EDGAR?
3. Does the grantee use grant funds only for purposes under,
and within the limitations of, title III or the NFP, as
applicable?
4. Does the grantee comply with indirect cost requirements?
5. Does the grantee comply with audit requirements?
38
Questions about Program Management
6. Does the grantee exercise appropriate fiscal and
performance oversight of the AFP/Telework Program through
its contract with the CBO? If not, does the Lead Agency have
a plan for improving its oversight?
7. Is the contract structured to meet the requirements and
intent of title III of the NFP, as applicable? If note does the
Lead Agency have a plan for changing its contract to make it
more consistent with title III or the NFP, as applicable?
39
Questions about Program Management
8. Does the AFP/Telework Program have an appropriate number
of staff to implement the grant? If not, does the Lead Agency have
a plan for changing its contract to make it more consistent with title
III or the NFP, as applicable?
9. Do the Lead Agency and CBO distribute the time of their
personnel appropriately to implement the grant based on their
responsibilities? If not, does the grantee have a plan to distribute
time more appropriately?
10. Do the Lead Agency and CBO use grant funds to support only
personnel with responsibilities germane to the grant?
40
Assessing Compliance of
Program Management
There is no overall rating of compliance.
For each element of Program Management, a
grantee will receive 1 of 3 ratings:
•
•
•
Compliant
In Need of Improvement
Fails to Substantially Comply
41
Assessing Compliance of
Program Management
The rating is based on how RSA answers the
questions:
Initial question: Does the AFP/Telework
Program have an appropriate number of staff
to implement the grant?
• Subquestion: If not, does the grantee have a
plan for adjusting the number of staff?
•
42
Assessing Compliance of
Program Management
•
•
•
•
If the answer to the initial question is “Yes” – the
grantee is rated as compliant.
If the answer to the initial question is “No” – the
rating depends on the answer to the subquestion.
If the answer to the subquestion is “Yes” – the
grantee is rated as in need of improvement.
If the answer to the subquestion is “No” – the
grantee is rated as failing to substantially comply.
43
Assessing Compliance of
Program Management
Questions for which “needs improvement” doesn’t apply:
•
•
•
•
•
Does the grantee account for grant funds accurately and
completely in accordance with EDGAR?
Does the grantee use grant funds only for purposes under, and
within the limitations of, title III or the NFP, as applicable?
Does the grantee comply with indirect cost requirements?
Does the grantee comply with audit requirements?
Does the grantee use grant funds to support only personnel with
responsibilities germane to the grant?
44
Assessing Compliance of
Program Management
The onus is on the grantee to present
evidence/plan sufficient to get to a “yes” answer.
RSA will ask questions or provide examples
of sufficient evidence, if needed.
• Grantee likely will know their status before
the conversation is over.
•
45
Program Performance
The review of Program Performance assesses whether
the AFP/Telework Program is implementing its grant in
accordance with title III or the NFP, as applicable. This
includes:
• Extent to which the program emphasizes and expands
consumer choice and control.
• Extent to which program provides timely and equitable
access to financial assistance.
• Extent to which the program can continue on a
permanent basis.
46
Third-Party Reviewers
Peer reviewers participate in the review of Program Performance:
3 current or former program directors (or position with
similar duties).
• At least 1 with prior review experience on each team.
• Trained, paid volunteers from a standing pool.
• No reviewers from NATTAP or NISAT.
• Participation is not anonymous, but conclusions are not
attributed to individual reviewers.
•
Peer reviewers learn about the program through review of documents,
looking at data, and discussion on teleconferences/webinars.
47
How to prepare for a review of
Program Performance
Submit to RSA:
• Narrative explaining how consumer choice and control is
emphasized and expanded.
• Copies of administrative policies and procedures manuals.
• Copies of lender agreement(s), investment plan, if
applicable, and narrative or plan for sustainability.
• Reviewers already will have AFP/Telework Program data.
There is no prescribed format for these but must be
accessible.
48
How to prepare for a review of
Program Performance
Develop teleconference/webinar
presentations that serve as a virtual on-site
visit to your program.
• Length, format, and content of presentation
are up to the program.
• The “presenter” during a teleconference or
webinar is up to the program.
• Need to be accessible.
•
49
Emphasis and Expansion of
Consumer Choice and Control
Title III or NFP requirements:
• Assurance that, and information describing
the manner in which, the grantee will expand
and emphasize consumer choice and control.
50
How to prepare for a review of
Consumer Choice and Control
Submit to RSA:
• A narrative describing how individuals with
disabilities are involved in organizational
decision-making at all organizational levels to
administer the loan program.
51
How to prepare for a review of
Consumer Choice and Control
What RSA is looking for:
• Appropriate emphasis and expansion of
involvement in organizational decisionmaking by individuals with disabilities
• The grantee should also be prepared to
discuss consumer choice and control during
webinars/teleconferences
52
Timely and Equitable Access to
Financial Assistance
Title III or NFP requirements:
•
(1)
(2)
(3)
Assurance that the grantee enter into a contract
with a CBO that has policies and procedures to:
Review and process requests for financial
assistance in a timely manner.
Ensure equitable access to all consumers for which
financing is requested through the program.
Ensure consumer-controlled oversight of the
program.
53
How to prepare for a review of Timely and
Equitable Access to Financial Assistance
Submit to RSA:
• A dated copy of its most recent revised and
adopted AFP/Telework policies and procedures
manual.
• A dated copy of its most recent revised and
adopted lender agreement(s).
54
How to prepare for a review of Timely and
Equitable Access to Financial Assistance
What RSA is looking for:
• Appropriate access to financial assistance.
• The grantee should also be prepared to
discuss how the program assures the review
and process of financial loans in a timely
manner and describe the flow of an
application for an AFP/Telework loan during
teleconferences or webinars.
55
Capacity to Continue on a
Permanent Basis
Title III or NFP requirements:
•
(1)
(2)
(3)
Assurance that the AFP/Telework Program will:
Continue on a permanent basis.
Invest the funds within the permanent separate
account in low-risk securities.
Administer the funds with prudence, discretion,
and intelligence.
56
How to prepare for a review of Capacity
to Continue on a Permanent Basis
Submit to RSA:
• A dated copy of its investment plan, if applicable, or
narrative describing how loan program funds are
invested.
• A copy of its investment portfolio.
• A dated copy of the its plan for sustainability, if one
has been developed, or narrative explaining how the
program will continue on a permanent basis.
57
Investment Plan
What RSA is looking for:
• Appropriate investment of AFP/Telework
Program funds in the permanent separate
account.
• The grantee should also be prepared to discuss
the implementation and regular revision of the
AFP/Telework Program’s investment plan.
58
Sustainability Plan
What RSA is looking for:
• Appropriate implementation of strategies for
sustaining the AFP/Telework Program.
• The grantee should also be prepared to
discuss how the program is implementing a
plan for sustainability during a
webinar/teleconference.
59
How to prepare for a review of
Program Performance
Based on documents, data, and discussion, thirdparty reviewers respond to a series of questions:
• Yes/no with explanation.
• Evidence can come from any source of the review.
• Getting a “yes” answer does not depend on an
AFP/Telework Program having any particular
structure or type.
• A “no” answer does not automatically mean a
grantee is out of compliance.
60
Questions about Program Performance
1. Is there evidence that individuals with disabilities are
involved in the administrative decision-making of the
AFP/Telework Program?
a) If not, does the grantee provide justification for the current
level of involvement of individuals with disabilities in the
administrative decision-making of the loan program?
b) If not, does the grantee have a reasonable plan for
improving the involvement of individuals with disabilities in
the administrative decision-making of the loan program?
61
Questions about Program Performance
2. Is there evidence that the AFP/Telework Program
emphasizes and expands consumer choice and control?
a) If not, does the grantee provide justification for why
the current level of consumer choice and control is
appropriate?
b) If not, does the grantee have a reasonable plan for
improving its emphasis and expansion of consumer
choice and control?
62
Questions about Program Performance
3. Is there evidence that the policies and procedures of the
AFP/Telework Program make it accessible to consumers
regardless of type of disability, age, income level, type of AT
device, equipment, or service for which financing is requested
through the program?
a) If not, does the grantee provide justification for why the
program is not accessible to all individuals with disabilities in the
State?
b) If not, does the grantee have a reasonable plan for improving
the accessibility of the program?
63
Questions about Program Performance
4. Is there evidence that the structure and practices of
the AFP/Telework Program make it possible to meet the
needs of individuals in most areas of the State?
a) If not, does the grantee provide justification for why
the areas reached in the State are sufficient?
b) If not, does the grantee have a reasonable plan for
improving their ability to reach consumers in other areas
of the State?
64
Questions about Program Performance
5. Is the lender agreement structured to promote the
goals of the AFP/Telework Program?
a) If not, does the grantee provide justification for
the structure of the current lender agreement?
b) If not, does the grantee have a reasonable plan for
improving the lender agreement?
65
Questions about Program Performance
6. Does the AFP/Telework Program administer its
funds with prudence, discretion, and intelligence?
a) If not, does the grantee provide justification for
why the program administers its funds in the current
manner?
b) If not, does the grantee have a reasonable plan for
improving the administration of program funds?
66
Questions about Program Performance
7. Does the AFP/Telework Program invest funds in lowrisk securities in which a regulated insurance company
may invest under the law of the State?
a) If not, does the grantee provide justification for why
the program invests in the manner it currently does?
b) If not, does the grantee have a reasonable plan for
investing program funds more appropriately?
67
Questions about Program Performance
8. Does the AFP/Telework Program have an investment
plan that is implemented and revised regularly?
a) If not, does the grantee provide justification for why
the program has not implemented an investment plan
that is reviewed regularly?
b) If not, does the grantee have a reasonable plan for the
implementation and regular revision of an investment
plan?
68
Questions about Program Performance
9. Is the AFP/Telework Program implementing
appropriate strategies for sustainability?
a) If not, does the grantee provide justification for
why the program is not implementing appropriate
strategies for sustainability?
b) If not, does the grantee have a plan for
sustainability that could be implemented?
69
Assessing Compliance of
Program Performance
There is no overall rating of compliance.
For each element of Program Performance, a
grantee will receive 1 of 3 ratings:
•
•
•
Compliant
In Need of Improvement
Fails to Substantially Comply
70
Assessing Compliance of
Program Performance
The rating is based on how peer reviewers answer the questions:
Initial question: Is there evidence that the structure and
practices of the AFP/Telework Program makes it possible to
meet the needs of individuals with disabilities in most areas
of the State?
• Subquestion 1: If not, does the grantee provide justification
for why the areas reached by the program in the State
currently are sufficient?
• Subquestion 2: If not, does the grantee have a reasonable
plan for improving the capacity of the program to reach
individuals in more areas of the State?
•
71
Assessing Compliance of
Program Performance
•
•
•
•
•
•
If the answer to the initial question is “Yes” – the grantee is rated as
compliant.
If the answer to the initial question is “No” – the rating depends on the
answer to the subquestions.
If the answer to subquestion 1 is “Yes” – the grantee is rated as
compliant.
If the answer to subquestion 1 is “No” – the the rating depends on the
answer to subquestion 2.
If the answer to subquestion 2 is “Yes” – the grantee is rated as in need
of improvement.
If the answer to subquestion 2 is “No” – the grantee is rated as failing
to substantially comply.
72
Assessing Compliance of
Program Performance
The onus is on the grantee to present
evidence/plan sufficient to get to a “yes” answer.
RSA and third party reviewers will ask
questions or provide examples of sufficient
evidence if needed.
• Grantee likely will know their status before
the conversation is over.
•
73
Program Data
AFP/Telework Program data will not determine
compliance.
• Data will be discussed during a teleconference
or webinar and reflected in the report:
(1) Grantees strengths and areas in need of
improvement when it comes to data; and
(2) How the grantee is learning from and using
its data.
•
74
Program Data
Data collection infrastructure:
How the data is collected.
• How the data is checked.
• Strengths and areas in need of
improvement.
•
75
Program Data
Understanding your data:
•
•
•
How the nature of your state and structure
of your program influence the data.
What your data tells you about the program
and how it influences implementation.
Strengths and areas in need of
improvement.
76
Assessing Compliance of
Program Performance
Activities of peer reviewers:
• Read documents.
• Participate in teleconferences/webinars.
• Complete review forms (answer questions).
• Meet as a group with RSA to discuss
results.
• Finalize review forms and submit to RSA.
77
Concluding the Program Review
RSA compiles Program Management and Program
Performance information into a comprehensive report.
• Review of both AFP and Telework grants, if
applicable, are consolidated into one report summarizing
findings, recommendations, and highlighting differences.
• RSA shares report with grantee (on-site available).
• RSA finalizes and receives approval of report.
• Grantee may respond to report in writing.
• Report and response posted on ed.gov.
•
78
Concluding the Program Review: Report
Introduction
Executive Summary
Review of Program
Management
• Fiscal Management
• Contract Oversight
• Personnel
Management
Review of Program Performance
•
Emphasis and Expansion of
Consumer Choice and
Control
•
Timely and Equitable
Access to Financial
Assistance
•
Capacity to Continue on a
Permanent Basis
Data
Conclusion
Appendices
79
Outcomes of a Review
Compliant = No action necessary.
In Need of Improvement = AFP/Telework grantee
implements its own plan for improvement.
Failure to Substantially Comply =
• Prior to imposing sanctions, RSA and grantee will
negotiate corrective action plan.
• Possible referral for TA from NATTAP or NISAT.
• Failure to implement corrective action may result
in sanctions.
80
Timeline
Pre-review
Prior spring = Notify grantee
Prior to review start = Agree on
time for review, find reviewers,
grantee prepares and submits
documents, grantee prepares for
teleconferences/webinars.
Review
2 weeks – Read documents,
Program Management call with
RSA
2 weeks – Teleconf./Webinars
2 weeks – Reviewers complete
forms and meet with RSA
3 weeks – RSA develops draft
report and shares with grantee
Post-review
Finalize report and share
Grantee has two weeks to write
response
ED approval of report and
posting on ed.gov
81
Other Information
•
•
Read the Program Review Manual.
This is a work in progress.
82