This is an inter-active self-evaluation tool, created using PowerPoint.
To use it as it is intended, please start a Slide Show, either from the “View” menu,
or by pressing F5.
Program Review
Guide-Through
An interactive, step by step guide to
preparing for RSA’s program review
Begin
Rik Opstelten- NATTAP
About the Guide
The purpose of this guide is to allow program mangers to work their
way through process of preparing for RSA’s Program Review.
You will be guided through the process of:
 Understanding the basis for the program review
 Preparing program staff and procedures to participate in the
review process
 Gathering and checking documents to be reviewed by RSA
 Preparing for the Webinar-based virtual site visit component of
the review
 Understanding the questions asked by RSA and the grading
criterion
Navigating Through the Guide
In the next page, you will be shown a Table of
Contents. You can use this to skip to a
particular part of the guide, or you can use the
forward and backward buttons to move through
the process in sequence.
Other navigation buttons you will see:
Home: Returns you to the Table of Contents
Info: Brings you to further
detail on a particular point
Table of Contents





Requirement for Program Review
Role of Lead Agencies and CBOs
How are Program Selected for Review?
What Happens When a Program is Selected?
How Does a Grantee Prepare for the Review?

Program Management


Program Performance




Fiscal Management, Contract Oversight ,Personnel Management
Consumer Choice and Control, Timely and Equitable Access to Financial
Assistance, Capacity to Continue on a Permanent Basis
The Virtual On-Site Visit. Preparing the Webinar
Appendix 1- RSA’s rubric on Program Management
Appendix 2- RSA’s rubric on Program Performance
ED Monitoring Policy
•
•
It is ED’s policy to monitor active
discretionary grants with a focus on technical
assistance, continuous improvement, and
attaining promised results.
Monitoring a grantee shall continue for as
long as ED retains a residual financial
interest in the project, whether or not ED is
providing active grant support.
Requirements for Program Review
Section 5.3.2 Monitoring Policy
ED staff is to monitor each grantee to the
extent appropriate so as to achieve expected
results under approved performance measures,
while assuring compliance with grant
requirements. Existing requirements in
EDGAR extend equally to all grantees and their
partners.
Requirements for Program Review
Section 5.3.3 Monitoring Purpose
Systematic and regular monitoring
by ED staff of a grantee’s activities
measures the project quality and
progress, including strengths and
weaknesses.
Requirements for Program Review
1. A grantee’s project must:
a) Conform to the grantee’s approved application and any
approved revisions, as well as the effectiveness and quality of
the project (Program Management);
b) Progress against previously established performance
measures (Performance Measurement);
c) Adhere to laws, regulations, conditions of the grant,
certifications and assurances (Compliance); and
d) Manage Federal funds according to Federal cash
management requirements, including expenditures of funds for
authorized purposes (Fiscal Accountability).
Requirements for Program Review
•
•
•
•
Regular monitoring enables ED staff to
provide customized technical
assistance, appropriate feedback, and
follow-up to help grantees to:
Improve areas of need
Identify project strengths
Recognize significant achievements
Role of Lead Agencies and
Community Based Organizations in
Program Reviews
•
•
•
•
The LEAD AGENCY is being reviewed.
Participation from BOTH the Lead Agency and CBO
is expected.
Participation of all personnel being paid using grant
funds is required (see exception in the next bullet).
Participation of the Certifying Representative is
encouraged but not required.
How are AFP/Telework
Programs selected for review?
Random selection of an equal number of grantees in
two categories:
(1) Title III AFP/Telework grantees claimed by a
Statewide AT Program as a state financing activity;
OR
(2) Title III AFP/Telework grantees not claimed by a
Statewide AT Program as a state financing activity.
How are AFP/Telework
Programs selected for review?
Exceptions to random selection:
• Recent major change in the
AFP/Telework Program or State
Plan for AT
• Recent major disaster
• RSA is concerned
How are AFP/Telework
Programs selected for review?
6 grantees per cycle (3 x 3)
• A “cycle” is a fiscal year (Oct. 1 to Sept. 30)
• 9 weeks per grantee (some program reviews
may overlap)
• First year of program reviews:
•
•
•
Jan. 1 to Sept. 30
Only 3-4 reviews
What happens when an
AFP/Telework Program is
selected?
•RSA
notifies grantee in spring prior to next cycle.
• RSA and grantee negotiate 9-week window within
next cycle.
• RSA recruits third-party (peer) reviewers.
• Grantee begins preparing for review.
How does a grantee prepare
for the review?
A review assesses two “Core
Components” of grant implementation,
each in a slightly different way.
• Core Component 1 – Program
Management
• Core Component 2 – Program
Performance
Core Component Selection
Now that you have learned about the reasoning
behind the Program Review, and how the
process is structured, we will begin going
through each of the “Core Components” of
the Program Review.
Each Core Component has several sub-points
that comprises it.
Choose a Core Component

Program Management




Fiscal Management
Contract Oversight
Personnel Management
Program Performance



Emphasis and Expansion of Consumer Choice
and Control
Timely and Equitable Access to Financial
Assistance
Capacity to Continue on a Permanent Basis
Program Management



Fiscal Management
Contract Oversight
Personnel Management
Fiscal Management

Documents Required




Expenditure Report
Proof of Title III or NFP fiscal management
assurances
Indirect Cost Agreement
Copy of most recent audit report or Questionnaire
RSA Detail:
Fiscal Management
EDGAR requirements:
• Fiscal control and accounting procedures to insure proper
disbursement of and accounting for Federal funds; controls and
procedures adequate to establish that such funds have not been
used in violation of applicable statutes.
• Records that fully show the amount of funds under the grant, how
the grantee uses the funds, the total cost of the project, the share
of the cost provided by other sources, and other records to
facilitate an effective audit.
• Audits in accordance with OMB A-133.
• Lead Agencies must be able to show how they ensure
appropriateness of indirect costs used by Implementing Agencies
or CBOs.
RSA Detail:
Fiscal Management
Title III or NFP requirements:
• Assurance that the grantee will provide the nonfederal share of the cost of the program in cash, from
state, local, or private sources.
• Assurance that the grant funds will be used to
supplement and not supplant other federal, state, and
local public funds.
• Assurance that the grant funds will be placed in a
permanent separate account and identified and
accounted for separately from any other fund.
• Assurance that the funds are administered, expended,
and invested in accordance with title III or the NFP.
Help:
Choosing a Core Component


To proceed through the guide, you must
choose the Core Component of the Program
Review you would like to learn about.
Remember, you can always use the Table of
Contents page to navigate in greater detail
Contract Oversight

Documents Required

Dates copy of the Grantee’s most recent contract
with the CBO.
Contract Oversight Detail
EDGAR requires grantees to:
• Directly administer or supervise the administration of
each project.
• Have procedures for providing TA, evaluation and
for performing other administrative responsibilities
necessary to ensure compliance.
• Monitor activities to assure compliance with
applicable Federal requirements and that
performance goals are being achieved.
Contract Oversight Detail

•
•
•
Grantees are required to enter into a contract with a CBO or
group of CBOs that has individuals with disabilities involved in
the organizational decision-making to administer the
AFP/Telework Program. The contract must:
Include a provision requiring funds be administered consistent
with title III or the NFP.
Include any provision the Secretary requires concerning
oversight and evaluation of federal financial interests.
Require the CBO(s) enter into a contract with commercial lending
institutions or organizations or state financing agencies.
Contract Oversight Detail
What RSA is looking for:
• Appropriate oversight to protect Federal
interests and ensure implementation
• Relevant to title III or the NFP
Personnel Management
Documents Required:
(A)
(B)
Position descriptions of record for those being
paid with grant funds and personnel activity
reports for them; AND
Synopsis for key personnel describing his or her
role and responsibilities associated with the
grant.
Personnel Management Detail
What RSA is looking for:
• Appropriate number of personnel
• Appropriate time commitments of
personnel
• Appropriate duties for personnel
Personnel Management Detail
OMB Circulars:
•
•
•
•
Compensation to personnel must be reasonable.
Charges to Federal awards for salaries and wages are based on
documented payrolls.
Charges for salaries and wages of employees who work solely
on a single Federal award are supported by periodic
certifications.
Charges for salaries and wages of employees who work on
multiple activities are supported by personnel activity reports.
Program Performance



Consumer Choice and Control
Timely and Equitable Access to Financial
Assistance
Capacity to Continue on a Permanent Basis
Consumer Choice and Control

Documents Required:
•
A narrative describing how individuals with
disabilities are involved in organizational decisionmaking at all organizational levels to administer
the loan program.
Consumer Choice and Control
Detail
Title III or NFP requirements:
Assurance that, and information describing
the manner in which, the grantee will expand
and emphasize consumer choice and control.
•
Consumer Choice and Control
Detail
What RSA is looking for:
• Appropriate emphasis and expansion of
involvement in organizational decision-making
by individuals with disabilities
• The grantee should also be prepared to
discuss consumer choice and control during
webinars/teleconferences
Timely and Equitable Access
to Financial Assistance
Documents Required
•
•
A dated copy of its most recent revised and
adopted AFP/Telework policies and
procedures manual.
A dated copy of its most recent revised and
adopted lender agreement(s).
Timely and Equitable Access Detail
Title III or NFP requirements:
Assurance that the grantee enter into a contract with a
CBO that has policies and procedures to:
•
Review and process requests for financial assistance
in a timely manner.
•
Ensure equitable access to all consumers for which
financing is requested through the program.
•
Ensure consumer-controlled oversight of the program.
Timely and Equitable Access Detail
What RSA is looking for:
• Appropriate access to financial assistance.
• The grantee should also be prepared to
discuss how the program assures the review
and process of financial loans in a timely
manner and describe the flow of an application
for an AFP/Telework loan during
teleconferences or webinars.
Capacity to Continue on a
Permanent Basis
Documents Required
•
•
•
A dated copy of investment plan, if
applicable, or narrative describing how loan
program funds are invested.
A copy of investment portfolio.
A dated copy of the plan for sustainability, if
one has been developed, or narrative
explaining how the program will continue on a
permanent basis.
Capacity to Continue Detail
Title III or NFP requirements:
•
•
•
•
Assurance that the AFP/Telework Program
will:
Continue on a permanent basis.
Invest the funds within the permanent
separate account in low-risk securities.
Administer the funds with prudence,
discretion, and intelligence.
Capacity to Continue DetailInvestment Plan
What RSA is looking for:
Appropriate investment of AFP/Telework
Program funds in the permanent separate
account.
• The grantee should also be prepared to
discuss the implementation and regular
revision of the AFP/Telework Program’s
investment plan.
•
Capacity to Continue DetailSustainability Plan
What RSA is looking for:
• Appropriate implementation of strategies for
sustaining the AFP/Telework Program.
• The grantee should also be prepared to
discuss how the program is implementing a
plan for sustainability during a
webinar/teleconference.
Virtual On-Site Visit/Webinar
Develop teleconference/webinar
presentations that serve as a virtual on-site visit
to your program.
• Length, format, and content of presentation
are up to the program.
• The “presenter” during a teleconference or
webinar is up to the program.
• Need to be accessible.
•
RSA’s rubric on Program
Management
The following slides provide detailed
information about the way in which RSA
evaluates program management. Outlined
are:




How to prepare for evaluation
Questions used for evaluation
The rating scale of evaluation
How ratings are assigned
How to prepare for a review of
Program Management
•
•
•
•
Submit all the previously described
documents.
RSA reviews the documents.
RSA discusses the documents via
conference call (the grantee is responsible for
bringing together the appropriate parties).
RSA uses this information to answer
questions about program management.
•
Yes/no with explanation.
Questions about Program Management
1. Has the grantee obtained its full matching funds
from appropriate sources? If not, does the grantee
have a plan for obtaining the remaining match funds?
2. Does the grantee account for grant funds
accurately and completely in accordance with
EDGAR?
3. Does the grantee use grant funds only for purposes
under, and within the limitations of, title III or the NFP,
as applicable?
4. Does the grantee comply with indirect cost
requirements?
5. Does the grantee comply with audit requirements?
Questions about Program Management
6. Does the grantee exercise appropriate fiscal and
performance oversight of the AFP/Telework Program
through its contract with the CBO? If not, does the
Lead Agency have a plan for improving its oversight?
7. Is the contract structured to meet the requirements
and intent of title III of the NFP, as applicable? If note
does the Lead Agency have a plan for changing its
contract to make it more consistent with title III or the
NFP, as applicable?
Questions about Program Management
8. Does the AFP/Telework Program have an appropriate number of
staff to implement the grant? If not, does the Lead Agency have a
plan for changing its contract to make it more consistent with title III
or the NFP, as applicable?
9. Do the Lead Agency and CBO distribute the time of their
personnel appropriately to implement the grant based on their
responsibilities? If not, does the grantee have a plan to distribute
time more appropriately?
10. Do the Lead Agency and CBO use grant funds to support only
personnel with responsibilities germane to the grant?
Assessing Compliance of
Program Management
There is no overall rating of compliance.
For each element of Program Management, a
grantee will receive 1 of 3 ratings:
•
•
•
Compliant
In Need of Improvement
Fails to Substantially Comply
Assessing Compliance of
Program Management
The rating is based on how RSA answers the
questions:
Initial question: Does the AFP/Telework
Program have an appropriate number of staff to
implement the grant?
• Subquestion: If not, does the grantee have a
plan for adjusting the number of staff?
•
Assessing Compliance of
Program Management
•
•
•
•
If the answer to the initial question is “Yes” – the grantee is rated
as compliant.
If the answer to the initial question is “No” – the rating depends
on the answer to the subquestion.
If the answer to the subquestion is “Yes” – the grantee is rated as
in need of improvement.
If the answer to the subquestion is “No” – the grantee is rated as
failing to substantially comply.
Assessing Compliance of
Program Management
Questions for which “needs improvement” doesn’t apply:
•
•
•
•
•
Does the grantee account for grant funds accurately and completely in
accordance with EDGAR?
Does the grantee use grant funds only for purposes under, and within the
limitations of, title III or the NFP, as applicable?
Does the grantee comply with indirect cost requirements?
Does the grantee comply with audit requirements?
Does the grantee use grant funds to support only personnel with
responsibilities germane to the grant?
Assessing Compliance of
Program Management
The onus is on the grantee to present
evidence/plan sufficient to get to a “yes”
answer.
RSA will ask questions or provide examples of
sufficient evidence, if needed.
• Grantee likely will know their status before the
conversation is over.
•
RSA’s Rubric on Program
Performance
The following slides provide detailed information about
the way in which RSA evaluates program
performance. Outlined are:
 How to prepare
 Questions asked
 Grading Scale
 Grading Rubric
 Activities of Program Reviewers
 The Role of Data
How to prepare for a review of
Program Performance
Based on documents, data, and discussion, third-party
reviewers respond to a series of questions:
• Yes/no with explanation.
• Evidence can come from any source of the review.
• Getting a “yes” answer does not depend on an
AFP/Telework Program having any particular structure
or type.
• A “no” answer does not automatically mean a
grantee is out of compliance.
Questions about Program Performance
1. Is there evidence that individuals with disabilities are involved in
the administrative decision-making of the AFP/Telework Program?
a) If not, does the grantee provide justification for the current level
of involvement of individuals with disabilities in the administrative
decision-making of the loan program?
b) If not, does the grantee have a reasonable plan for improving the
involvement of individuals with disabilities in the administrative
decision-making of the loan program?
Questions about Program Performance
2. Is there evidence that the AFP/Telework Program
emphasizes and expands consumer choice and control?
a) If not, does the grantee provide justification for why the
current level of consumer choice and control is
appropriate?
b) If not, does the grantee have a reasonable plan for
improving its emphasis and expansion of consumer choice
and control?
Questions about Program Performance
3. Is there evidence that the policies and procedures of the
AFP/Telework Program make it accessible to consumers regardless
of type of disability, age, income level, type of AT device, equipment,
or service for which financing is requested through the program?
a) If not, does the grantee provide justification for why the program
is not accessible to all individuals with disabilities in the State?
b) If not, does the grantee have a reasonable plan for improving the
accessibility of the program?
Questions about Program Performance
4. Is there evidence that the structure and practices of the
AFP/Telework Program make it possible to meet the needs
of individuals in most areas of the State?
a) If not, does the grantee provide justification for why the
areas reached in the State are sufficient?
b) If not, does the grantee have a reasonable plan for
improving their ability to reach consumers in other areas of
the State?
Questions about Program Performance
5. Is the lender agreement structured to promote the
goals of the AFP/Telework Program?
a) If not, does the grantee provide justification for the
structure of the current lender agreement?
b) If not, does the grantee have a reasonable plan for
improving the lender agreement?
Questions about Program Performance
6. Does the AFP/Telework Program administer its
funds with prudence, discretion, and intelligence?
a) If not, does the grantee provide justification for why
the program administers its funds in the current
manner?
b) If not, does the grantee have a reasonable plan for
improving the administration of program funds?
Questions about Program Performance
7. Does the AFP/Telework Program have an investment
plan that is implemented and revised regularly?
a) If not, does the grantee provide justification for why the
program has not implemented an investment plan that is
reviewed regularly?
b) If not, does the grantee have a reasonable plan for the
implementation and regular revision of an investment plan?
Questions about Program Performance
8. Is the AFP/Telework Program implementing
appropriate strategies for sustainability?
a) If not, does the grantee provide justification for why
the program is not implementing appropriate strategies
for sustainability?
b) If not, does the grantee have a plan for
sustainability that could be implemented?
Assessing Compliance of
Program Performance
There is no overall rating of compliance.
For each element of Program Performance, a
grantee will receive 1 of 3 ratings:
•
•
•
Compliant
In Need of Improvement
Fails to Substantially Comply
Assessing Compliance of
Program Performance
The rating is based on how peer reviewers answer the
questions:
• Initial question: Is there evidence that the structure and
practices of the AFP/Telework Program makes it possible to
meet the needs of individuals with disabilities in most areas
of the State?
• Subquestion 1: If not, does the grantee provide
justification for why the areas reached by the program in
the State currently are sufficient?
• Subquestion 2: If not, does the grantee have a
reasonable plan for improving the capacity of the program
to reach individuals in more areas of the State?
Assessing Compliance of
Program Performance
•
•
•
•
•
•
If the answer to the initial question is “Yes” – the
grantee is rated as compliant.
If the answer to the initial question is “No” – the
rating depends on the answer to the subquestions.
If the answer to subquestion 1 is “Yes” – the grantee
is rated as compliant.
If the answer to subquestion 1 is “No” – the the
rating depends on the answer to subquestion 2.
If the answer to subquestion 2 is “Yes” – the grantee
is rated as in need of improvement.
If the answer to subquestion 2 is “No” – the grantee
is rated as failing to substantially comply.
Assessing Compliance of
Program Performance
Activities of peer reviewers:
• Read documents.
• Participate in teleconferences/webinars.
• Complete review forms (answer questions).
• Meet as a group with RSA to discuss results.
• Finalize review forms and submit to RSA.
Program Data
AFP/Telework Program data will not
determine compliance.
• Data will be discussed during a
teleconference or webinar and reflected in
the report:
(1) Grantees strengths and areas in need of
improvement when it comes to data; and
(2) How the grantee is learning from and
using its data.
•
Program Data
Data collection infrastructure:
How the data is collected.
• How the data is checked.
• Strengths and areas in need of
improvement.
•
Program Data
Understanding your data:
•
•
•
How the nature of your state and structure of
your program influence the data.
What your data tells you about the program
and how it influences implementation.
Strengths and areas in need of improvement.
Conclusion
We hope this guide has been useful in
understanding the steps that programs can
take to prepare for the RSA’s Program
Review Process.
If you have any further questions, please feel
free to contact NATTAP.