Moving Toward EvidenceBased Practice:
Innovative Ideas for the CSBG Network
NASCSP
September 2012
Barbara Mooney, Ed. D.
National ROMA Training and
Certification Project
Karen Walker, Ph.D.
Performance? Evidence?
We know we have a groundwork in the CSBG
program for Performance Measurement and
Performance Management, but how do we build
on this foundation to move forward?
We need strategies for moving the CSBG
Network toward greater adoption of evidencebased practices.
Workshop Objectives

Definitions and examples of





Evidence-based, evidence-informed
Participants will understand the regional and national
factors pushing toward new practice.
Participants will consider how the core activities in IM
49 support universal implementation of high quality
practices
Participants will learn about examples of using
evidence to evaluate and improve programs
Participants will brainstorm “next steps” for action to
move us toward more evidence based practices
What Does it Mean to be
“Evidence-based”?
What Do People Mean by Evidencebased?

Multiple terms are in use:



Evidence-based practice
Evidence-based program
Evidence-informed program
…How do we make sense of these terms?
What do we mean by Evidencebased programs and practices?


Evidence-based programs:
 Interventions that have been found to have
positive effects or impacts on targeted
outcomes (Nurse Family Partnership, Parent
Management Training)
Evidence-based practices:
 Intervention strategies or program practices
that have been found to improve outcomes
for children and youth (role-modeling,
positive reinforcement, avoiding grouping
children with problem behaviors together)
Evidence-informed Program


Intervention that has been designed with
research evidence in mind. Some of the
program’s strategies may be evidencebased practices, they may be informed
by the basic research, or both.
Evidence-informed program may also
refer to a program that uses
performance information to monitor and
guide its progress.
Time Out for a brief description of what we
mean by Randomized-Controlled Impact and
Quasi-Experimental Evaluations

Randomized-Controlled Impact
Evaluation



Provides most certainty about program
effectiveness, but must be welldesigned and well-implemented
Study participants randomly assigned
to control or intervention group (think
lottery).
Compares the difference in outcomes
between the intervention and control
group.
Quasi-Experimental Study



Like randomized-controlled impact
study (RCT) a quasi-experimental
study includes a comparison
between people who received the
intervention and those who did not.
Unlike an RCT, the comparison is
not based on random assignment.
As a result, researchers are less
certain that the comparison is a
good one.
What Can You Do to Become EvidenceBased or Evidence-Informed?
Adopt evidence-based programs and
monitor their implementation to ensure
that they are being replicated correctly
 Assess “home-grown” programs to
determine whether or not they include
evidence-based or evidence-informed
practices
 Set the program on a path toward rigorous
evaluation
…These strategies are not mutually exclusive

Collect and Use Data on Performance & Outcome
Measures
Targeting
Conduct
Needs
Assessment
Identify
Population
Select Intervention,
Develop Logic
Model & Identify
Indicators
Implement Program
Conduct Ongoing
Performance Management
Conduct an Implementation
Evaluation
Conduct a Quasi-Experimental
Outcomes Evaluation
Figure adapted from: Performance Management and
Evaluation: What’s the Difference by Karen E Walker and
Kristin Anderson Moore. Child Trends
Conduct a RandomizedControlled Impact Evaluation
Federal, State and Local Efforts
and Policies


Widespread move toward evidence-based
programs and practice and performance
management
 Federal funding tied to evidence
 State initiatives and funds for
intermediaries
 Local efforts include both public and
non-profit efforts
Administration’s guidance from OMB to
agencies about encouraging evidencebased efforts.
Definitions from OCS:
Evidence-Based Practice –
Approaches to prevention or treatments that are validated by
some form of documented scientific evidence. These could be
findings established through scientific research, such as
controlled clinical studies or other comparable and rigorous
methods.
Evidence-Informed Practice –
Approaches that use the best available research and practice
knowledge to guide program design and implementation within
context. This informed practice allows for innovation and
incorporates the lessons learned from the existing research
literature.
Use Data to Show Results
and Improve Performance
• “Success should be judged by results, and
data is a powerful tool to determine results.
We can’t ignore facts. We can’t ignore data.”
President Barack Obama
July 24, 2009
• “The test of a performance management
system is whether it’s actually used….Federal
managers and employees at all levels must
use performance goals and measures to set
priorities, monitor progress, and diagnose
problems.”
Chief Performance Officer Jeffrey Zients
October 29, 2009
14
New Guidance:
GPRA Modernization Act 2011


Shift emphasis from production of annual
reports to collecting and using performance
information more frequently throughout the
year to set priorities and make decisions that
improve results
Establish common Federal Government
performance indicators with Quarterly Targets
and Quarterly Milestones
The plan to target CSBG
resources
“Building on the President's budget request to target CSBG
resources to high-performing, innovative agencies, ACF plans
to work with Congress and a wide variety of organizational
stakeholders to establish a set of core Federal standards that
States will be required to use to assess whether an eligible
entity is meeting a high standard of service delivery.
These standards of service delivery will include a focus on
compliance with basic requirements of the CSBG Act, and will
also go beyond basic compliance to focus on high-quality
performance across the CSBG Network.
Core performance standards will focus on organizational
management issues, as well as strategic process for
performance management.”
(OCS T/TA overview, 2012)
The CSBG National T/TA
Strategy is designed to:







Measure and document performance in accordance with CSBG reporting
requirements.
Assure performance measurement and use of performance data at all
levels within the CSBG program (local, State, and Federal).
Mitigate risks associated with the use of CSBG grant funds.
Promote exemplary practices and innovative programming that
stimulates the creation and sharing of information and knowledge in the
CSBG T/TA Network.
Measure and document performance in accordance with CSBG reporting
requirements.
Work with States to help ensure CSBG-eligible entity boards know,
understand, and meet their fiduciary responsibilities to CSBG (i.e.,
participation in the development, planning, implementation, and
evaluation of CSBG-funded activities and services).
Ensure the management and delivery of CSBG-funded services creates
and strengthens sustainable economic resources in communities
ROMA Next Generation and
Organizational Standards

Two targeted T/TA Centers of
Excellence embody OCS focus for
coming years
Organizational Standards
The CSBG T/TA Organizational Standards Center of Excellence
(COE) will coordinate the development and dissemination of a
core set of standards with input from the State CSBG Lead
Agencies as appropriate, regional and national partners,
including the CSBG T/TA Regional Performance and
Innovation Consortia (RPIC).
The central mission of this effort will be to ensure that all CSBGeligible entities have the capacity to provide high-quality
services to low-income individuals and communities, and are
able to meet high-quality organizational performance
standards.
The movement toward continued accountability, risk mitigation, and
performance plays a key role in developing organizational standards
toward improved performance management and the capacity to thrive
within the CSBG Network.
ROMA Next Generation
Progress has been made in recent years in the quality and
breadth of State and local performance reporting systems
through ROMA and the NPI framework… (which) provides a
foundation for additional targeting of resources based on
outcomes and effectiveness of services.
However, practices of data collection and reporting within the
ROMA system vary from State to State; while some States‘
reporting systems reflect the full ROMA cycle, other States
only report NPI data.
Such inconsistencies within the ROMA system minimize the
capacity of ROMA to be an integrated and comprehensive
effort that lends to accurate and reliable performance
management and accountability within the CSBG model.
Key goals of the ROMA Next
Generation COE will be:




To review and strengthen the current performance
measures and performance indicators with input from
experienced CSBG organizations, as well as performance
management experts from outside the CSBG Network;
To work with State CSBG agencies to communicate
consistent standards for performance management
throughout the ROMA cycle, including community needs
assessment, program planning, and performance
measurement;
To assure a coordinated, national strategy for nationwide
implementation of an updated performance
management system;
To assure that a performance management system can be
used, as necessary, for comparison of the effectiveness of
local entity use of CSBG resources.
We are keeping the basics:






Six National Goals
ROMA Cycle (core activities for
eligible entities)
Core activities for state offices
Checklist for Monitoring (IM102)
Risk Mitigation (IM 112)
Key elements of the Logic Model
Six National Goals
1. Low-income people become more self-sufficient.
2. The conditions in which low-income people live are
improved.
3. Low-income people own a stake in their community.
4. Partnerships among supporters and providers of
services to low-income people are achieved.
5. Agencies increase their capacity to achieve results.
6. Low-income people, especially vulnerable populations,
achieve their potential by strengthening family and other
supportive systems.
24
IM 49
Core Activities for State Offices
1. The state has developed, in coordination with eligible entities and
the State CAA association, a State-wide vision statement that
speaks to the goals and purposes of community action within the
State and that supports the 6 national ROMA goals.
The state is encouraged to participate in, and contribute to,
broader State anti-poverty/community development initiatives
with outcome measures and goals compatible with ROMA;
2. The agency has trained all its eligible entities (staff and boards) in
outcome-based management, and that 80% of the entities use
ROMA concepts to guide needs assessment, agency mission
review, activity planning, resource allocations, service delivery,
measuring and reporting results;
25
Core Activities for State Offices (cont)
3. 80%of the plans and program reports received from eligible
entities in the State describe plans to achieve projected
outcomes, and evaluate results based on measurable
improvements of condition(s) among clients and/or communities
served;
4. The agency submits complete, accurate, and timely annual
reports to OCS on the measured performance of the State and
the eligible entities in the State as required by Section 678E of
Public Law 105-285, the Community Services Block Grant
Reauthorization Act of 1998.
26
ROMA Logic Model
Identified Problem,
Need, Situation
Service or Activity
(Output)
Identify the timeframe.
(1) Planning
Identify the # of clients
served or the # of units
offered.
(2) Intervention
Outcome
Outcome Indicator
Actual Results
(General statement
of results expected)
Projected # and % of
clients who will
achieve each outcome.
Actual # and % of
clients who achieve
each outcome.
(3) Benefit
Short Term
(4) Benefit
Short Term
(5) Benefit
Short Term
Intermediate Term
Intermediate Term
Intermediate Term
Long Term
Long Term
Long Term
Measurement Tool
Data Source,
Collection Procedure,
Personnel
Frequency of Data
Collection and
Reporting
(6) Accountability
(7) Accountability
(8) Accountability
Community
Federal, State and Local
Policies
Population
Culture (norms and behaviors)
Funding
Organization
Mission & Goals
Leadership
Resources
Policies
Program
Program Monitoring
Activities
Staff assessments
Staff roles and skills
Training
Quality and Fidelity
Participation
Outcomes
28
Example
Performance Management In Action


After-school tutoring program assisting at-risk
youth – attempting to improve their math and
reading performance.
Collect the following data:
 Demographics and risk/protective factors
 Program attendance
 Amount of service (hours and topics of
tutoring)
 Report card grades (first / last grading
period)
29
Example (cont.)
Performance Management in Action


Finding #1: Halfway through program,
program attendance is low among youth from
Neighborhood A.
 Action: Reach out to teachers in schools in
Neighborhood A
 Action: Talk with youth in Neighborhood A
Finding #2: Math grades increased, but
reading grades did not change.
 Action: Investigate.
 Finding: Tutors unfamiliar with reading
curriculum
 Action: Train tutors in reading curriculum
30
Consider Resources for Performance
Management

Executive leadership


Staff expertise


To collect, enter, analyze, share, and discuss data
Technology

31
To identify measures, to collect data, and to analyze
data
Staff time


To set the tone, to implement policies
Forms (paper or electronic), spreadsheet software,
management information systems/commercial
software
Create Policies for Performance
Management

Performance management policy
outlining:




Staff roles and responsibilities related to
performance management
Description of data collection and reporting
methods
How often regular meetings should take place
to review and discuss findings
Performance appraisal process which
evaluates staff on how well they carry out
performance management roles and
responsibilities
32
Example Questions in Performance
Management






Are we reaching our intended population with
the intended services?
Are participants getting the amount of program
time we want them to get?
Are participants engaged in the process?
Are participants’ outcomes changed?
Do our staff carry out activities as intended?
Are we providing the training our staff need?
If the answers are no, organizations need to
determine why and what needs to change
Video Clip from
“Saving Philanthropy”

An example about how sharing performance
management data on outcomes can increase staff
buy-in, inspire youth, and create a learning culture
http://www.savingphilanthropy.org/previewfootage/
34
Implementation will take resources


Time and attention to develop a
plan of action
Staff time and support (travel,
meeting costs, provision of
materials, etc.)


At the state level, for local EEs
Follow up to evaluate actions and
assess new needs/resources
T/TA from National ROMA Trainers


The demand for training and technical assistance
exceeded the available resources in the ROMA Trainer
network to address the needs.
Two specific areas were identified:

The certified trainers do not have any resources to bring local CAAs
together as they begin to implement the full range of ROMA activities.
Implementation support is critical to working out solutions to local challenges and establishing a
consistent use of terms, concepts and processes.


NCRTs do not have the time or the expertise to provide individual onsite technical assistance to local CAAs as they apply the principles to
agency systems.
We are working to increase the number of available
NCRTs in partnership with the RPICs and State Offices
State CSBG Discretionary Dollars

NASCSP Resources on discretionary
funds:


Annual Reports and appendix tables
Issue Brief: State Discretionary
Spending
Distribution of FY 2011 CSBG
Discretionary Funds by Purpose
CSBG National Resource Center &
Clearinghouse




Resources online
Evidence-based discussions
Forums
Lists of consultants
Child Trends’ Resources




Using Data in Multi-Agency Collaborations to Ensure Accountability
and Improve Performance.
http://www.childtrends.org/Files//Child_Trends2012_02_23_FR_UsingData.pdf
Performance Management and Evaluation: What’s the Difference?
http://www.childtrends.org/Files//Child_Trends2011_01_19_RB_PerformMgmt.pdf
at www.childtrends.org
Leap of Reason: Managing to Outcomes in an Era of Scarcity,
available at leapofreason.org
What’s next?



How can you identify and focus
resources so our network can move
toward evidence based practices?
What specific steps do you need to
take to begin to identify what works
and how to collect the evidence?
What do you need to consider?
Steps toward improving practice
IMPROVING REPLICATION AND
“SCALE-UP”
The history of replication and scale-up efforts
indicates the difficulty associated with these
efforts.


As President Bill Clinton frequently declared, “You can find
virtually every problem in our country solved by somebody
somewhere in an astonishingly effective fashion….The
challenge for us is to figure out how to replicate that.”
The day before his first inauguration, Clinton told the nation’s
governors that his number one disappointment as governor
had been that it was so hard to “take something that works
to the next level.” He said he “could never figure out a way to
make the exception the rule, and that is our enduring
problem in America in public life.”
For more discussion:
Barbara Mooney
Karen Walker
Project Director
Senior Research Scientist and Senior Program
Area Co-Director
National ROMA Training and
Certification Project
243 E. High St.
Waynesburg, PA
724-852-2272
barbaramooney@
windstream.net
Child Trends
4301 Connecticut Ave NW
Suite 350
Washington, DC 20008
202-572-6005
kwalker@childtrends.org