UNDERSTANDING EVALUATION FROM
MULTIPLE PERSPECTIVES: MAXIMIZING
VR’S PARTNERSHIPS UNDER WIOA
Summit Reading Group Facilitator
Darlene A. G. Groomes
Fundamental Group
Barbara Burkett, Karen Carroll, HarrietAnn Litwin,
Jeff Stevens, Paige Tidwell
Principal Group
Allison Flanagan, Mark Kinnison
8th Summit Conference
San Diego, California
September 2015
SuRGE-6: SUMMIT READING
GROUPS FOR EXCELLENCE
•
Represented seven states across the nation
•
Read Evaluation Roots: A Wider Perspective of
Theorists’ Views and Influences by Marvin C. Alkin
(2013)
•
Session PowerPoint and Presentation Takeaway
Element will be available at: http://vocationalrehab.com/summit-reading-groups/projectdevelopment/
EVALUATION ROOTS: A WIDER PERSPECTIVE OF
THEORISTS’ VIEWS AND INFLUENCES ALKIN (2013)
 Focused on three chapters with which SuRGE-6 members
connected best:
 Enlightenment
 Sensemaking
 Getting People Involved
 Challenged by theoretical orientation; how to make
information application-oriented?
 Connection to WIOA and its application in the changing
environment
 Present: Regulations out January 2016
 Future: As programs roll-out; performance reports and
evaluations
ADMINISTRATORS’ CONTEXT
 Utilizing partnerships to enhance performance and
continuous improvement
o Unified or Combined State Plan
o Common performance measures
o Dual-customer approach: Individual with a disability and
employers
o Accessibility
o Services to groups
o Funding
 Leveraging resources through program evaluation
o Existing data within core partners
o Streamlining
ENLIGHTENMENT
“The aim is to provide sound information that people in authority can use to make things better for
the often disadvantaged people the serve.”
Carol Hirschon Weiss
 Evaluators strive to work for the improvement of a
program
 The three distinctive attributes of evaluation:
 It’s task- What type of interventions are being used, and
how well do they function?
 It’s situation- Is it responsive to the desires and
needs of program constituents?
 It’s mission- Is it helping society?
ENLIGHTENMENT
 The Evaluator’s role as an expert on method
 Method is the main touchstone
 The method’s credibility offers the promise of unbiased and
accurate information
 Policy makers and administrators want to learn about which
programs and procedures stand the best chance of improving the
lot of their people
HOW TO ADDRESS UNDER WIOA
• Emphasis on partnerships in the world of WIOA
o How to align evaluation as a professional practice
o How does VR maintain its mission while partnering with others?
• Performance measures
o The role of politics, and acknowledging the limits of evaluation
o Staying relevant during all phases of evaluation
o What will be measured? Closer look at rehab rates while at the
same time emphasizing other measures such as activity in PETS.
PERCEPTION….
...is not always REALITY
SENSEMAKING
“Deliberate decision-making and action, informed by thoughtfully planned evaluation.”
Melvin M. Mark and Gary T. Henry
 Social betterment as the guiding star
• Values & well-being of others: Planning identifies what is most
valued by the stakeholders & infuses into the evaluation;
• Policy analytic: Planning involves awareness of the big & little picture,
impact of evaluation, anticipated intended & unintended consequences;
• Decision-making environment: Choosing method that has greatest
impact on social betterment, decision making, and action.
SENSEMAKING
 Alternative pathways of influence
• Draw out the expected impact made by the evaluation
• Sequence of influence, along with potential risks of interruption within the
sequence
• Identifies the purpose, condition of evaluation, and way to obtain
meaningful information versus simply meeting requirements,
 Share findings in a way that connects with the natural
“sense-making” processes of relevant stakeholders
• Know the audience and intended users of the information
• Get the most accurate answer to the most pressing question
HOW TO ADDRESS UNDER WIOA
 Emphasis on partnerships in the world of WIOA
As we work to put Americans with significant barriers to employment,
back to work, evaluation planning identifies the multiple methods of
valuing the partnerships required under the law.
 Questions could ask:
 What is the level of awareness among local staff of the various
workforce programs that serve out of school youth? Why is this
important?
 How would knowing level of awareness impact local or state
agreements?
 How would information that has the greatest impact on putting youth
to work be best captured ?
HOW TO ADDRESS UNDER WIOA
 Performance measures
Performance measure reporting, in itself, is not program evaluation. Shear numbers
do not tell the story.
That data, however, can be used to pose evaluation questions. Sense-making can
help to prioritize:
 Questions if more than one surfaces
 The multiple methods of evaluation
 The consequences of the information learned, and
 How the information should be disseminated.
Example:
 “Increasing self-sufficiency and prosperity of workers” (one of WIOA stated
purposes and a public policy issue)
 As the guiding star in evaluation planning, this purpose could potentially
assist in the development of a focused customer satisfaction survey for
those who sought services for career advancement
HOW TO ADDRESS UNDER WIOA
 Closer look at rehabilitation rate (Specific States Need to Show
Percentage Increases by 2018)
Difference between using evaluation as means to satisfy a reporting
requirement and as a means to provide important information, which
informs policy and decision-making.
By using social betterment as a guiding star, evaluation activities can
be narrow or broad, but all feed the over-arching need of this one
question.
Focus on WHY this information is important and how it contributes to
the well-being of others, allows for the program evaluation activities to
be built strategically and thoughtfully.
GETTING PEOPLE INVOLVED
“It was the users, not the report, who played a critical role in the evaluation process.”
Jean A. King
Employ “Utilization Focused Evaluation”
 Four key concepts
 The evaluator needs to facilitate usage of the evaluations in the agency
 Building evaluation capacity within the organization is important
 Interpersonal skills are crucial (communication, relationships, etc.)
 Evaluators need to teach participants (management and counselors)
GETTING PEOPLE INVOLVED
Michael Quinn Patton
 Process use (from Chapter 23)
 Draws attention to individual changes in thinking and behavior among those
involved in the evaluation as a result of learning that occurs
 Helps those we work with learning to think evaluatively





place value on specificity and clarity
being systematic
making assumptions explicit
distinguishing inputs and processes from outcomes
separating statements of fact from interpretation and judgments
 HELP PEOPLE THINK IN THESE WAYS; Learn how to learn, not just rely on
a delimited set of findings
HOW TO ADDRESS UNDER WIOA
WIOA and good agency management require that evaluation findings be
utilized for improvement of outcomes for both staff and clients

Evaluate and ACT on trends in client outcomes to identify regions/counselors
that may need assistance

Evaluate and ACT on the current usage of PETS dollars to ascertain if this will
be an issue for your agency
 Evaluate and ACT on resources and client population to avoid OOS
Capacity building of evaluation will be essential as the need to examine
agency needs increases
HOW TO ADDRESS UNDER WIOA
Good communication/interpersonal skills

In order to set up fiscal/counselors’ tracking for PETS monies

Communicate to staff and clients Order of Selection (OOS) policies and status

Communicate the change in common measures so counselors and managers are
cognizant of what is measured
 Why it is necessary
 Estimated time on the waiting list
 Procedures to take clients off the waiting list
Teaching the findings and process of evaluation

Teach common measures and how these are calculated to managers and staff so
that they can track progress

Have management and staff understand what led the agency to OOS and develop
strategies to operate without this limitation
PERFORMANCE MEASURES
“Sanctions for states that fail to meet targets”
 Employment
(2nd and 4th quarter accountability; retention)




Earnings
Credential attainment
Measurable skill gains
Serving employers
These indicators add employment outcomes on earnings and
effectiveness in serving employers, and include educational
measures for credential attainment and skill gains.
 Workforce3 One Webinar: Session 4, Performance Accountability System (September 2014)
 WIOA Game Plan for Low-Income People: http://www.clasp.org/issues/postsecondary/wioa-game-plan
 ION https://wioa.workforce3one.org/page/about
HOW AND WHY ALKIN (2013)
PROMOTES ACCOUNTABILITY WITH
THE INDICATORS
 Performance reports
10 reports required to inform on outcomes for core programs










Level of performance for core programs
Level of performance for individuals with barriers to employment (disaggregated)
Total number of participants served
Number of participants who received and exited career and training services within 3 years
Average cost per participant for career and training services
Percentage of participants who obtain unsubsidized employment in relevant field
Number of individuals with barriers to employment served
Number of participants co-enrolled
Percentage of annual allotment spent on administrative costs
Other information for program comparisons with other States
 Evaluations
 Promote continuous improvement, achieve high performance, achieve outcomes
 Fiscal and management accountability
 Promote efficient collection and use for reporting, monitoring, and preparing
annual reports. Must have access to quarterly wage records and interstate
arrangements
EXAMPLE OF EVALUATION UNDER WIOA
PARTNERSHIP * PLANNING * METHOD * IMPROVEMENT * VR MISSION
 Youth program elements across all workforce programs





Financial literacy
Entrepreneurial skills training
Labor market services
Transition to postsecondary education and training
Education offered in same context as workforce preparation activities
and training for occupation/cluster
 PETS






Job exploration counseling
Work-based learning experiences
Counseling on opportunities for enrolling in postsecondary education
Workplace readiness training
Instruction in self-advocacy
Other
 LEAD Center Webinar: Understanding changes regarding youth services, Part 3 (June 2015)
GROUP DISCUSSION
 What are your experiences with PLANNING FOR EVALUATION and
IMPROVEMENT under WIOA?
 State requirement for a unified approach
 Individual program planning approach (internal plans for integrity with
PETS/Business)
 What action steps are you taking to ‘engage’ customers/obtain
FEEDBACK-CONTINUOUS IMPROVEMENT in these busy times?
 What common TOOLS can be used to complete EVALUATION so that
we have comparison across states?



Pre-employment Transition Services (PETS)
Order of Selection (OOS)
Business Services
Notes Posted on SG Website
FOR MORE INFORMATION
Darlene Groomes
248-370-4237
groomes@oakland.edu
Summit Group Website
www.vocational-rehab.com
Three takeaway publications that can help your team to conceptualize working in
partnership and striving to improve how and why these partnerships work:
Evaluating
Partnerships
Collective
Impact
Evaluating
Strategic
Alliances
To join a future reading group, visit:
http://vocational-rehab.com/summit-reading-groups/