Evaluation Overview
Plus 50 Conference
July 29 - 30, 2009
Washington, DC
Faces to Names and Voices:
Your Evaluation Liaisons
Kathleen Hentz
• Spokane
• Luzerne
• Richland
Laura Bekes
• Central Florida
• Wake Tech
• Northern Virginia
Samantha Chen
• Cape Cod
• Santa Fe
• Saint Louis
• Clark
• Western Dakota
Overview
• Why Evaluate?
• How Do You Evaluate the Plus 50
Initiative?
• How Do You Evaluate the Plus 50
Grantees?
Why Evaluate?
• DOCUMENT implementation
• CAPTURE lessons learned
• TRACK outcomes over time
• SHOW the effectiveness of a model
Overview
• Why Evaluate?
• How Do You Evaluate the Plus 50
Initiative?
• How Do You Evaluate the Plus 50
Grantees?
How Do You Evaluate the
Plus 50 Initiative?
Putting the evaluation puzzle
pieces together:
– Research Questions
– Research Tools
How Do You Evaluate the Plus 50 Initiative?
Identifying Research Questions
•Program Implementation:
•What progress have grantees made in implementing programs?
•Lessons Learned:
•What have grantees learned about what works and doesn’t work
in program implementation?
•Participant Outcomes:
•Is program participation having the hoped-for results among the
participants? Are they satisfied with the program?
How Do You Evaluate the Plus 50 Initiative?
Identifying Appropriate Research
Tools
•Data Collection Worksheets
•Participant Surveys
•Key Informant Interviews
How Do You Evaluate the Plus 50 Initiative?
Implementation Assessment
How Do You Evaluate the Plus 50 Initiative?
How is implementation progressing?
• What is the progress on workforce training
and career development goals for each
college?
•Data Collection Worksheets
•Key Informant Interviews
Average Number of Courses for Students Age 50+
Plus 50 Initiative Baseline Data
Training and Re-training Non-credit Courses
for Students Age 50+
July 1, 2007 - June 30, 2008
30
25
20
15
18
10
5
0
5
2
1
2
4
Unique
Total
1
2
Unique
Total
Mentor Colleges
(n=4)
Newly designed
Demonstration Colleges
(n=10)
Redesigned
Existing marketed
How Do You Evaluate the Plus 50 Initiative?
Lessons Learned
"The fact that we have monthly
enrollment is very helpful for the
students. …People are laid off
year-round not necessarily aligned
with an academic calendar.”
-Plus 50 College
"Short-term is very much in
demand, as opposed to something
long-term over many semesters.
We’re looking at short-term
programming that gets people out
and on their way.”
-Plus 50 College
"The registration process has a lot
of steps. We do it for them – if
they had to jump through all those
hoops they might give up. The
college has also allowed us to skip
a few steps.”
-Plus 50 College
How Do You Evaluate the Plus 50 Initiative?
Participant Outcomes Evaluation
How Do You Evaluate the Plus 50 Initiative?
Participant Outcomes Evaluation
• What outcomes do Plus 50 participants
experience as a result of the their
participation in training courses and
career services?
•Participant Surveys
How Do You Evaluate the Plus 50 Initiative?
Participant Outcomes Evaluation
Skills Acquired by Plus 50 Workforce Training
Participants
Completely Disagree
Mostly Disagree
Mostly Agree
Completely Agree
Acquisition of New Job Skills (n=130)
5% 5%
Improvement in Job Skills (n=131)
5% 6%
35%
54%
Improvement in Technology Skills
(n=119)
6% 6%
34%
55%
0%
31%
59%
25%
50%
75%
Percentage of Respondents
100%
How Do You Evaluate the Plus 50 Initiative?
Participant Outcomes Evaluation
Job Outcomes of Plus 50 Workforce Training Participants
Completely Disagree
Landed a job
(n=107)
Mostly Disagree
15%
14%
Mostly Agree
31%
Completely Agree
40%
Advanced in my
job
(n=56)
20%
Got a promotion
(n=49)
18%
29%
29%
25%
Got a raise
(n=49)
20%
25%
33%
22%
0%
13%
38%
30%
25%
50%
75%
Percentage of Respondents
100%
How Do You Evaluate the Plus 50 Initiative?
Participant Outcomes Evaluation
“I will always remember you for caring and the
important part you played in my success in
securing a position with FedEx Express. …I
received a call on Thursday to sign my offer letter
on Friday at 10AM!!! ….There were times when I
was almost running on empty. I wish many more
success stories for the 50+ workforce people.”
-Plus 50 Participant
How Do You Evaluate the Plus 50 Initiative?
Participant Outcomes Evaluation
“The classes made me more aware of where
I wanted to be going in my career at this
time in my life. Also reviewing my skills and
abilities, personality traits and strengths, I
was able direct my energies toward solid
decisions.”
-Plus 50 Participant
How Do You Evaluate the Plus 50 Initiative?
Participant Outcomes Evaluation
“Perhaps you could offer training in specifically
targeted skills. For example, a one-hour
[practical workshop] on Saturday morning –
practice with e-mailing attachments, practice
with storing folders, files, etc…..Practice time
with a knowledgeable person would be good.”
-Plus 50 Participant
How Do You Evaluate the Plus 50 Initiative?
Additional Research Questions
How Do You Evaluate the Plus 50 Initiative?
Other Things we Want to Know
• How is the Plus 50 Initiative raising public
awareness about the issues of Plus 50
learners, and the model?
• What have we learned about what works well in
mentorship relationships?
• How has the AACC staff supported grantees?
• Are the programs becoming sustainable, and
what strategies work for sustainability?
Overview
• Why Evaluate?
• How Do You Evaluate the Plus 50
Initiative?
• How Do You Evaluate the Plus 50
Grantees?
How Do You Evaluate the Plus 50 Grantees?
Looking at Individual Colleges
How Do You Evaluate the Plus 50 Grantees?
What is Success for Individual Colleges?
• Are colleges making progress against the
goals set out in the proposal?
• Are colleges appropriately responding to
lessons learned by finding ways to stay true
to the overarching program objectives while
changing implementation tactics to adjust to
circumstances?
How Do You Evaluate the Plus 50 Grantees?
What is Success for Individual Colleges?
•Information that helps inform AACC on
individual college Plus 50 implementation
and development:
•Progress Reports
•Site Visits
•Conference Calls
•Data Collection Worksheets
•Participant Surveys
Thoughts on the Year 1 Evaluation
• What’s working?
• What would work
better in the coming
year?
Evaluation Next Year
• Participant Surveys
• Data Collection
Forms for Year 2
• Key Informant
Interviews
Questions
Thank You!