Continuous
Improvement
Collecting, Analyzing,
and Sharing Data
At the end of this presentation,
you will know:
1)
2)
3)
4)
5)
6)
All the wonderful ways that you can use data
for important purposes
What “formative evaluation” is, and how it
relates to continuous improvement
How to organize yourselves for formative
evaluation
How to collect data and use it to fuel continuous
improvement
Different approaches to analyzing, reporting on,
reflecting on, and sharing your data
How to work with C-PAD to accomplish datarelated tasks
The Joys of Data!
Data are not just for grant
requirements anymore!
Continuous
Improvement
Positioning
your
Program
for Funding
Marketing
your
Program
By using data for continuous
improvement, you will increase:
 Participation
 Satisfaction
 Tailoring
 Effectiveness
Using Formative Evaluation
for Continuous Improvement
What is formative evaluation?
• Evaluation = using data for
assessment and learning
• Formative = for the purposes of
creating, developing, and improving
• Formative (program) evaluation
= Using data to inform how you can
continually pursue program
excellence
Formative Evaluation as a Tool
for Continuous Improvement
Formative evaluation
asks:
So that your team can
further investigate:
• Bullet Points
In order to
achieve:
Is the program reaching its
participation targets?
How can we attract more
people to the program?
Increased
participation
Are participants satisfied
with the offerings?
How can we increase
satisfaction levels?
Increased
satisfaction
Does the design of services
meet the needs of its
participants?
How can we better align
service offerings with
participant needs?
Increased
tailoring
Do participants succeed in
desired outcomes?
How can we increase the
rate of positive outcomes?
Increased
effectiveness
We are busy delivering a
program – how can we be
evaluators too?
• Forget: “evaluation”
• Remember: “leveraging data
for program excellence!”
Story from the Field:
Leveraging data for program
excellence
• Julie Seeley, Spoon River
College
The Formative Evaluation Toolkit:
A Guided Tour
What you will find in the FET:
• Many pages (but not to worry!)
• Step-by-step guidance through the phases of evaluation
• Tools and templates for conducting your evaluation
• Templates for working meetings to accomplish
evaluation tasks
• Checklists at the end of each chapter to keep you on
track
• Blank pages for notes, questions, and insights
• Support for:
– Completing deliverables due to Champion Colleges
– Fulfilling grantee reporting requirements
Step-by-step guidance for:
•Data collection (chapters 3 & 4)
•Organizing for evaluation (chapter 2)
•Reporting on findings (chapter 5)
•Reflecting on findings (chapter 5)
Data Collection
(Chapters 3 and 4)
Data Types
• Process data
• Outcome data
• Stakeholder feedback
Process Data
(Chapter 3)
• Counts of program participants
– Students completing workforce program
courses
– Students participating in support services
• Information about program components,
describing:
– Workforce programs and courses
– Math, English, and computer courses
Outcome Data
(Chapter 3)
• Number of students who have:
– Received credits for prior learning
– Received a degree
– Received a certificate
– Received a non-credit certificate
– Become employed
Important Considerations for
Collecting Process & Outcome Data
• “Extractable” v. “Real-Time” Data
• Make a data collection plan
– Identify the data you need to collect
– Figure out how you’ll collect it
• Work with an IR Partner
• Plan early for real-time data collection
• Stay organized with data storage
FET Tools for
Process and Outcome Data
• Guidance for a working meeting to develop
process and outcome data collection plans
• Tools:
– Tool 2: Data Choices Table (p. 42)
– Tool 3: Process Data Collection Plan Worksheet (p. 45)
– Tool 4: Outcome Data Collection Plan Worksheet (p. 54)
– Tool 5: Options for Collecting Employment Data (p. 57)
– Data Storage Excel Workbook (URL on p. 17)
Stakeholder Feedback
(Chapter 4)
• Hearing from students, internal partners, and
community partners
• Looking for feedback that will help you to:
– Attract more students to the program
– Tailor services better
– Raise satisfaction levels
– Support students better for completion and
employment
Important Considerations for
Collecting Stakeholder Feedback
• Decide what you want to learn
• Make a data collection plan
– Decide which stakeholders you want to hear from
– Identify the data collection methods you want to
use
– Identify people responsible for collecting
feedback
• Make data collection instruments
• Don’t collect too much data!
FET Tools for
Stakeholder Feedback
• Guidance for two working meetings to develop:
– A stakeholder feedback data collection plan
– Data collection instruments
• Tools:
– Tool 6: Stakeholder Feedback Evaluation Questions
Worksheet (p. 61)
– Tool 7: Stakeholder Feedback Data Collection Plan
Worksheet (p. 63)
– Tools 8 & 9: Survey, Interview, and Focus Group
Protocol Templates (pp. 66, 82)
– Tools 10-12: Guidance for Conducting Interviews
and Focus Groups (pp. 88, 90, 91)
Organizing for Evaluation
(Chapter 2)
Assembling an Evaluation Team
• Evaluation Team Manager
• IR Partner
• Process and Outcome (P&O) Data
Lead
• Stakeholder Feedback Lead
Convene the Team for an
Evaluation Launch Meeting
• Review how the Plus 50 data tasks fit
together
• Customize your evaluation timeline
Reflection and Reporting
(Chapter 5)
Analysis, reflection, and
reporting as an iterative process
o Preliminary reporting: pulling together
data with initial analysis
o Reflecting on findings
o Writing up memos:
 Findings/Learnings
 Recommendations (based on
learnings and reflection)
FET Tools for
Reflecting on and Writing up
Evaluation Findings
• Guidance for working meetings to reflect:
– With the Plus 50 Team on key findings from the
process and outcome data
– With the Plus 50 Team on key findings from the
stakeholder feedback
– With the Plus 50 Advisory Committee on all key
evaluation findings
• Tools:
– Tool 13: Evaluation Questions Tool (p. 92)
– Tool 14: Qualitative Data Analysis Template (p. 95)
– Tool 15: Memo Write-up Tool (including an memo
outline) (p. 99)
Working with C-PAD
to Support Continuous
Improvement
Phase 5
1. Collect Process and Outcome Data
2. Gather Feedback from Program
Participants and Partners
3. Share Results with Key Stakeholders
4. Plan for Program Improvement
Feeding two birds with one seed:
Process and Outcome Data
in C-PAD
P&O data used for
continuous
improvement
=
P&O data used for
grant-required
reporting to AACC
Questions?