Data Analysis for Evaluation
Eric Graig, Ph.D.
Purpose of this Training
 To increase your skills in analysis and
interpretation of the information you collect
 To increase your ability to report your findings
to a variety of audiences
 To learn how to make evaluation results
actionable
Innovation Network, Inc.
Slide 2
Three Steps to Data Analysis
 Analyze Results
 Communicate Findings
 Use Findings for Program Improvement
Innovation Network, Inc.
Slide 3
Keep it Simple
 Aim for a systematic effort
 Keep your audience in mind
 Pay attention to the usability of your
evaluation report
Innovation Network, Inc.
Slide 4
Two Kinds of Data
 Quantitative
 Qualitative
Innovation Network, Inc.
Slide 5
Quantitative Data
 Data that is numerical, counted, or
compared on a scale




Demographic data
Answers to closed-ended survey items
Attendance data
Scores on standardized instruments
Innovation Network, Inc.
Slide 6
Qualitative Data
 Narratives, logs, experience
Focus groups
 Interviews
 Open-ended survey items
 Diaries and journals
 Notes from observations

Innovation Network, Inc.
Slide 7
Summary Statistics
 Measures of Central Tendency
 Mean
 Median
 Range
 Frequency distributions
 Percentage distributions
Innovation Network, Inc.
Slide 9
Frequency Distributions
To what extent did you increase your skills in
putting together a household budget?
Women (N=30)
Innovation Network, Inc.
A lot
Some
A little
Not at all
14
9
5
2
Slide 10
Percentage Distributions
To what extent did you increase your skills in
putting together a household budget?
Women (N=30)
Innovation Network, Inc.
A lot
Some
A little
Not at all
46%
30%
17%
7%
Slide 11
Graphing Frequency Data
How
about
thethe
web-site?
web site?
about
hear
firsthear
youfirst
didyou
How did
Court Referral
Social Worker
Friend
Web Search Engine
Librarian
Newspaper Story
Other
Innovation Network, Inc.
Percent
N
24.4%
10Referral
Court
Social
12.2%
5 Worker
Friend
12.2%
5 or Acquaintance
Librarian
19.5%
Web8Search Engine
Newspaper
Story 22.0%
9
Other
7.3%
3
2.4%
1
100.0%
41
Slide 12
Means and Medians
History
Math
English
English
95
98
96
96
Biology
History
Latin
Music
Math
Biology
Music
Latin
93
95
92
94
98
93
94
92
40
Gym
Innovation Network, Inc.
Mean = 87
Median = 94
Slide 13
Some Subtleties
40
50
55
94
100 100 100
Mean = 81
40
92
93
94
95
Mean = 87
Innovation Network, Inc.
96
98
Slide 14
Histograms
7
6
6
5
Frequency
Frequency
5
4
3
4
3
2
2
1
1
0
0
60
Innovation Network, Inc.
70
80
90
100
60
70
80
90
100
Slide 15
Cross Tabulations
Program Type
Web site
I & R Line
Law clinic
Web site
I & R Line
I & R Line
Web site
Law clinic
I & R Line
I & R Line
Law clinic
Web site
Law clinic
Web site
Law clinic
I & R Line
I & R Line
I & R Line
Law clinic
Area of Inquiry
Employment law
Family law
Immigration
Immigration
Immigration
Family law
Employment law
Other
Other
Other
Employment law
Family law
Family law
Immigration
Immigration
Family law
Immigration
Employment law
Other
Innovation Network, Inc.
Outcome
Satisfied
Not satisfied
Pending
Satisfied
Satisfied
Not satisfied
Not satisfied
Satisfied
Not satisfied
Satisfied
Satisfied
Satisfied
Satisfied
Not satisfied
Not satisfied
Satisfied
Not satisfied
Not satisfied
Pending
Count of Outcome Outcome
Program Type
Not satisfied Pending Satisfied Grand Total
I & R Line
7
5
12
Law clinic
1
3
7
11
Web site
6
5
11
Grand Total
14
3
17
34
Count of Outcome Outcome
Program Type
Not satisfied Pending Satisfied Grand Total
I & R Line
58%
0%
42%
100%
Law clinic
9%
27%
64%
100%
Web site
55%
0%
45%
100%
Grand Total
41.18%
8.82% 50.00%
100.00%
Slide 16
Graphing comparisons
Satisfaction with Services
40
Satisfaction Score
35
30
25
20
15
10
5
0
A
B
C
D
E
Clinic Name
Innovation Network, Inc.
Slide 17
Some subtleties
Satisfaction with Services
16
Satisfaction Score
14
12
10
Staff
Advice
Facility
8
6
4
2
0
A
B
C
D
E
Clinic
Innovation Network, Inc.
Slide 18
Some more subtleties
Satisfaction with Services
16
Satisfaction Score
14
12
A
B
C
D
E
10
8
6
4
2
0
Staff
Advice
Facility
Satisfaction Component
Innovation Network, Inc.
Slide 19
A Note on Quantitative Data Entry
 Several approaches
Paper and pencil tally
 Word processing table
 Spreadsheet
 Custom database

Innovation Network, Inc.
Slide 20
Qualitative Data
 Textual data
Interview transcripts
 Case notes/ clinical notes
 Open-ended survey questions

 Photographs
 Video recordings
Innovation Network, Inc.
Slide 21
Qualitative Analysis
 Identify themes or patterns


Pre-set themes
Determine as you go
 Tag text segments with themes
Innovation Network, Inc.
Slide 22
Sample data sheet
Site
Interview Write Up Sheet
Naming Standard:
Siteinterviewtype#
Cluster ND Futures
Site
Oakville
Interviewee
Diane Janssen, Elementary School Principal
Date of Interview
December 2004
Tape # 02-1234-PCM
Length 31
Interviewee
Interview Types:
Date
Focus
Com
Coach
Sup
Plan
Other
Site
Name
Ellendal
Intervie
w Type/#
Plan
Questio
Text
Tape number
n ?#
PC.1
e
She states that she came from to the town to attend
the local Bible College, stayed to become a teacher in
the local school and then took a job teaching at the
Length
college before becoming the principal of the
elementary school. She states that she first got
involved with the program after the large community
meeting that took place in April of 2004. The school
Question number
leadership (she doesn't indicate precisely who) felt it
was important that someone from the school get
involved in the planning committee so she volunteered
since she was interested in getting involved anyway.
She mentions both a desire to bring to the committee
the perspective of the school while at the same time to
Innovation Network, Inc.
take back to the school whatever knowledge she
acquired through participation.
PC.2
She talks a little bit about the school. It is the K-12
school with about 350 children. She oversees the K-6
program while another principal supervises grades 7
through 12. There are 157 students in the elementary
Slide 23
Exercise
 Break into small groups
 Using the sheet provided search for snippets
of text that relate to the codes provided
 Identify any other themes that may be
important that are not provided on the sheet
Innovation Network, Inc.
Slide 24
Communicating Findings
“Information that is not effectively
shared with others will not be
effectively used.”
Source: Building a Successful Evaluation
Center for Substance Abuse Prevention
Innovation Network, Inc.
Slide 25
Audience
 Staff
 Partners
 Board
 Other agencies
 Funders
 Public
Innovation Network, Inc.
Slide 26
Reporting Formats
 Written report
 Data book
 Flyer
 Formal oral presentation
 Round table with partners/ stakeholders
 Poster
 Web site
Innovation Network, Inc.
Slide 27
Whatever strategy you choose:
 Link findings to the program’s outputs
and desired outcomes
 Include successes and challenges
 Support your claims with data
 Acknowledge knowledge gaps
Innovation Network, Inc.
Slide 28
Organizing Findings
 Overview

Describe program/ describe evaluation
 Program Implementation

What did you do, and how well did you do it?
 Program Outcomes:

What were the changes that occurred?
 Lessons learned

Sub-group differences
Innovation Network, Inc.
Slide 29
Overview Section
 Your organization
 Your program
 Purpose of the evaluation
 Evaluation design
 Structure of the report
Innovation Network, Inc.
Slide 30
Program Implementation
 Characteristics of program participants
 Demographics
 Outputs
 Number of sessions held
 Number of participants served
 Partnerships formed
 Satisfaction with program
Innovation Network, Inc.
Slide 31
Program Outcomes





What has changed?
What do participants now know?
What are they now able to do?
What do they actually do?
How are there lives different?
Innovation Network, Inc.
Slide 32
Lessons Learned
 Which of the program’s assumptions were validated,





which need to be reconsidered?
Does the program affect different kinds of people
differently?
Are results better with some groups?
How will findings be addressed?
What new hypotheses did the evaluation generate?
How did the evaluation work?
Innovation Network, Inc.
Slide 33
Additional Resources
 One-on-one technical assistance
 Synchronous course
 Asynchronous material
 InnoNet website www.innonet.org
Innovation Network, Inc.
Slide 34