Program Evaluation Essentials
Evaluation Support 2.0
Session 2
Anita M. Baker, Ed.D.
Jamie Bassell
Evaluation Services
Bruner Foundation
Rochester, New York
Evaluation Support 2.0
Sponsored by the Bruner Foundation www.evaluativethinking.org
and Evaluation Services www.evaluationservices.co
Free evaluation training and technical assistance
focused on development of evaluative capacity
including data analysis and reporting.

Four (4), on-site, hands-on training sessions.
Introduction to and use of free/low-cost tools to
facilitate data entry, management and analysis.



Guided evaluation project required.
Virtual conference with funder, other organization
participants.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
What do you need to do to conduct
Evaluation?

Specify key evaluation questions

Specify an approach (evaluation design)

Apply evaluation logic
 Collect

Bruner Foundation
Rochester, New York
and analyze data
Summarize and share findings
Anita M. Baker, Evaluation Services
1
What happens after data are collected?
1.
Data are entered, managed and analyzed
according to plans. Results/findings are
summarized.
2.
Findings must be converted into a format
that can be shared with others.
3.
Action steps should be developed from
findings.
“Now that we know _____ we will _____.”
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
2
Analyzing Quantitative Data:
A Few Important Terms*
•
Case: individual record (e.g., 1 participant, 1 day, 1 activity)
•
Demographics: descriptive characteristics (e.g., gender)
•
Disaggregate: to separate or group information (e.g., to look
at data for males separately from females) – conducting
crosstabs is a strategy for disaggregating data.
•
Partition(v): another term that means disaggregate.
•
Unit of Analysis: the major entity of the analysis – i.e., the
what or the whom is being studied (e.g., participants, groups,
activities)
•
Variable: something that changes (e.g., number of hours of
attendance)
*common usage
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
3
Plan your Analysis in Advance!
• What procedures will be conducted with each set of
data and who will do them?
• How will data be coded and recoded?
• How will data be disaggregated (i.e. “broken out for
example by participant characteristics, or time).
• How will missing data be handled.
• What analytical strategies or calculations will be
performed (e.g., frequencies, cross-tabs).
• How comparisons will be made.
• Whether/which statistical testing is needed.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
4
Quantitative Data Analysis: Basic Steps
1. Organize and arrange data (number cases
as needed).
2. Scan data visually.
3. Code data per analysis plan.
4. Enter and verify data.
5. Determine basic descriptive statistics.
6. Recode data as needed (including missing
data).
7. Develop created variables.
8. Re-calculate basic descriptive statistics.
9. Conduct other analyses per plan
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
5
Coding and Data Entry
1. Create codebook(s) as needed (identify codes
and affix them to instrument copies).
2. Create electronic database when possible (use
Excel, Survey Monkey, SPSS, SAS, others).
3. ID/create unique identifiers for cases and affix
or enter as needed.
4. Enter or extract data as needed (do not recode
as data are entered).
5. Make (electronic or paper) copies of your data.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
6
Strategies for Analyzing Quantitative Data
Important Things to Look at or Summarize
Frequencies: How often a response or status occurs.
Total and Valid Percentages: Frequency/total *100
Measures of Central Tendency: Mean, Median, (Modes)
Distribution: Minimum, Maximum, Groups (*iles)
Cross-Tabulations: Relationship between two or more
variables (also called contingency analyses, can include
significance tests such as chi-square analyses)
Useful, 2nd Level Procedures
Means testing (ANOVA, t-Tests)
Correlations
Regression Analyses
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
7
Analyzing Quantitative Data
Important Things to Look at or Summarize
What to Do
Calculate Frequencies
What That Means
Count how many there are of
something.
Count how often something (e.g., a
response) occurs.
Calculate Total and/or
Valid Percentages
Frequency/total *100
Example Questions You
Could Answer
How many participants were in
each group?
What were the demographics
of participants?
How many answered “Yes” to
Question 2?
What proportion of participants
met intensity targets?
What proportion of all those
who answered question 2, said
“Yes.”
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
8
Analyzing Quantitative Data
Important Things to Look at or Summarize
Example Questions You
Could Answer
What to Do
What That Means
Determine
Central Tendencies
Calculate the average (mean), or
identify the median (middle) or
mode (most common value).
What is the average number of
hours participants attend?
Avg. =
What is the most common
numbers of days attended in a
week? (mode)
Sum of Values
Total Number of Values
Total # of hours
Total # of people with hours
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
9
Analyzing Quantitative Data
Important Things to Look at or Summarize
What to do
What That Means
Example Questions You
Could Answer
Determine Distributions
Determine the minimum value, the
maximum, and/or how the data
are grouped
What was the least amount of
attendance for the group?
What was the most?
(e.g, high, medium, or low values,
quartiles, percentiles, etc.).
How many participants fall into
low, medium, and high
intensity groups?
Cross-Tabulations
(pivot tables are
crosstabs)
Relationship between 2 or more
variables (also called contingency
analyses, can include significance
tests such as chi-square analyses)
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
Are there relationships
between participant
characteristics and outcome
changes?
10
Measuring Change or Difference
Sometimes analysis focuses on change between two
(or more) points in time and/or on differences between results.
What to Do
What That Means
Calculate percentage
change or percentage
difference
Difference between two
NUMBERS
Example Questions You
Could Answer
How much did the program grow in
terms of participants or dollars used
or hours spent in year 2 vs. year 1?
How different was site 1 from site 2
in terms of program hours?
Calculate percentage
point change
Difference between two
PERCENTAGES
Which site had proportionally more
students who achieved outcomes?
Did the proportion of students
getting the correct answer change
over time?
Conduct means testing Use tests to determine if results
or chi square analyses are statistically different (means What is the probability that
tests such as t tests or ANOVA
for numbers, chi square
commonly for percentages)
observed differences are due strictly
to chance?
11
Excel Basics
Adding/deleting rows
 Undo
 Formatting cells


Hide/Unhide

Freeze panes

Sorting

Copying and pasting (formulas)

Counts, sums, and averages in status bar
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
12
Using Excel to Analyze Data
 In
addition to automatic re-coding
formulas, summary formulas or Automatic
Calculators can be added to databases.
 The
results in the automatic calculators
can be used like a look-up table to answer
analytical questions.
 Formulas
can be copied and pasted and
modified as needed.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
13
Using Databases: Summarizing
CALCULATORS
Denominator
Function (with argument) = Result
Result/Denominator = %
Checking and Verifying
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
14
Function =
COUNTING
=COUNTA(D2:D298)
Counts any occupied cell
=COUNT(G2:G298)
Counts any cell with a number in it
(also in
status bar)
=COUNTIF(V2:V:298,1)
Finds all the 1’s in Column V (1 is the code for new to YMCA)
(Similarly, =COUNTIF(D2:D298,”F”) will count all cells in Column D containing the
letter F – the code for Female). Please note, for alpha data, you MUST use full
quotation marks around the code you are searching for. AND IT MUST BE
EXACT.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
15
Basic Analyses

Frequencies for all descriptive data (e.g.,
gender, race/ethnicity, age, living status, location, grade,
program participation etc.)
Calculating age from birth-date
 Re-coding data


Cross-tabulating data

Determining adjusted attendance rates

Automatically comparing data to targets
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
16
Summarizing and Recoding:
Capturing Attendance Data/Hitting Targets
Attendance tracking is relatively straightforward.
 Rows = individuals eligible to attend
 Columns = all the possible dates for attendance
 Data = 1 if in attendance
(0 if absent, blank if not expected)
Add a calculator:
 Total number of sessions attended per person
 Total number of persons recorded per activity
(Attendance Report – Signed Up)
 Total
activity attendance for those enrolled
Add a TOTALS WORKSHEET
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
17
Summarizing and Recoding:
Capturing Attendance Data/Hitting Targets
Recode attendance results to determine if targets have been met!

Did individual participants attend the desired number of
sessions?

How many participants attended the desired number of
sessions?

You could ask many other questions such as who attended
the desired number of sessions, who did not, etc., by any of
your partitions of interest.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
18
Summarizing and Recoding:
Adjusting Attendance Data/Hitting Targets

If (logical _test,[value_if true], value_if false])
Target: Participants will attend at least 8 activities in a
month.
 if (AA278>=8, “YES”,”NO”)
IN ENGLISH:
The total number of sessions attended in cell AA278 is
compared to see if it is greater than or equal to 8. If so, the
case gets a YES to signify meeting the target, if not the case
gets a NO.
Note that the total number of participants meeting the target
is also summarized (AC298)
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
19
Multivariate Analysis: Crosstabs
Like everything else in Excel, there is more than one way
to conduct multivariate analyses – i.e., to look at more
than one variable at a time.


Pivot Tables
Cheaters Cross-tabs
 Sort database
 Copy database – make as many copies as partitions*
 Delete those not in the partition, use the calculator
as a look up for each partition.
*A partition is a variable that divides the data into groups of interest. For
example, RACE/ETHNICITY, SEX, AGE, INCOME LEVELS are all partitions.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
20
Other Handy Strategies
Link formulas across sheets: =‘EXACTNAMEOFOTHERSHEET’!B3
or other cell you want
='2015 DATA'!B2
Calculating averages:
*Use the click and drag function when you can OR
* =AVERAGE(G2:G298)
Be sure to decide what to do if your data has zeros in it. They
may artificially lower the average. Try sorting, and setting
your data range to include only cells with non-zero numbers.
Bruner Foundation
Rochester, New York
Anita M. Baker, Evaluation Services
21