The schedule moving forward…
• Today
– Evaluation Research
– Remaining time = start on quantitative data analysis
• Thursday
– Methodology section due
– Quantitative analysis continued (start Exercise 10)
• Next Tuesday
– Exercise 10 Due
– Maahs returns Exercise #9
• Research proposal (everything is now online) is due on
Dec 21st (Monday)
Evaluation Research
•
•
•
•
PURPOSE rather than specific method
Much more popular in last 20 years
A form of “applied” research
Results, though designed to impact decisionmaking, often have no impact
Types of Evaluation Research
•
•
•
•
Needs assessment
Cost-benefit study
Monitoring study
Program evaluation
Doing evaluation (program) research
• Research questions / measurement as Job #1
– Political context how do measure
– Operationalize “outcome” of interest
• “Response” variable
• NEED AGREEMENT: what is the program trying to do?
– Operationalize “processes”
• Intermediate objectives
– Context of
–”
Research designs for evaluation
• Experimental
– Problems, ethics (placebo)
– The “black box” issue
– Quasi-experimental designs
• Time Series (multiple time series)
• Qualitative evaluations
– Low birth weight study
Evaluation Research ISSUES
• Evaluation research as “MESSY”
– Administrative control, context of “real life”
• Importance of looking at “black box”
• Ethics
– The intervention itself can raise issues
– Control group members
– Violating Ethics 101 (Tuskegee)
– Evaluation impacts people’s lives
The (non) use of findings
• Purely rational/scientific world vs. the one we inhabit
– Nixon panel on porn
– Scared Straight
– 3 Strikes and You’re Out legislation
• Studies gone wrong…
– Fire the evaluator (or dismiss as pointy headed idiot) and
keep the program
– Why not trust/use the results?
• Scientist/practitioner gap
• “True believers”
• Vested interests
Bivariate Analysis
Backed up with a little inferential
statistics……yeah baby!
Review
• Descriptive statistics
– Purpose?
– Types?
– These are “univariate” statistics
• Explanatory research
– Attempt to demonstrate cause-effect
• Necessary Criteria?
Demonstrating Associations
• Bivariate (2 variables) analysis
– There are a number of ways to do this
• The method you choose depends largely on
characteristics of the two variables
– Levels of Measurement?
Contingency Tables (cross tabs)
• Some find these very intuitive…others struggle
– It is very easy to misinterpret these critters
• Convention: the independent variable is on the top
of the table (dictates columns) and the dependent
variable is on the side (dictates rows).
– What is in the individual “cells?”
• Frequencies (number of cases that fit criteria)
• Convert to Percentages: a way to standardize cells and make
relationships more apparent
Example
• A survey of 10,000 U.S. residents
• Research question: is one’s political view
related to attitudes towards police?
– What is the DV and IV?
– In constructing a table, what goes where?
An Example
Total
Political Party
Attitude
Towards
Police
Repub
Democrat
Libertarian Socialist
Favorable
2900
2100
180
30
5210
Unfavorab
1900
1800
160
28
3888
Total
4800
3900
340
58
9098
The Percentages of Interest
Total
Political Party
Attitude
Towards
Police
Repub
Democrat
Libertarian Socialist
Favorable
2900
(60%)
2100
(54%)
180
(53%)
30
(52%)
5210
Unfavorab
1900
1800
160
28
3888
Total
4800
3900
340
58
9098
Inferential Statistics
• Researchers are typically not interested in
whether there is a relationship in the sample
– Want to know about the population
– Why might there be a difference between what
you find in the sample and what actually exists in
the pop?
• Even with a probability sample, there is always
sampling error
• Without a probability sample = error + bias
Inferential Statistics II
• Cannot assume that the relationship in your
sample is true in population
– BUT: probability theory allows us to estimate:
“The likelihood of obtaining a particular finding
if in the population, there was no
relationship” OR
“The likelihood of obtaining a particular finding
assuming the null hypothesis”
Test Statistics and Significance Tests
•
What does “statistically significant” mean?
– Not due to sampling error
•
What do you need to do to test for statistical
significance?
1. A “test statistic”
•
An indicator of how different the sample is from “no
relationship”
2. The level of error that is acceptable (.05, .01)
3. Degrees of Freedom
The Test Statistic for Contingency Tables
• Chi Square, or χ2
– Calculation
• Observed frequencies
• Expected frequencies
– Can get confusing calculating these critters by hand
– Intuitive: how different are the observed cell frequencies
from the expected (under null hypothesis) cell frequencies
– Degrees of Freedom:
• (# of Rows -1) (# of Columns -1)
Political View (3 Category) * Respondent's Sex Crosstabulation
Political View
(3 Category)
Liberal
moderate
conservative
Total
Count
Expected Count
Count
Expected Count
Count
Expected Count
Count
Expected Count
Res pondent's Sex
Male
Female
157
229
166.9
219.1
215
312
227.9
299.1
252
278
229.2
300.8
624
819
624.0
819.0
Total
386
386.0
527
527.0
530
530.0
1443
1443.0
The Chi-Square Sampling Distribution
(Assuming Null is True)
Conventional Significance Testing
1. Calculate the test statistic
2. Set your “Type II” error level
–
The risk of being wrong that you are willing to live with
3. Find the “critical region” within the distribution for
your sample statistic
–
How far out on the curve does your test statistic have to
get before you can reject the null hypothesis
4. Decide whether to reject, or fail to reject the null
hypothesis
Significance Testing in SPSS
• Within your crosstabs, select “Chi-Square”
– SPSS gives you the χ2 value
• What you would calculate based on the observed and
expected cell frequencies
– SPSS also gives you “p”
• The exact probability of obtaining this χ2, if indeed the
null hypothesis was correct
• In newer versions of SPSS, the p values are listed under
“sig” or “significance” or something similar
Review of Contingency Tables
• Level of Measurement
– Both IV and DV are Nominal or Ordinal
(Categorical data)
• Constructing a table
– IV on top (columns), DV on bottom (rows)
• With this format, select “column percentages”
Review of Inferential Statistics
• Purpose of inferential statistics
– Figure out the odds that a finding from sample is due to
“chance” or “sampling error”
• Process
– Assume null hypothesis is correct
• Calculate the odds of obtaining your particular finding under this
assumption
• If the odds are very low, you may be suspicious that the null
hypothesis is incorrect
• At some point (research sets level), you say that the odds are so
low that you are going to go ahead and reject the null hypothesis
Review of Chi-Square
• A measure of how different observed (from
your sample data) cell frequencies are from
what would be expected under the null
hypothesis (e.g., no relationship)
– The Chi-square distribution changes shape with
different degrees of freedom
• This is because, by definition, the more cells you have,
the higher χ2 gets
– df = (R-1)(C-1)
Interpreting Chi-Square
• Chi-square has no intuitive meaning, it can
range from zero to very large
– The real interest is the “p value” associated with
the calculated chi-square value
– This is the exact probability of obtaining the chisquare if in the population there was no
relationship
• In other words, the exact probability of finding that chisquare, under the null hypothesis