Midterm Review
Evaluation & Research Concepts
Proposals & Research Design
Measurement
Sampling
Survey methods
Purposes of Proposal
• Communicate with Client
• Demonstrate your grasp of problem
• Plan the study in advance, so others can
evaluate the study approach
–
–
–
–
will it work?
have you overlooked something?
will results be useful to client?
Can we afford it?
Proposal Format
1. Problem Statement - define program to be evaluated/problem to be studied,
users & uses of results. Justify importance of the problem/study.
2. Objectives : Concise listing . In evaluation studies, the objectives usually
focus on the key elements of program to be evaluated & the evaluation criteria.
These are the study objectives NOT the program objectives.
3. Background/Literature Review - place for more extensive
history/structure of program. Focus on aspects most relevant to proposed
evaluation. Discuss previous studies or the relevant methods.
4. Methods - details on procedures for achieving objectives - data gathering and
analysis, population, sampling, measures, etc. Who will do what to whom, when,
where, how and why?
5. Attachments - budget, timeline, measurement instruments, etc.
NOTE: Most “programs” must be narrowed to specific components to be evaluated.
Think of a “Program of studies” rather than a single evaluation study. The proposal
should define this specific study & how it fits into a broader program of studies.
Sample Objectives
1. Estimate benefits and costs of program
2. Estimate economic impacts of program on local
community (social, environmental, fiscal).
3. Determine effects of program on target
population.
4. Describe users and non-users of program
5. Assess community recreation needs, preferences
6. Determine market/financial feasibility of
program
7. Evaluate adequacy or performance of program
Research Process
Define Problem, Research Objectives
What?
•Concepts
•Variables
•Measures
HOW?
Overall Method
•Survey
•Experiment
•Case Study
•Secondary Data
Data Gathering
Analysis
Application
Who?
•Population
•Sampling
Methods Choices
• Overall Approach/Design
– Qualitative or Quantitative
– Primary or secondary data
– Survey, experiment, case study, etc.
• Who to study - population, sample
– individuals, market segments, populations
• What to study - concepts, measures
– behavior, knowledge, attitudes
• Cost vs Benefit of Study
Major Design Types
•
•
•
•
•
Surveys
Experiments
Observation
Secondary Data
Qualitative Approaches
– Focus Group
– Case Study
Research Designs/Data Collection Approaches
How ....Where
Gathered
Household
On-Site
Laboratory
Personal
Interview
Surveys
Surveys,
Field Expmts
Focus Groups
Telephone/
Computer
Self-Admin.
Quest.
Surveys
Computer
Interviews
Surveys,
Field Expmts
Computer
Interviews
Experiments
Observable
Observable
Characteristics
Characteristics
Observation
& Traces
NA
Secondary
Sources
NA
Internal
Records
NA
General Guidelines on when to
use different approaches
1. Describing a population - surveys
2. Describing users/visitors - on-site survey
3. Describing non-users, potential users or
general population - household survey
4. Describing observable characteristics of
visitors - on-site observation
5. Measuring impacts, cause-effect relationships experiments
Guidelines (cont)
6. Anytime suitable secondary data exists secondary data
7. Short, simple household studies - phone
8. Captive audience or very interested population
- self-administered survey
9. Testing new ideas - experimentation or focus
groups
10. In-depth study - in-depth personal interviews,
focus groups, case studies
Primary or Secondary Data
• Secondary data are data that were
collected for some purpose other than
your study, e.g. government records, internal documents,
previous surveys
• Choice between Primary /Secondary
Data
– Costs (time, money, personnel)
– Relevance, accuracy, adequacy of data
Qualitative vs Quantitative Approaches
Qualitative
Focus Group
In-Depth Interview
Case Study
Participant observation
Secondary data analysis
Quantitative
Surveys
Experiments
Structured observation
Secondary data analysis
Survey vs Experiment
Survey - measure things as they are, snapshot
of population at one point in time, generally
refers to questionnaires
(telephone, self-administered, personal interview)
Experiment - manipulate at least one variable
(treatment) to evaluate response, to study
cause-effect relationships
(field and lab experiments)
Definition & Measurement
“measurement is the beginning of science, … until you
can measure something, your knowledge is meager and
unsatisfactory” Lord Kelvin
Nominal/Conceptual Definition - define concept
in terms of other concepts, links concepts
without tying them to real world
Operational definition - equates definition with
measurement, specify procedures/operations
to generate the concept.
Levels of Measurement
Level
Characteristic Example
Nominal
Unordered
categories
Ordinal
Ordered categories Sm, med.lg
Hardness scale
Interval
Consistent distance Temp in fahrenheit
between categories or Celsius
Ratio
Natural zero
Race, gender
Temp in Kelvin
Validity vs Reliability
Questionnaire Design
1. Preliminary Info
Information needed
Who are subjects
Method of communication
2. Question Content
3. Question Wording
4. Response Format
5. Question Sequencing/Layout
What Info?
Demographic, Socioeconomic, Physical
Cognitive - Knowledge & beliefs
Affective - attitudes, feelings, preferences
Behavioral - actions
Sampling
• Always define study population first
• Use element/unit/extent/time for complete
definition
•
•
•
•
element - who is interviewed
sampling unit - basic unit containing elements
extent - limit population (often spatially)
time - fix population in time
Types of Sampling Approaches
•
•
•
•
Probability vs non-Probability
Judgment, Simple Random, Systematic
Stratify or Cluster (Area Sample)
Time Sampling
Sample size
• Based on four factors
•
•
•
•
Cost/budget
Accuracy desired
variance in popln on variable of interest
subgroup analysis planned
• Formula:
•
•
n= Z2 2 / e 2
n= sample size
Z indicates confidence level (95% = 1.96)
•  = standard deviation of variable in population
• e = sampling error
Sampling errors for binomial
(95% confidence interval)
percent distribution in population
Sample
size
100
50/50
60/40
70/30
80/20
90/10
10.0%
9.8%
9.2%
8.0%
6.0%
200
7.1%
6.9%
6.5%
5.7%
4.2%
400
5.0%
4.9%
4.6%
4.0%
3.0%
1000
3.2%
3.1%
2.9%
2.5%
1.9%
1500
2.6%
2.5%
2.4%
2.1%
1.5%
2000
2.2%
2.2%
2.0%
1.8%
1.3%
Computing 95% confidence interval
• N= 100 , sample mean = 46%, use p= 50/50,
• sampling error from table = 10%
• 95% CI is 46% + or - 10% = (36, 56)
• N=1,000 sample mean =22%
• sampling error from table = 2.5%
• 95% CI is 22% + or - 2.5% = (19.5, 24.5)
STEPS IN A SURVEY
1. Define problem and study objectives
2. Identify information needs & study population(s)
3. Determine basic design/approach
- cross sectional vs longitudinal
- on-site vs household vs other
- self-admin. vs personal interview vs phone
- structured or unstructured questions
4. Questionnaire design
5. Choose sample (frame, size, sampling design)
6. Estimate time, costs, manpower needs, etc.
Survey Implementation
7. Proposal & “Human subjects” review
8. Line up necessary resources
9. Pre-test instruments and field procedures
10. Data gathering and follow-up procedures
11. Coding, cleaning and data processing
12. Analysis: preliminary, then final.
13. Communication and presentation of results.