Midterm Review Evaluation & Research Concepts Proposals & Research Design Measurement Sampling Survey methods Purposes of Proposal • Communicate with Client • Demonstrate your grasp of problem • Plan the study in advance, so others can evaluate the study approach – – – – will it work? have you overlooked something? will results be useful to client? Can we afford it? Proposal Format 1. Problem Statement - define program to be evaluated/problem to be studied, users & uses of results. Justify importance of the problem/study. 2. Objectives : Concise listing . In evaluation studies, the objectives usually focus on the key elements of program to be evaluated & the evaluation criteria. These are the study objectives NOT the program objectives. 3. Background/Literature Review - place for more extensive history/structure of program. Focus on aspects most relevant to proposed evaluation. Discuss previous studies or the relevant methods. 4. Methods - details on procedures for achieving objectives - data gathering and analysis, population, sampling, measures, etc. Who will do what to whom, when, where, how and why? 5. Attachments - budget, timeline, measurement instruments, etc. NOTE: Most “programs” must be narrowed to specific components to be evaluated. Think of a “Program of studies” rather than a single evaluation study. The proposal should define this specific study & how it fits into a broader program of studies. Sample Objectives 1. Estimate benefits and costs of program 2. Estimate economic impacts of program on local community (social, environmental, fiscal). 3. Determine effects of program on target population. 4. Describe users and non-users of program 5. Assess community recreation needs, preferences 6. Determine market/financial feasibility of program 7. Evaluate adequacy or performance of program Research Process Define Problem, Research Objectives What? •Concepts •Variables •Measures HOW? Overall Method •Survey •Experiment •Case Study •Secondary Data Data Gathering Analysis Application Who? •Population •Sampling Methods Choices • Overall Approach/Design – Qualitative or Quantitative – Primary or secondary data – Survey, experiment, case study, etc. • Who to study - population, sample – individuals, market segments, populations • What to study - concepts, measures – behavior, knowledge, attitudes • Cost vs Benefit of Study Major Design Types • • • • • Surveys Experiments Observation Secondary Data Qualitative Approaches – Focus Group – Case Study Research Designs/Data Collection Approaches How ....Where Gathered Household On-Site Laboratory Personal Interview Surveys Surveys, Field Expmts Focus Groups Telephone/ Computer Self-Admin. Quest. Surveys Computer Interviews Surveys, Field Expmts Computer Interviews Experiments Observable Observable Characteristics Characteristics Observation & Traces NA Secondary Sources NA Internal Records NA General Guidelines on when to use different approaches 1. Describing a population - surveys 2. Describing users/visitors - on-site survey 3. Describing non-users, potential users or general population - household survey 4. Describing observable characteristics of visitors - on-site observation 5. Measuring impacts, cause-effect relationships experiments Guidelines (cont) 6. Anytime suitable secondary data exists secondary data 7. Short, simple household studies - phone 8. Captive audience or very interested population - self-administered survey 9. Testing new ideas - experimentation or focus groups 10. In-depth study - in-depth personal interviews, focus groups, case studies Primary or Secondary Data • Secondary data are data that were collected for some purpose other than your study, e.g. government records, internal documents, previous surveys • Choice between Primary /Secondary Data – Costs (time, money, personnel) – Relevance, accuracy, adequacy of data Qualitative vs Quantitative Approaches Qualitative Focus Group In-Depth Interview Case Study Participant observation Secondary data analysis Quantitative Surveys Experiments Structured observation Secondary data analysis Survey vs Experiment Survey - measure things as they are, snapshot of population at one point in time, generally refers to questionnaires (telephone, self-administered, personal interview) Experiment - manipulate at least one variable (treatment) to evaluate response, to study cause-effect relationships (field and lab experiments) Definition & Measurement “measurement is the beginning of science, … until you can measure something, your knowledge is meager and unsatisfactory” Lord Kelvin Nominal/Conceptual Definition - define concept in terms of other concepts, links concepts without tying them to real world Operational definition - equates definition with measurement, specify procedures/operations to generate the concept. Levels of Measurement Level Characteristic Example Nominal Unordered categories Ordinal Ordered categories Sm, med.lg Hardness scale Interval Consistent distance Temp in fahrenheit between categories or Celsius Ratio Natural zero Race, gender Temp in Kelvin Validity vs Reliability Questionnaire Design 1. Preliminary Info Information needed Who are subjects Method of communication 2. Question Content 3. Question Wording 4. Response Format 5. Question Sequencing/Layout What Info? Demographic, Socioeconomic, Physical Cognitive - Knowledge & beliefs Affective - attitudes, feelings, preferences Behavioral - actions Sampling • Always define study population first • Use element/unit/extent/time for complete definition • • • • element - who is interviewed sampling unit - basic unit containing elements extent - limit population (often spatially) time - fix population in time Types of Sampling Approaches • • • • Probability vs non-Probability Judgment, Simple Random, Systematic Stratify or Cluster (Area Sample) Time Sampling Sample size • Based on four factors • • • • Cost/budget Accuracy desired variance in popln on variable of interest subgroup analysis planned • Formula: • • n= Z2 2 / e 2 n= sample size Z indicates confidence level (95% = 1.96) • = standard deviation of variable in population • e = sampling error Sampling errors for binomial (95% confidence interval) percent distribution in population Sample size 100 50/50 60/40 70/30 80/20 90/10 10.0% 9.8% 9.2% 8.0% 6.0% 200 7.1% 6.9% 6.5% 5.7% 4.2% 400 5.0% 4.9% 4.6% 4.0% 3.0% 1000 3.2% 3.1% 2.9% 2.5% 1.9% 1500 2.6% 2.5% 2.4% 2.1% 1.5% 2000 2.2% 2.2% 2.0% 1.8% 1.3% Computing 95% confidence interval • N= 100 , sample mean = 46%, use p= 50/50, • sampling error from table = 10% • 95% CI is 46% + or - 10% = (36, 56) • N=1,000 sample mean =22% • sampling error from table = 2.5% • 95% CI is 22% + or - 2.5% = (19.5, 24.5) STEPS IN A SURVEY 1. Define problem and study objectives 2. Identify information needs & study population(s) 3. Determine basic design/approach - cross sectional vs longitudinal - on-site vs household vs other - self-admin. vs personal interview vs phone - structured or unstructured questions 4. Questionnaire design 5. Choose sample (frame, size, sampling design) 6. Estimate time, costs, manpower needs, etc. Survey Implementation 7. Proposal & “Human subjects” review 8. Line up necessary resources 9. Pre-test instruments and field procedures 10. Data gathering and follow-up procedures 11. Coding, cleaning and data processing 12. Analysis: preliminary, then final. 13. Communication and presentation of results.