Program Evaluation Essentials Evaluation Support 2.0 Session 2 Anita M. Baker, Ed.D. Jamie Bassell Evaluation Services Bruner Foundation Rochester, New York Evaluation Support 2.0 Sponsored by the Bruner Foundation www.evaluativethinking.org and Evaluation Services www.evaluationservices.co Free evaluation training and technical assistance focused on development of evaluative capacity including data analysis and reporting. Four (4), on-site, hands-on training sessions. Introduction to and use of free/low-cost tools to facilitate data entry, management and analysis. Guided evaluation project required. Virtual conference with funder, other organization participants. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services What do you need to do to conduct Evaluation? Specify key evaluation questions Specify an approach (evaluation design) Apply evaluation logic Collect Bruner Foundation Rochester, New York and analyze data Summarize and share findings Anita M. Baker, Evaluation Services 1 What happens after data are collected? 1. Data are entered, managed and analyzed according to plans. Results/findings are summarized. 2. Findings must be converted into a format that can be shared with others. 3. Action steps should be developed from findings. “Now that we know _____ we will _____.” Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 2 Analyzing Quantitative Data: A Few Important Terms* • Case: individual record (e.g., 1 participant, 1 day, 1 activity) • Demographics: descriptive characteristics (e.g., gender) • Disaggregate: to separate or group information (e.g., to look at data for males separately from females) – conducting crosstabs is a strategy for disaggregating data. • Partition(v): another term that means disaggregate. • Unit of Analysis: the major entity of the analysis – i.e., the what or the whom is being studied (e.g., participants, groups, activities) • Variable: something that changes (e.g., number of hours of attendance) *common usage Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 3 Plan your Analysis in Advance! • What procedures will be conducted with each set of data and who will do them? • How will data be coded and recoded? • How will data be disaggregated (i.e. “broken out for example by participant characteristics, or time). • How will missing data be handled. • What analytical strategies or calculations will be performed (e.g., frequencies, cross-tabs). • How comparisons will be made. • Whether/which statistical testing is needed. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 4 Quantitative Data Analysis: Basic Steps 1. Organize and arrange data (number cases as needed). 2. Scan data visually. 3. Code data per analysis plan. 4. Enter and verify data. 5. Determine basic descriptive statistics. 6. Recode data as needed (including missing data). 7. Develop created variables. 8. Re-calculate basic descriptive statistics. 9. Conduct other analyses per plan Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 5 Coding and Data Entry 1. Create codebook(s) as needed (identify codes and affix them to instrument copies). 2. Create electronic database when possible (use Excel, Survey Monkey, SPSS, SAS, others). 3. ID/create unique identifiers for cases and affix or enter as needed. 4. Enter or extract data as needed (do not recode as data are entered). 5. Make (electronic or paper) copies of your data. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 6 Strategies for Analyzing Quantitative Data Important Things to Look at or Summarize Frequencies: How often a response or status occurs. Total and Valid Percentages: Frequency/total *100 Measures of Central Tendency: Mean, Median, (Modes) Distribution: Minimum, Maximum, Groups (*iles) Cross-Tabulations: Relationship between two or more variables (also called contingency analyses, can include significance tests such as chi-square analyses) Useful, 2nd Level Procedures Means testing (ANOVA, t-Tests) Correlations Regression Analyses Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 7 Analyzing Quantitative Data Important Things to Look at or Summarize What to Do Calculate Frequencies What That Means Count how many there are of something. Count how often something (e.g., a response) occurs. Calculate Total and/or Valid Percentages Frequency/total *100 Example Questions You Could Answer How many participants were in each group? What were the demographics of participants? How many answered “Yes” to Question 2? What proportion of participants met intensity targets? What proportion of all those who answered question 2, said “Yes.” Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 8 Analyzing Quantitative Data Important Things to Look at or Summarize Example Questions You Could Answer What to Do What That Means Determine Central Tendencies Calculate the average (mean), or identify the median (middle) or mode (most common value). What is the average number of hours participants attend? Avg. = What is the most common numbers of days attended in a week? (mode) Sum of Values Total Number of Values Total # of hours Total # of people with hours Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 9 Analyzing Quantitative Data Important Things to Look at or Summarize What to do What That Means Example Questions You Could Answer Determine Distributions Determine the minimum value, the maximum, and/or how the data are grouped What was the least amount of attendance for the group? What was the most? (e.g, high, medium, or low values, quartiles, percentiles, etc.). How many participants fall into low, medium, and high intensity groups? Cross-Tabulations (pivot tables are crosstabs) Relationship between 2 or more variables (also called contingency analyses, can include significance tests such as chi-square analyses) Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services Are there relationships between participant characteristics and outcome changes? 10 Measuring Change or Difference Sometimes analysis focuses on change between two (or more) points in time and/or on differences between results. What to Do What That Means Calculate percentage change or percentage difference Difference between two NUMBERS Example Questions You Could Answer How much did the program grow in terms of participants or dollars used or hours spent in year 2 vs. year 1? How different was site 1 from site 2 in terms of program hours? Calculate percentage point change Difference between two PERCENTAGES Which site had proportionally more students who achieved outcomes? Did the proportion of students getting the correct answer change over time? Conduct means testing Use tests to determine if results or chi square analyses are statistically different (means What is the probability that tests such as t tests or ANOVA for numbers, chi square commonly for percentages) observed differences are due strictly to chance? 11 Excel Basics Adding/deleting rows Undo Formatting cells Hide/Unhide Freeze panes Sorting Copying and pasting (formulas) Counts, sums, and averages in status bar Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 12 Using Excel to Analyze Data In addition to automatic re-coding formulas, summary formulas or Automatic Calculators can be added to databases. The results in the automatic calculators can be used like a look-up table to answer analytical questions. Formulas can be copied and pasted and modified as needed. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 13 Using Databases: Summarizing CALCULATORS Denominator Function (with argument) = Result Result/Denominator = % Checking and Verifying Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 14 Function = COUNTING =COUNTA(D2:D298) Counts any occupied cell =COUNT(G2:G298) Counts any cell with a number in it (also in status bar) =COUNTIF(V2:V:298,1) Finds all the 1’s in Column V (1 is the code for new to YMCA) (Similarly, =COUNTIF(D2:D298,”F”) will count all cells in Column D containing the letter F – the code for Female). Please note, for alpha data, you MUST use full quotation marks around the code you are searching for. AND IT MUST BE EXACT. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 15 Basic Analyses Frequencies for all descriptive data (e.g., gender, race/ethnicity, age, living status, location, grade, program participation etc.) Calculating age from birth-date Re-coding data Cross-tabulating data Determining adjusted attendance rates Automatically comparing data to targets Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 16 Summarizing and Recoding: Capturing Attendance Data/Hitting Targets Attendance tracking is relatively straightforward. Rows = individuals eligible to attend Columns = all the possible dates for attendance Data = 1 if in attendance (0 if absent, blank if not expected) Add a calculator: Total number of sessions attended per person Total number of persons recorded per activity (Attendance Report – Signed Up) Total activity attendance for those enrolled Add a TOTALS WORKSHEET Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 17 Summarizing and Recoding: Capturing Attendance Data/Hitting Targets Recode attendance results to determine if targets have been met! Did individual participants attend the desired number of sessions? How many participants attended the desired number of sessions? You could ask many other questions such as who attended the desired number of sessions, who did not, etc., by any of your partitions of interest. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 18 Summarizing and Recoding: Adjusting Attendance Data/Hitting Targets If (logical _test,[value_if true], value_if false]) Target: Participants will attend at least 8 activities in a month. if (AA278>=8, “YES”,”NO”) IN ENGLISH: The total number of sessions attended in cell AA278 is compared to see if it is greater than or equal to 8. If so, the case gets a YES to signify meeting the target, if not the case gets a NO. Note that the total number of participants meeting the target is also summarized (AC298) Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 19 Multivariate Analysis: Crosstabs Like everything else in Excel, there is more than one way to conduct multivariate analyses – i.e., to look at more than one variable at a time. Pivot Tables Cheaters Cross-tabs Sort database Copy database – make as many copies as partitions* Delete those not in the partition, use the calculator as a look up for each partition. *A partition is a variable that divides the data into groups of interest. For example, RACE/ETHNICITY, SEX, AGE, INCOME LEVELS are all partitions. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 20 Other Handy Strategies Link formulas across sheets: =‘EXACTNAMEOFOTHERSHEET’!B3 or other cell you want ='2015 DATA'!B2 Calculating averages: *Use the click and drag function when you can OR * =AVERAGE(G2:G298) Be sure to decide what to do if your data has zeros in it. They may artificially lower the average. Try sorting, and setting your data range to include only cells with non-zero numbers. Bruner Foundation Rochester, New York Anita M. Baker, Evaluation Services 21