Collecting and Reporting Follow-up Data Or What’s Behind Table 5? American Institutes for Research February 2005 Table 5 and Follow-up Table 5: Outcomes 2 Earned a High School Diploma or GED Entered post-secondary School Entered Employment Retained Employment Table 5 and Follow-up Table 5: Outcomes 3 States submit six (6) data points for each of the four outcomes: 1 2 3 4 5 6 Core Followup Outcome Measures # Participants With Main or Secondary Goal (A) (B) # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (C) (D) (E) (F) (G) Table 5 and Follow-up Table 5: Column (B) Core Followup Outcome Measures # Participants With Main or Secondary Goal (A) (B) # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (C) (D) (E) (F) (G) All participants who had the outcome as a goal and who exited during the program year. 4 Table 5 and Follow-up Table 5: Column (C) Core Followup Outcome Measures # Participants With Main or Secondary Goal (A) (B) # Participant s Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (C) (D) (E) (F) (G) The number of participants who were surveyed to assess whether they had achieved the outcome. Note: C should be equal to B unless sampling was used, in which case C would be less than B. For data matching C = [BLANK] 5 Table 5 and Follow-up Table 5: Column (D) Core Followup Outcome Measures # Participants With Main or Secondary Goal (A) (B) # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (C) (D) (E) (F) (G) Of those surveyed or data matched, the number who responded or were used for data match. Note: D should be equal to or less than B and C each. 6 Table 5 and Follow-up Table 5: Column (E) Core Followup Outcome Measures # Participants With Main or Secondary Goal (A) (B) # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (C) (D) (E) (F) (G) Percent of respondents or participants successfully matched; column D divided by column B. E = D/B * 100 Should be > 50%, but likely not 100% 7 Table 5 and Follow-up Table 5: Column (F) Core Followup Outcome Measures # Participants With Main or Secondary Goal (A) (B) # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (C) (D) (E) (F) (G) Number of participants achieving outcome. Note: F must be equal to or less than D. 8 Table 5 and Follow-up Table 5: Column (G) Core Followup Outcome Measures # Participants With Main or Secondary Goal (A) (B) # Participants Included in Survey # Participants Responding to Survey or Used for Data Matching Response Rate or % Available for Match # Participants Achieving Outcome % Achieving Outcome (C) (D) (E) (F) (G) Percent achieving outcome. G= F/D * 100 9 Table 5 and Follow-up Table 5: Example # Participants # Responding Participants # to Survey or Response # Core Follow- With Main or Participants Used for Rate or % Participants % up Outcome Secondary Included in Data Available Achieving Achieving Measures Goal Survey Matching for Match Outcome Outcome 10 (A) (B) (C) (D) (E) (F) (G) Entered Employment 1,000 1,000 800 80% 200 25% Table 5 and Follow-up Real Data – PY 2002–03 Examples State # Participants # Responding Participants # to Survey or Response # With Main or Participants Used for Rate or % Participants % Secondary Included in Data Available Achieving Achieving Goal Survey Matching for Match Outcome Outcome (A) (B) (C) (D) (E) (F) (G) State 1 1,056 0 972 0% 356 37% What type of follow-up was used? 11 Table 5 and Follow-up Real Data – PY 2002–03 Examples State # Participants # Responding Participants # to Survey or Response # With Main or Participants Used for Rate or % Participants % Secondary Included in Data Available Achieving Achieving Goal Survey Matching for Match Outcome Outcome (A) (B) (C) (D) (E) (F) (G) State 1 1,056 0 972 0% 356 37% Why is this number less than column B? 12 Table 5 and Follow-up Real Data – PY 2002–03 Examples State # Participants # Responding Participants # to Survey or Response # With Main or Participants Used for Rate or % Participants % Secondary Included in Data Available Achieving Achieving Goal Survey Matching for Match Outcome Outcome (A) (B) (C) (D) (E) (F) (G) State 1 1,056 0 972 0% 356 37% What data should be reported here? 13 Table 5 and Follow-up Real Data – PY 2002–03 Examples State # Participants # Responding Participants # to Survey or Response # With Main or Participants Used for Rate or % Participants % Secondary Included in Data Available Achieving Achieving Goal Survey Matching for Match Outcome Outcome (A) (B) (C) (D) (E) (F) (G) State 1 1,056 0 972 0% 356 37% Is this percent computed correctly? 14 Table 5 and Follow-up Exercise Questions For states 2–6: 15 What method of follow-up did each state use? Which states, if any, filled in the table correctly? Which did not? Do you see any errors in the table? Do you have confidence in these data? Why or why not? Table 5 and Follow-up Exercise Example State 16 # Participants # Responding Participants # to Survey or Response # With Main or Participants Used for Rate or % Participants % Secondary Included in Data Available Achieving Achieving Goal Survey Matching for Match Outcome Outcome (A) (B) (C) (D) (E) (F) (G) State 1 1,056 0 972 0% 356 37% State 2 3,325 NOT AVAIL. 0% 1,062 32% State 3 8,644 0 0 0% 4,067 47% State 4 31,363 0 31,363 100% 21,779 69% State 5 9,633 3,853 1,156 30% 543 13% State 6 224 224 164 73% 96 59% Table 5 and Follow-up Top Errors for Table 5 Missing Data Unreasonably high response rates 17 Put “0” for # surveyed in Column (C) Put “0” for # responded or matched in Column (D) Many 100% Unreasonably high data matching Many 100% (DOL goal for matching = 90%) Unreasonable matching of # who had goal to # who responded Column (D) = Column (B) Table 5 and Follow-up Top Errors for Table 5 (cont’d.) Incorrectly computed percentages: Response rates calculated as D/C instead of D/B (B and C should be the same unless using random sampling, however) 18 Number achieving outcome computed as F/B instead of F/D Table 5 and Follow-up 19 Table 5 and Follow-up 2002–03 Data Collection Methods Data Matching Methodology Produces better quality data: 20 More student coverage Better data validity Better response rate Matches Title I—better for OMB common measures 28 states use it for employment measures—we need more states! What are the reasons for not data matching? How can we promote use of this methodology? Table 5 and Follow-up What’s Behind the Data? State responsibility to ensure accuracy and quality of follow-up measures Requires monitoring of local program procedures Key steps: 21 Identify students—goal setting and tracking Uniform data collection Training Reporting Table 5 and Follow-up Percent of Students with Goals Measure # with Goal Total # Students % with Goal Entered Employment—Average ALL states 6,904 52,548 10%* State 1 417 69,284 1% State 2 2,624 138,184 2% State 3 1,609 70,893 2% 2,919 52,548 6% State 4 159 33,062 0% State 5 333 31,574 1% State 6 14,523 565,311 3% 7,207 52,548 20%** State 7 7,192 114,008 6% State 8 2,768 32,492 9% State 9 33,118 387,710 9% Entered post-secondary—Average ALL states High School Completion—Average ALL states * 38% Unemployed (Table 6) ** 18% Enrolled in ASE 22 Table 5 and Follow-up Quality Control: Surveys Identify students Data Collection 23 Set student goals appropriately By exit quarter for employment measures Quarterly collection for employment State survey instrument Sufficient resources (staff, time, funds) Sampling method (if applicable) Table 5 and Follow-up Quality Control: Surveys (cont’d.) Training Reporting 24 Trained staff on survey procedures Improving response rate Quarterly reporting Database to link outcomes to programs and students Table 5 and Follow-up Quality Control: Data Matching Procedure for collecting and validating Social Security numbers 25 Report all with goal, prior to eliminating invalid or missing numbers (Table 5, Col. D) Data have exit quarter and employment records matched after correct exit quarter Only students with goal matched and/or reported—should not have 100% with outcome Table 5 and Follow-up Discussion: Issues and Strategies 26 Improving reporting and collecting Resources Training on Monitoring Table 5 and Follow-up