Sampling for the Follow-Up Surveys Beginning in Program Year 2012, students are identified for NRS post-program outcomes according to their cohort membership. Cohorts are determined based on pre-defined characteristics rather than by goal setting. As a result, the number of learners to follow for each outcome is likely to increase in some states and programs. precision.) To help states make these estimates, a total survey sample size (ns) for each outcome has been set for states. The size is dependent on the number of students needed to produce accurate estimates about the population. A state sample of 400 to 1,100 students will be required, depending on the number of students in the cohort. Although some states may prefer to do data matching, this approach may not be possible for some states because of issues such as high proportions of students with missing social security numbers. Instead, states may need to conduct surveys to measure post-program outcomes. States using the survey method of follow-up data collection, especially those with large cohort sizes, may want to consider using sampling rather than collecting data from all students. This NRSTips describes the benefits of sampling, the procedures to follow when sampling, using sampling for state and national estimates, and how to determine cohort and sample sizes. Although not required, a state may also choose to use sampling to allow estimates of measure for local programs. When doing sampling at the local level, a substantially larger sample size maybe required than the sample size needed for a state or national estimate. The procedures to be followed to create the estimate, however, are the same as those listed above. Why Sample? The number of learners in each cohort may be much larger than the groups states previously had to follow for each post-program outcome. Sampling can be used to reduce the burden of this increased population size of cohorts because this approach allows you to collect data from a smaller, random sample that represents the total population. Based on data obtained from a sample, an estimate of the true number can be calculated, within a range of error. Sampling allows you to focus on quality of data collected rather than quantity. However, note that the response rate requirement for sampling is 70% rather than the data match requirement of 50%; this is meant to ensure that the sample is representative of the larger population. How are the Cohort and Sample Sizes Determined? Although there may be changes in population size from year to year, to estimate the size of your state’s cohort for this year, look at last year’s data. If you have the data available, you will be able to use prior years’ reported cohort sizes to make the estimates. The estimates for the cohorts should be calculated separately for each cohort: Entered employment: The number of unemployed in the labor force at entry who exited the program Retained employment: The number of employed students at entry and number of students in the entered employment cohort who obtain employment one quarter after exit Secondary credential attainment: The number of students taking GED tests, adult high school students in high ASE, and EDP assessment phase students who exited the program Entry into postsecondary education or training: The number of students with a secondary Using Sampling for Estimates Sampling allows for state and national estimates to be computed accurate enough to detect a difference of 5%. (States can sample more if they need greater credential and/or in transition classes who exited the program The sample size you need is based on the eligible number of students in the cohort. (See table below.) The sample may be greater, equal or nearly equal to the entire cohort. Keep in mind: When you have a very small cohort, rather than sampling, you will need to survey everyone in cohort. When you have a very large cohort, you can obtain a representative sample with a fairly small sample relative to the population size. A good response rate is more important than a very large sample. Number of Eligible Learners in Cohort Number in Sample 1 to 200 All 201 to 400 All 401 to 600 400 601 to 1,000 500 1,001 to 1,200 550 1,201 to 1,400 630 1,401 to 1,600 660 1,601 to 1,800 715 15,001 to 16,000 1,030 16,001 to 25,000 1,035 25,001 to 35,000 1,065 35,001 to 40,000 1,070 40,001 to 100,000 1,085 100,001 and up 1,100 How Do I Sample? To do the sampling, states should perform the following steps: 1. Determine the total number of students in the cohort for each outcome measure for which you will be sampling. Remember that to be a member of a cohort, a student must have exited the program. For example, the entered employment cohort includes the unemployed in the labor force who exit, not simply the unemployed. 2. Determine the overall sample size required for each state cohort, based on the overall cohort size, using the table. 3. Calculate the proportion of students in each state cohort that come from each local program. For example, if there were 1000 students total in the state’s entered employment cohort, and Program A had 100 students in the cohort, then Program A represents 10% of the total cohort. 4. Calculate each program’s sample size, based on steps 2 and 3 above. For example, if there were 1000 total in the state’s entered employment cohort, the state sample size for that cohort should be 500. If Program A represents 10% of the overall state cohort, they should represent 10% of the sample as well. Therefore, the sample size for Program A should be 10% of 500 (500 x .10), or 50. 5. Randomly sample the required number of eligible students from each program. In conducting the survey, states can distribute the student sample list to each program and have local programs conduct the survey OR the state can both sample and conduct the survey. Additional Resources For more information on the new employment reporting or for answers to questions, visit http://www.nrsweb.org or e-mail: NRS@air.org. Local program staff should first consult their state office for their state policy. NRS Tips is a quick reference tool for state staff, program directors, and adult education teachers. NRS Tips are written and produced by the staff at the American Institutes for Research (AIR), a nonprofit research and policy organization (see http://www.air.org), under contract with the Division of Adult Education and Literacy (DAEL) of the U.S. Department of Education.