2

Preface ....................................................................................................................................................... 4
Introduction ........................................................................................................................................... 5
Guidelines for Weighting and Variance Estimation ................................................... 6
Weighting Steps for Population Weights .......................................................................... 7
Jackknife replicate weights ..................................................................................................... 7
Annex: Excerpt from 'Introduction to Variance Estimation, Kirk M. Wolter
........................................................................................................................................................................ 9
3

The Weighting Guide builds upon a project “Common Framework for a Literacy
Survey” which was executed by the Caribbean Community (CARICOM)
Secretariat under funding provided by the Inter-American Development Bank
(IDB) Regional Public Goods Facility. The main aim of the project was intended to design a common approach to the measurement of literacy in countries. This common framework is built upon international methodologies and fundamentally the International Survey of Reading Skills (ISRS) that enable reliable measurement of literacy than what presently exists in the Region.
The literacy Survey is designed to measure functional literacy. In other words, it will determine an individual’s literacy level by employing a series of questions designed to demonstrate the use of their literacy skills. This involves two steps – the objective testing of an adult’s skill level and the application of a proficiency standard that defines the level of mastery achieved. The assessment measures the proficiency of respondents on three continuous literacy scales - prose, document and numeracy. In addition, it will collect information on reading component skills. Component skills are thought to be the building blocks upon which the emergence of reading fluency is based. Information on the reading component skills will be collected from people at the lower end of the literacy scale only. The testing phase is preceded by a selection phase, which includes the administering of a Background or Household questionnaire and post the selection of the respondent from the specific household an initial preassessment is undertaken through a filter test booklet to determine what type of assessment should be undertaken in the testing phase.
A consultant, Mr. Scott Murray of Canada was hired to undertake the provision of services on this project. The CARICOM Secretariat (including Regional
Statistics and Human and Social Development Directorate) and the CARICOM
Advisory Group on Statistics (AGS) were instrumental in the execution of the project throughout all phases. In addition, there was participation by Member
States and some Associate Members relative to the technical rollout of the instruments and documents.
<< The paragraph that follows can be country-specific>>
This Weighting Guide is aimed at providing <<country undertaking a Literacy
Survey>> with guidelines that can enable the weighting of the literacy survey data as recommended under the IDB-funded CARICOM project.
4

Literacy Assessments that are designed to yield reliable and representative population-level results demand the selection of probability sample of the adult population and the collection of background information and test data from as high a percentage as possible of selected respondents.
Experience with other Literacy Assessments reveals several types of nonresponse as follows:
1. Selected households that are vacant or where no contact can be established
2. Selected households in which there are no eligible adults aged
16 and over
3. Selected households where there are eligible adults but the selected respondent refuses to answer the background questionnaire or who fail to provide key variables. These records are dropped from the response file.
4. Selected households where there are eligible adults, where the selected respondent completes enough of the background questionnaire but who refuses to be tested for non-literacy related reasons. These records are retained and proficiency scores imputed using a multivariate model.
As a first step, design weights are calculated that represent the inverse probability of selection for each strata in the design. Since the design will vary from country to country, this step must be undertaken for each country separately.
As a second step for each country, the design weights within each strata are adjusted to compensate for the loss of households for the first three reasons listed above. In this case, the inverse of the probability of selection is adjusted upward to compensate for the loss of households. At a minimum, respondents must provide age, gender, education, immigrant status and mother tongue to be retained as a respondent.
Adjusted weights are further adjusted to reflect benchmark totals within each strata e.g. age group, gender.
5

The Literacy Survey uses a probability sample design. As such, the estimates produced from the survey should be weighted estimates.
Countries are responsible for determining the survey weights and including them in the survey file. The weighting procedures are to be carried out using the 'clean' data file, i.e., the file resulting from the editing process. The survey weights will be appended to each respondent record on the clean survey file. The following weights will comprise part of each sample file record: a) Theoretical or sample design weight - the inverse of the probability of selection at the sample selection stage. b) Non-response adjusted sample weight - based on the sample design weight and adjusted for non-response. c) Benchmark weight - the weight resulting from the adjustment of the survey results to known population totals. d) Jackknife replicate weights - there should be between 10 and 30 of these weights that are used to calculate the standard error of the survey estimates.
In general, two types of weights need to be calculated,
1.
Population weights-
The population weights are used in the production of survey estimates, e.g., percentages, averages. Each respondent record should contain three population weights namely, theoretical or sample design weight, nonresponse adjusted sample weight, and benchmark weight.
2.
Jackknife replicate weights
The jackknife replicate weights are used to calculate the standard error of the survey estimates. For each respondent record, there should be from 10 to 30 such replicate weights. Note that countries are not required to produce jackknife variance estimates. A technical consultant should be recruited to handle this aspect of estimation once population weights and proficiency estimates are available.
6

The population weights are calculated according to the sample design used. For example, if a stratified sample design was used then the population weighting should be carried out independently within each stratum according to the method of sample selection.
These population weights are determined in three steps as follows:
Step 1: Calculate the theoretical sample weights according to the sample design.
For each sampled record, the theoretical sample weight is the inverse of the probability of selection. In other words, the theoretical sample weight is calculated by dividing the pertinent population size by the corresponding initial sample size.
Step2: Calculate the non-response adjusted weight according to the sample design.
For each respondent record, multiply the 'theoretical sample weight' by a
'non-response adjustment factor' to arrive at the 'non-response adjusted weight'. The 'non-response adjustment factor' is the result of dividing the pertinent initial sample size by the corresponding final sample size.
Note: Since the data file will contain a record for each selected person, there will be respondent records and non-respondent records on the data file. At this step of the weighting process, the ‘non-response adjusted weight’ should be set to zero for the non-respondent records.
Step 3: Calculate the benchmark weights.
The weights from step 2 should be adjusted to be consistent with known population totals, e.g., census counts. As a minimum, the weights should be adjusted to be consistent with known population totals for age and gender. In addition, the weights from Step 2 should be adjusted to known population totals for education and labour force status, provided a reliable source of population counts can be obtained for these characteristics.
Once the population weights have been calculated and included on the survey file, then the jackknife replicate weights should be calculated and appended to the survey file. The steps for calculation of the jackknife
7

replicate weights are as follows:
1: Form the replicates.
There should be between 10 and 30 replicates (more replicates are better). Each replicate is a random group of records formed with the same sample design as the main sample. The choice of the number of replicates depends on computing cost constraints and the precision of the estimates - one must try to find a balance between the two. On the one hand, the more replicates (up to a threshold), the more accurate the estimates, while on the other hand, one would prefer as few groups as possible for computational reasons. It is not necessary to have exactly the same number of units in each replicate, but it is recommended that all replicates be similar in size. Kirk M. Wolter provides a good outline of the rules for forming jackknife replicates in the monograph "Introduction to Variance Estimation". The relevant excerpt from this monograph can be found at the end of this appendix. (Note that a random group is a replicate.)
2. Calculate the replicate weights
The procedure for producing the replicate weights consists of removing each of the R replicates, in turn, from the weighting process and then weighting the remaining replicates. The records belonging to the removed replicate should be assigned a replicate weight of 0. Each sample record will contain R replicate weights where R is the total number of replicates
(i.e., R is between 10 and 30). The following procedure must be repeated
R times in order to calculate the jackknife replicate weights. i.
Remove one replicate, in turn, from the weighting process, and set the replicate weight to zero for the records in this replicate. ii.
Multiply the theoretical weight of all remaining replicate records by
R/R-1 where R is the total number of replicates (i.e., R is between 10 and 30). iii.
Repeat steps 2 & 3 (i.e., population weighting steps) for the remaining replicate records. iv.
Repeat i to iii until each replicate, in turn, has been removed from the weighting process.
8

"To ensure that the random group estimator of variance has acceptable statistical properties, the random groups must not be formed in a purely arbitrary fashion. Rather, the principal requirement is that they be formed so that each random group has essentially the same sampling design as the parent sample. This requirement can be satisfied for most survey designs by adhering to the following rules:
(i) If a single-stage sample of size n is selected by either srs wor (simple random sample without replacement) or pps wor (probability proportional to size without replacement), then the random groups should be formed by dividing the parent sample at random. This means that the first random group (RG) is obtained by drawing a simple random sample without replacement (srs wor) of size m = [n/k] from the parent sample; the second RG by drawing a srs wor of size m from the remaining n - m units in the parent sample; and so forth. If n/k is not an integer, i.e., n = km + qwith0 < q < k, then the q excess units may be left out of the k random groups. An alternative method of handling excess units is to add one of the units to each of the first q RGs.
(ii) If a single-start systematic sample of size n is selected with either equal or unequal probabilities, then the random groups should be formed by dividing the parent sample systematically. This may be accomplished by generating a random integer between 1 and k, say <*.
The first unit in the parent sample is then assigned to random group <*, the second to random group <* + 1, and so forth in a modulo k fashion.
Variance estimation for systematic sampling is discussed more fully in
Chapter 7 [of the aforementioned monograph].
(iii) In multistage sampling, the random groups should be formed by dividing the ultimate clusters, i.e., the aggregate of all elementary units selected from the same primary sampling unit (PSU), into k groups. That is, all second, third, and successive stage units selected from the PSU must be treated as a single unit when forming RGs. The actual division of the ultimate clusters into random groups is made according to either rule (i) or (ii), depending upon the nature of the first-stage sampling design. If this design is srs wor or pps wor, then rule (i) should be used, whereas for systematic sampling designs rule (ii) should be used.
Particular care is required when applying the ultimate cluster principal to so-called self-representing PSUs, where terminology may cause considerable confusion 1 . From the point of view of variance estimation, a
1 A self-representing PSU is a PSU selected with probability one.
9

self- representing PSU should be considered a separate stratum, and the units used at the first stage of subsampling are the basis for the formation of random groups.
(iv) In stratified sampling, two options are available. First, if we wish to estimate the variance within a given stratum, then we invoke either rule
(i), (ii), or (iii) depending upon the nature of the within stratum sampling design. For example, rule (iii) is employed if a multistage design is used within the stratum. Second, if we wish to estimate the total variance across all strata then each random group must itself be a stratified sample comprised of units from each stratum. In this case, the first RG is obtained by drawing a srs wor of size Mh = nh/k from the parent sample nh in the h-th stratum, for h = 1, . . ., L. The second RG is obtained in the same fashion by selecting from the remaining nh - Mh units in the h-th stratum. The remaining RGs are formed in like manner.
If excess observations remain in any of the strata, i.e., nh = kMh + qh, they may be left out of the k random groups or added, one each, to the first qh RGs. If the parent sample is selected systematically within strata, then the random groups must also be formed in a systematic fashion.
In other words, each random group must be comprised of a systematic subsample from the parent sample in each stratum.
(v) If an estimator is to be constructed for some double sampling scheme, such as double sampling for stratification or double sampling for the ratio estimator (see Cochran (1977, Chapter 12)), then the n' sampling units selected into the initial sample should be divided into the k random groups. The division should be made randomly for srs wor and pps wor designs and systematically for systematic sampling designs. When n' is not an integer multiple of k, either of the procedures given in rule (i) for dealing with excess units may be used. The second phase sample, say of size n, is divided into random groups according to the division of the initial sample. In other words, a given selected unit i is assigned the same random group number as it was assigned in the initial sample.
This procedure is used when both initial and second phase samples are selected in advance of the formation of random groups. Alternatively, in some applications it may be possible to form the random groups after selection of the initial sample but before selection of the second phase sample. In this case, the sample n' is divided into the k random groups and the second phase sample is obtained by independently drawing m = n/k units from each random group.
These rules, or combinations thereof, should cover many of the popular sampling designs used in modern, large-scale sample surveys. The rules
10

will, of course, have to be used in combination with one another in many situations. An illustration is where a multistage sample is selected within each of L >=2 strata. For this case, rules (iii) and (iv) must be used in combination. The ultimate clusters are the basis for the formation of random groups, and each random group is composed of ultimate clusters from each stratum."
11