Multiple Indicator Cluster Surveys
Survey Design Workshop
Sampling:
Advanced Sampling
MICS Survey Design Workshop
Major steps in designing MICS sample
• Define objectives
– Key indicators
– Desired level of precision
– Sub-national domains of estimation
• Identify most appropriate sampling
frame
– Most recent census of population and housing
– Master sample or sample for another survey
conducted recently
Major steps in designing MICS sample
• Determine sample size and
allocation
–Determine availability of previous
MICS or DHS results to provide
measures of sampling parameters
Sampling Frame
• Sampling frame:
– Nationally-representative
– Complete coverage
– Measures of size (households or
population) for small area units
• Generally most recent census is the most
effective sampling frame
Sampling Frame
• In some cases more recent pre-census
listing may be available
• When no census is available, identify
most complete geographic frame
available (e.g. list of villages/localities
with estimated population)
Sampling Frame
• Common problems with area frames:
– Coverage issues
– Census maps of poor quality
– Errors and changes in area boundaries
– Inappropriate type and size of area
units
– Lack of auxiliary information
SAMPLE SIZE DETERMINATION
• n is the required sample size (number of
households)
• 4 is a factor to achieve the 95 percent level of
confidence
• r is the predicted or estimated value of the
indicator in target population
• deff is the design effect
• RR is the response rate
• pb is the proportion of the target
subpopulation in total population (upon
which the indicator, r, is based)
• AveSize is the average household size (that is,
average number of persons per household)
• e is the margin of error to be tolerated at the
95% level of confidence
• Currently, note that e = 0.12r [defined as 12%
of r, in this case the relative standard error of
r is 6% because e = 2 standard error (r)]
Previously in MICS2
• 2 different values for margin of error
– Margin of error was 5 percentage points for high
values of r (over 25%)
– Margin of error was 3 percentage points for low
values of r (25% or less)
• Difficulty for users in deciding on the sample
size for their surveys.
MICS template for sample size
calculation - EXCEL FILE
Selection of key indicators
• Choose an important indicator that will yield
the largest sample size
• Step 1: Select 2 or 3 target populations
representing each a small percentage of the
total population (pb); typically
– Children 12-23 months: 2-4% or
– Children under 5 years: 7%-20%
Selection of key indicators
• Step 2: Review important indicators for these
target groups but ignore indicators with very
low or very high prevalence (less 10% or over
40%, respectively)
• Do not choose from the desirably low
coverage indicators an indicator that is
already acceptably low
• Do no choose childhood and maternal
mortality ratios
Explicit Stratification
• Explicit stratification: dividing the sampling
frame into sub-groups (called strata) of
homogeneous (similar) PSUs.
• Advantages:
– Better precision because reduced variance
within stratum given similarity of units
– Flexible design, sub-national estimates for
smaller domains (differential sampling rates)
• Example of stratification: region, urban/rural
Implicit Stratification
• Sort the sampling frame according to certain
characters such as regions, urban-rural
residence, sub-regions, districts, etc., then
select a systematic pps sample.
• Ensures a representative sample for each
subgroup
• Automatically provides proportional allocation
by size of subgroup
Allocation of sample to strata/domains
• Proportional allocation
– Effective for precision of estimates at the national level
• Equal allocation to each domain
– Used when each domain requires same level of precision
• Optimum allocation – takes into account differential
variance and costs by stratum
– For example, variability may be higher in urban areas and
enumeration costs may be higher in rural areas – use
higher sampling rate for urban areas
Subnational estimates
• Number of separate areas (domains) for which
separate, equally reliable estimates are wanted
affects sample size
• For example, if 10 regional estimates are wanted,
theoretically the sample should be increased by
factor of 10
• As a compromise, larger sampling errors accepted
for subnational estimates
– One proposal (by Dr. Vijay Verma) – increase national
sample size by factor of D0.65, where D is the number of
domains
– Results in an average increase in the sampling errors for
domain estimates by a factor of about 1.5
Sampling Stages
• Ideal to have two-stage sample design, with
EAs defined as PSUs
• In some countries only frame of larger
administrative units available
– Three-stage sample design: larger area units
selected as PSUs
– Necessary to delineate smaller segments in each
sample PSU
Number of PSUs and Cluster Size
• Survey costs depend not only on number of
households but their distribution among
primary sampling units (PSUs)
• Important to determine effective balance
between number of sample PSUs and number
of sample households per cluster
• In general, the more PSUs the better for
reliability but the greater the cost (mostly
costs of travel and listing)
Number of PSUs and Cluster Size
• Example: 8000 households selected in 400
PSUs of 20 sample households each is a much
more reliable sample than 200 PSUs of 40
households each, but more expensive
• Number of sample households per cluster
should be as small as practical for reliability
• A range of 15-25 households for MICS appears
to be effective
Design Effect (DEFF)
• Deff - ratio of variance of estimate based on
stratified multi-stage sample design and
corresponding variance from simple random
sample of same size
• Measure of the relative efficiency of the
sample design
• Effective stratification reduces the deff
• Cluster sampling increases the deff
Design Effect (DEFF)
• In case of cluster sampling, deff generally measures
effect of clustering
_
deff  1   (m 1)
• δ = intraclass correlation coefficient, or measure of
homogeneity
within cluster
_
• m = average number of households per cluster
• Design effect increases with intraclass correlation
and cluster size
First Stage Selection of PSUs
• Standard methodology for MICS and other
household surveys – select EAs or clusters
systematically with PPS
• Important to sort frame before selection, in
order to ensure effective implicit stratification
• Traditional procedure – cumulate measures of
size, determine sampling interval and random
start, generate selection numbers
Large sample PSUs in PPS sampling
• Sometimes a PSU may have a measure of size larger
than the sampling interval
• PSU may be selected more than once in the
systematic PPS selection
• Option 1 – if the PSU is selected two or more times,
multiply the number of households to be selected by
the number of “hits”
• Option 2 – separate the large PSUs and include in
sample with a probability of 1
MICS Sampling Option 1 –
new sample with household listing
• Design new MICS sample
• Two stages with census as frame
• Use of implicit stratification, systematic selection
of census EAs at first stage with pps
• List households in selected EAs/segments
• Select households systematically from listing
• Interview selected households, no replacement will
be allowed
Sampling Option 1 - continued
• Advantages of option 2
- simple design
- probability-based
- if possible self-weighting (national level)
• Limitations of option 2
- expense of listing households
- time necessary to list households
[Example, sample size of 5000 households may require 25000 to
50000 households to be listed]
MICS Sampling Option 2 –
use an existing sample
• Design MICS as a rider to another survey if timely
and feasible
• Use sample from a previous survey and re-interview
households for MICS
• Or, use old survey sample EAs and construct new
listing of households to select for MICS
• Old sample must be probability-based, national in
scope
• Possibilities – DHS, other national health survey, recent
labour force survey
• Important: design parameters must be known (such as
selection probability, stratification, etc.)
Sampling option 2 - continued
• Use of existing master sampling frame
• Some countries use master sample design for
intercensal national household surveys
• Master samples generally sufficiently large for
MICS; subsample of PSUs can be selected
• Advantage – updated maps may be available
for master sample of PSUs, and perhaps
updated listing
Sampling option 2 - continued
• Advantages of using previous sample
- cost savings
- maps available for interviewers
- appropriate sampling plan available
- simplicity
• Limitations of using old sample
- burden on respondents
- sample design may need modification
* sample size
* sub-national coverage
* number of PSUs or clusters
• Balance between loss and gain
Listing and Selection of Households
• Household listing manual is available
• Importance of new listing to represent current
population
• Problems with using previous listing (older
than 1 year)
– Does not represent newer households
– Distribution of sample population by age group
distorted, generally with higher median age
– Difficulty of finding households in old list
Listing and Selection of Households
• MICS recommends a separate household
listing operation
– More reliable as listing staff are less likely than
interviewers to bias the sample by excluding
households that are difficult to reach
– Allows household selection to be done in a
single central location using reliable and
uniform procedures
Listing and Selection of Households
• Household selection in the office:
– Advantages – conducted by specialized staff,
possible to avoid selection bias in the field,
possible to control overall sample size
– Disadvantage – increased costs from having two
field visits
• Selection in the field: use household selection table
– Advantage – cost savings of having one integrated
field operation
– Disadvantage - correct sampling may be difficult
for field staff, selection may be biased
Listing and Selection of Households
• Excel template for generating automatically
the sample of households based on the
number of households listed(see spreadsheet)
• Common problems found in listing operations
– Problem with quality of sketch maps – difficult to
determine segment boundaries
– Sometimes large differences found between
number of households in frame (census) and
number listed.
Sampling strategy for low fertility
countries
• In MICS 4 and 5, some low fertility countries
are using second-stage stratification of listing
by households with and without children
under 5
• Higher sampling rate used for households
with children
• Increases number of households with children
in MICS sample, and therefore number of
sample children
Sampling strategy for low fertility
countries (continued)
• Improves the reliability of the child indicators
without increasing the sample size to a very high
level
• This procedure also increases the variability in the
weights and the design effects for the overall sample
• Important to avoid very large variability in the
weights for households with and without children
– Differential weights between households with and without
children generally should not exceed a factor of about 4
Implications of sampling strategy
on sample size calculations
• One parameter in the sample size calculation
template is the proportion of the indicator
subpopulation
• Using a higher sampling rate for households with
children increases the proportion of children under 5
in the sample
• The proportion of children under 5 (or smaller age
groups) should be multiplied by a factor that reflects
the increase in sample households with children
Implications of sampling strategy
on weighting procedures
• Under normal MICS sample design, weights
vary by sample cluster
• With second stage stratification by
households with and without children, two
weights need to be calculated for each
cluster: for households with and without
children
Survey weighting procedures
• Survey data collected using a complex design
featuring clustering, unequal probabilities of
selection and stratification:
– All analyses must apply survey weights in order to
prevent biased results
• Formulas for calculating weights depend on
the exact sample design used in each country
• MICS has 4 set of weights: households,
women, men and children
Survey weighting procedures
• Components of MICS survey weights:
– Design weight: inverse of the final probability of
selection for households
– Adjustment factors for nonresponse (cluster,
household, woman, child level)
• Normalized weights so that the total weighted
number of observations is equal to the total
unweighting number (sample size)
Survey weighting procedures
Sampling Error Estimation
•
•
•
•
•
Necessary to evaluate reliability of survey estimates
Possible only when probability sampling is used
Should be done for 30-50 important indicators
Methodology is complex and design-specific
Several software packages:
– SPSS Complex Samples module – used in MICS
– SAS, Stata, SUDAAN, Clusters, WesVar, CENVAR,
PCCarp, etc.
• Standard error, confidence intervals and DEFF
Sampling Error Estimation
SPSS Complex Samples module
• Advantages:
– Simple to use
– Template syntax available for standard indicators
– Supported by MICS Global and Regional staff
• Steps:
– Set up sampling parameter specifications file
(csplan)
– Define variables for stratum, PSU and weight
Sampling Error Estimation
SPSS Complex Samples module
• Stratum should be lowest level of explicit
stratification (for example, province,
urban/rural)
• Necessary to have minimum of two sample
PSUs per stratum
Reducing bias
• Accuracy of survey results depends on both variance and bias
(mostly from nonsampling errors)
• Bias should be minimized with quality control for all survey
operations
• Basic data quality determined during enumeration
– Important to have good training and supervision in the field
• Data capture should include 100% or sample verification
• Important to have quality control for editing and coding
procedures
• Computer consistency and range checks