2. Key Vocabulary & Definitions
Make sure you know these terms and can apply them in context:
Term
Definition / explanation
Population
The entire group of individuals or items that we are interested in
studying.
Sample
A subset of the population, selected for study.
Parameter
A numerical summary of a population (e.g., population mean,
population proportion).
Statistic
A numerical summary of a sample (e.g., sample mean, sample
proportion).
Sampling frame
The list or mechanism used to define who/what is eligible to be
sampled.
Simple random
sample (SRS)
A sample in which every possible subset of the population of a
given size has the same chance of being selected.
Stratified random
sample
The population is divided into strata (groups) and random samples
taken from each stratum.
Cluster sample
The population is divided into clusters (often naturally occurring
groups) and a sample of clusters is selected, then all (or a random
sample) of elements within those clusters are studied.
Systematic sample
Every kᵗʰ unit is selected from a list or sequence, after a random
start.
Convenience sample
Selecting what is easiest or most convenient to sample — often
leads to biased results.
Voluntary response
sample
A sample in which respondents choose themselves (e.g., call-in
polls) — very susceptible to bias.
Bias
A systematic error that tends to favour certain outcomes or
responses and misrepresent the population.
Observational study
A study in which the researcher observes and records information
but does not attempt to influence responses.
Experiment
A study in which the researcher imposes a treatment (or treatments)
on units and observes the effect.
Treatment
The condition applied to units in an experiment.
Explanatory
(independent)
variable
The variable manipulated (or considered as cause) in an experiment
or study.
Response
(dependent) variable
The outcome variable measured in a study or experiment.
Randomization
Assigning units to treatments by chance to reduce bias and ensure
comparability.
Control group /
baseline
A group in an experiment that receives no treatment (or standard
treatment) for comparison.
Blocking
Dividing experimental units into blocks based on some variable so
that units within a block are more similar; then randomize within
blocks.
Matched pairs design
A form of blocking where pairs of units are matched (or the same
unit receives both treatments at different times) and then
randomized or compared.
Confounding variable
A variable that is related both to the explanatory and response
variable, which may hide or distort the true relationship.
Generalizability
The extent to which results from a sample can be extended to the
population.
Causation vs.
Association
A cause-effect relationship (causation) only can be inferred under
appropriate experimental design; observational studies can typically
only show association.
3. Types of Data Collection / Study Design
3.1 Sampling Surveys
●
Goal: estimate population parameters using a sample.
●
Good design: use probability-based sampling (SRS, stratified, cluster) so each unit has a
known chance of selection.
●
Beware bias: Examples include undercoverage (some units have zero or low chance of
selection), nonresponse (selected units don’t respond), question wording bias, response
bias (respondents give inaccurate answers).
●
Examples of bad design: convenience sampling, voluntary response.
●
Survey question: “Does the sample represent the population of interest?” and “What bias
might exist?”
●
Sampling error vs non-sampling error: sampling error comes from the fact that you use a
sample instead of the full population (and can quantify this later); non-sampling errors
come from design or execution mistakes (bias) that you cannot simply reduce by sample
size.
3.2 Experiments
●
Goal: to assess cause-and-effect by manipulating treatments and controlling other
factors.
●
Key features of a well-designed experiment: random assignment of units to treatments,
control group (or standard treatment), replication (sufficient units per treatment), control
of confounding variables, blocking if needed.
●
Types of design:
○
Completely randomized design: all units assigned at random to treatments.
○
Randomized block design: units sorted into blocks (by some characteristic) then
randomized within each block.
○
Matched pairs design: special blocking of size 2; either two similar units each get
a different treatment, or the same unit receives both treatments in random order.
●
Observational vs experimental: Observational studies cannot reliably show causation
because of confounding; experiments (with random assignment) are needed to claim
causation (given no other major flaws).
●
When writing about an experiment: you should comment on whether the design supports
causation, whether randomization was used, what might limit generalizability (external
validity).
4. Planning a Study & Considerations
When planning a survey or experiment, you should think through the following steps/questions:
1. What is the population of interest? Define clearly.
2. What is the variable(s) of interest? Are they categorical or quantitative?
3. What is the sampling method / assignment method? How will you select sample or
assign treatments?
4. What is the sampling frame? Does it cover the population appropriately, or are any
units excluded (leading to undercoverage)?
5. What could cause bias? Non-response, question wording, convenience sampling,
voluntary responses, etc.
6. For experiments: What are the treatments? What is the control? Who are the
experimental units? How are they assigned? Are blocks/matching used? What
confounding variables might exist?
7. Will you generalize? To what population? Under what conditions?
8. Will you infer causation? Only if experiment has random assignment and other good
features; otherwise, you can only claim association.
9. What are ethical considerations / practical constraints? For example, human
subjects, cost, ability to blind, etc.
10. How will you collect the data? How will responses be measured; how will nonresponse or missing data be handled?
11. How will you analyse / infer later? Though data collection is the unit now, it connects
to the later units on inference, so keep good notes.
5. Common Sources of Bias & Mistakes
●
Undercoverage: some portion of the population is excluded from the sampling frame.
●
Nonresponse bias: selected units do not respond, and those who don’t respond differ
systematically from those who do.
●
Response bias: respondents give inaccurate answers (due to wording, memory, social
desirability, etc.).
●
Voluntary response bias: participants self-select (often only those with strong opinions
respond).
●
Convenience sampling: sample drawn from easiest accessible units but not random —
tends to misrepresent population.
●
Improper randomization: units not randomly assigned in an experiment → confounding.
●
Confounding variables: uncontrolled variables that affect both explanatory and
response variables, weakening causation claims.
●
Generalizability error: taking results from one sample and generalizing to a population
not properly covered or different from sample.
●
Mis-interpreting association as causation: especially in observational studies.
●
Improper treatment/control design: missing control group, insufficient replication, lack
of blinding when needed.
6. Sampling Methods Summarised
Method
Description
When/why to use
Potential pitfalls
Simple
Random
Sample (SRS)
Every possible sample of a When you can list all
given size has an equal
units and randomly
chance.
select.
May miss sub-groups
(unless sample size
large).
Stratified
Random
Sample
Divide population into
strata (e.g., male/female),
then take SRS from each.
When you expect
differences in strata
and want better
precision.
Must identify
appropriate strata; risk
of mis-stratification.
Cluster Sample Divide into clusters (e.g.,
schools), randomly select
clusters, then all (or some)
units within.
When listing all
individuals is hard,
but clusters are easy.
More sampling error
than SRS; clusters
may be alike (less
diversity).
Systematic
Sample
From a list, pick every kᵗʰ
unit after a random start.
When you have a list
and can sample in
order.
If list has pattern, may
bias results.
Convenience
Sample
Use available/ easy units.
Quick, practical, but
weak.
High risk of bias;
results rarely
generalizable.
Voluntary
Response
Sample
Participants choose
themselves.
Rarely legitimate for
generalizable
conclusions.
Very high risk of bias;
usually only strongopinion respondents
answer.
7. Experimental Design Types & Considerations
●
Completely Randomized Design: All units randomized to treatments.
●
Randomized Block Design: Units grouped into blocks (by a factor) and then within
each block randomized to treatments. Helps reduce variability.
●
Matched Pairs Design: Special block of size 2; either two similar units or same unit
takes both treatments in random order. Ideal when you want to reduce variability from
differences between units.
●
Steps to evaluate:
○
Are units randomly assigned?
○
Are treatments applied properly?
○
Is there a control group or baseline?
○
Is blocking/matching used when needed?
○
Are there lurking/confounding variables?
○
Is replication sufficient (enough units per treatment)?
○
Are results generalizable to a broader population?
●
Blinding / Double-blinding: When subjects or experimenters (or both) don’t know
which treatment is assigned, to reduce bias.
●
Placebo effect: In experiments with humans, those in the control may show
improvement just by belief.
●
Interpretation: If well-designed, you can infer causation. If not, you may only describe
association.
8. Writing & Interpreting Study Descriptions
When you see a study on the exam (or in class), you will likely be asked to:
●
Identify the population and sample (or experimental units).
●
Identify the sampling or assignment method.
●
Identify any bias or limitation in the design.
●
Decide whether the results can be generalized.
●
Decide whether the study supports causation or only association.
●
Suggest improvements (e.g., better randomization, blocking, reduce nonresponse, etc.).
●
Interpret findings in context: “We estimate that…” or “We conclude that…”, not just “It
shows…”.
9. Using Studies Wisely
●
●
●
Sampling Variability: Different random samples of the same size from the same
population produce different estimates.
Statistically Significant: When the observed results of a study are too unusual to be
explained by chance alone, the results are called statistically significant.
Scope of Inference:
○ Random selection of individuals allows inference about the population from which
the individuals were chosen
○ Random assignment of individuals to groups allows inference about cause and
effect.