THE RISK OF RANDOM ERROR
(PLAY OF CHANCE)
Goran Poropat
Introduction
No physical quantity can be measured with
perfect certainty
All measurements are prone to errors
Experimental errors
Do not refer to mistakes, blunders, or
miscalculations
(eg. measuring a width when the lenght should
have been measured)
Inherent in the measurement process
Experimental errors
Are measured by
a) Accuracy – how close a measured value is to
the true value or accepted value
b) Precision – how closely two or more
measurements agree with each other
(repeatability, reproducibility)
Experimental errors
Three dimensions particularly influence the reliability
of our observations in clinical research:
1. RANDOM ERRORS (PLAY OF CHANCE)
2. SYSTEMATIC ERRORS (BIAS)
3. DESIGN ERRORS
Bias
A systematic error – deviation from the truth, in
results or inferences
Overestimation or underestimation of the true
intervention effect
Bias
Affect accuracy of a measurement
Should not be confused with
imprecision
Multilpe replications of the same study –
wrong answer on average
Random error
Imprecision – refers to random error
The unpredictable variation between observed
values and some “true” value
Possible reason of misleading results in RCTs and
meta-analyses
Random error
Affects precision of measurement
Multilpe replications of
the same study
Different effect
estimates
SAMPLING VARIABILITY
Sampling variability
The actual study result will vary depending on
who is actually in the study sample
A sample – a subset of a population of
manageable size
Epidemiological studies
Impossible to evaluate every member of the
entire population
The relationship between exposure and healthrelated event is judged from observations on a
SAMPLE of the population
STATISTICAL INFERENCES
(extrapolations)
Sampling variability
50 x
x 50
N=4
2x
2x
x4
Sampling variability
Different inferences – various possible samples
Hypothesis
Information size
Probability of drawing a bad sample
Random errors tend to decrease as
information size increases
Sampling variability
RR
Clinically important
overestimate!
q
1
Number of patients randomised
Required
sample size
Epidemiological studies
The measure of association we observe
(inference) in our data may differ from the
“true” measure of association
- by chance alone
Probability that the observed difference is due
to play of chance
Hypothesis testing
Quantify the degree to which sampling
variability (chance) can explain the observed
association
Assume H0 is true, and not Ha
Assessing H0
P-value
The probability of obtaining an observed effect
(or larger) under a null-hypothesis
P-value
The likelihood of observing certain data given
that the null-hypothesis is true
P-value threshold = 0.05 – arbitrary
Data yielding a P-value = 0.05 – a 5% chance
obtaining the observed result, if no real effect
exists
P-value
A P-value is the probability of an observed
(or more extreme) result arising by chance
Misinterpretations
P>0.05
“the intervention has no effect”
“not strong evidence that the intervention has an effect”
P<0.05
“ an intervention has a strong benefit”
The P value addresses the question of whether the
intervention effect is precisely nil
Confidence intervals
Another approach to quantify sampling
variability
Range within which the true magnitude of effect
lies with a stated probability, or a certain
degree of assurance (usually 95%)
Confidence intervals
Point estimate – the actual measure of
association given by the data (OR, RR, RD)
The best guess of the magnitude and direction
of the experimental intervention’s effect
compared with the control intervention
Confidence intervals
Wider intervals – greater imprecision
CI width
• Sample size
• Precision of individual study estimates
• Number of studies combined
Confidence intervals
P-value – to which extent the null-hypothesis is
compatible with the data
CI – the range of hypothesis compatible with the
data
Sum up
• Random error (due to ’play of chance’) is
the unpredictable variation between
observed values and some ’true’ value
• Everything we attempt to estimate may be
subject to some degree of random error
Sum up
• Random error affects
• Statistical significance
• Estimated treatment effects
• Heterogeneity estimates
• Only a sufficient number of trials and
patients will ensure an acceptable risk
of random error
Thank you for your
attention!