Tossing the icosahedron (20-sided dice) and false-positive results.
A p-value of 0.05 or less is often taken as evidence of a "significant" or "true" treatment effect.
This conclusion of a positive treatment effect can be false. Under the null hypothesis that the
investigated treatments are identical, the probability for a false-positive " significant effect" is
typically set at 5% or less. In other words, one typically accepts in the design and conduct of
randomized controlled trials a 5% chance rate of false positive conclusions (a false-positive
conclusion meaning here that treatments truly are identical, but that by chance a positive or
significant effect was observed).
It may intuitively be apparent that the probability of finding at least one false-positive finding
goes up by conducting multiple RCT on the same hypothesis. This situation is analogous to
repeatedly throwing a dice until you see the number you desire. An icosahedron or a 20-sided
dice is a good die to illustrate this as the chance for throwing any specific number on the 20sided dice is also 5%.
The probability for obtaining a false-positive result in a clinical trial with a type I error rate of 5%
is the same as throwing any specific number on an icosahedron. Let's take number 7 on a 20sided dice as an example. The chance of throwing a 7 is 5% just like the chance of obtained a
false-positive results in a clinical trial with identical treatment is 5% if the type I error rate is set
at 5%.
If one throws the dice once, the chance of observing a "7" (for instance) is 5%.
If one throws the dice 5 times, the chance of observing a "7" is 1 - (0.95)**5 or 22.6%
If one throws the dice 60 times, the chance of observing a "7" is 1 - (0.95)**60 or 95.4%
As the number of RCTs with identical treatments (# of dice tosses) increases, the chance for
seeing at least one false-positive conclusion (appearance of #7) increases. Given sufficient
resources, one can conduct the number of clinical trials needed to have one clinical trial with
"significant" effects. The expected number of false positive trials in a series of n trials with a
type I error of 5% is 0.05*n. Thus, with 20 trials the expected number of false positives is
20*0.05 or 1 trial.