Quantifying Treatment Effects
Mark Pletcher
6/10/2011
Rationale

Any treatment involves tradeoffs

Weigh benefits against risks/costs
Benefit
$$
Harm
Rationale

Sometimes the decision is difficult!
Benefit
$$
Harm
Rationale
How big is this box?
Benefit
And this one?
$$
Harm
Rationale

Tests can help us understand who is
most likely to benefit from a treatment
How big is this box?
Benefit
And this one?
$$
Harm
Rationale

Tests can help us understand who is
most likely to benefit from a treatment



Rapid strep to decide who will benefit from
penicillin
BNP to decide who will benefit from
furosemide
CRP to decide who will benefit from statins
Rationale

The utility of a test depends on:




How beneficial the treatment is
How harmful the treatment is
How much the test tells us about these
benefits and harms in a given individual
Risk of harm from the test itself
Rationale

The utility of a test depends on:




How beneficial the treatment is
How harmful the treatment is
How much the test tells us about these
benefits and harms in a given individual
Risk of harm from the test itself
The topic for this lecture
Outline




Is an intervention really beneficial?
How beneficial is it?
Pitfalls
Examples
Is the intervention beneficial?

Randomized trials compare an outcome
in treated to untreated persons


MI in 10% vs. 15%
Duration of flu symptoms 3 vs. 5 days
Is the intervention beneficial?

Randomized trials compare an outcome
in treated to untreated persons



MI in 10% vs. 15%
Duration of flu symptoms 3 vs. 5 days
*Statistics* are used to decide if should
reject the “null hypothesis” and accept
that the intervention is beneficial
Is the intervention beneficial?

But statistics cannot help us interpret
effect size
Quantifying the Benefit

Effect size


How do we summarize and communicate this?
What is really important for clinicians and
policymakers?
Quantifying the Benefit

Effect size




How do we summarize and communicate this?
What is really important for clinicians and
policymakers?
Example:
MI in 10% vs. 15%
Q: What do we do with these two numbers?
Quantifying the Benefit

Two simple possibilities:


10% / 15% = 0.66
15% - 10% = 5%
Quantifying the Benefit

Two simple possibilities:


10% / 15% = 0.66
15% - 10% = 5%
Relative Risk (RR)
Absolute Risk Reduction (ARR)
Quantifying the Benefit

Relative risk as a measure of effect size

RR = 0.66 – is this big or small?
Quantifying the Benefit

Relative risk as a measure of effect size

RR = 0.66 – is this big or small?



MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
Quantifying the Benefit

Relative risk as a measure of effect size

Medium
RR = 0.66 – is this big or small?

Big

Small

MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
Quantifying the Benefit

Relative risk as a measure of effect size

RR = 0.66 – is this big or small?




MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR is NOT the best measure of effect size
Quantifying the Benefit

Absolute risk reduction (ARR) is better

ARR = Risk difference = Risk2 – Risk1
Quantifying the Benefit

Absolute risk reduction (ARR) is better
MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR
.66
.66
.66
ARR
5%
25%
1%
Q: What does the 34% reduction mean?
Nimotop® Ad Graph
22%




Risk1 = 61/278 = 21.8%
Risk2 = 92/276 = 33%
RR = 22%/33% = .66
ARR = 33% - 22% = 11%
33%
Nimotop® Ad Graph
22%




Risk1 = 61/278 = 21.8%
Risk2 = 92/276 = 33%
RR = 22%/33% = .66
ARR = 33% - 22% = 11%
33%
What is 34%?
Nimotop® Ad Graph
22%




Risk1 = 61/278 = 21.8%
Risk2 = 92/276 = 33%
RR = 22%/33% = .66
ARR = 33% - 22% = 11%
33%
Relative risk reduction (RRR) =
1 – RR = 1-.66 = .34 or 34%
Quantifying the Benefit

RRR is no better than RR
MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR
.66
.66
.66
RRR
34%
34%
34%
Quantifying the Benefit

RRR is ALWAYS bigger than ARR

(unless untreated risk is 100%)
Quantifying the Benefit

BEWARE of risk reduction language!!

ARR or RRR?


“We reduced risk by 34%”
“Risk was 34% lower”
Quantifying the Benefit

BEWARE of risk reduction language!!

ARR or RRR?



“We reduced risk by 34%”
“Risk was 34% lower”
can’t tell
can’t tell
Very hard to be unambiguous!
Quantifying the Benefit

Another reason that ARR is better:

Translate it into “Number Needed to Treat”

NNT = 1/ARR
Why is NNT = 1/ARR?
100 SAH patients
treated
67 no stroke anyway
R2
11 strokes prevented
R1
22 strokes with Nimotop®
33 strokes with
no treatment
22 strokes with
with treatment
Why is NNT 1/ARR?
Treat 100 SAH patients  prevent 11 strokes
Ratio manipulation:
100 treated = 1 treated
= 9.1 treated
11 prevented
.11 prevented
1 prevented
Why is NNT 1/ARR?
Treat 100 SAH patients  prevent 11 strokes
Ratio manipulation:
100 treated = 1 treated
= 9.1 treated
11 prevented
.11 prevented
1 prevented
1/ARR
=
NNT
Why is NNT 1/ARR?
NNT best expressed in a sentence:
“Need to treat 9.1 persons with SAH using
nimodipine to prevent 1 cerebral infarction”
Quantifying the Benefit

NNT calculation practice
MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR
.66
.66
.66
ARR NNT?
5%
25%
1%
Quantifying the Benefit

NNT calculation practice
MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR
.66
.66
.66
ARR NNT?
5% 20
25%
1%
Quantifying the Benefit

NNT calculation practice
MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR
.66
.66
.66
ARR NNT?
5% 20
25% 4
1%
Quantifying the Benefit

NNT calculation practice
MI:
Death:
Basal Cell CA:
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR
.66
.66
.66
ARR
5%
25%
1%
NNT?
20
4
100
Quantifying the Benefit

Statins
NNT expression practice
MI:
Death:
Chemo
Sunscreen Basal Cell CA:
every day
10% vs. 15% in 10 years
50% vs. 75% in 3 years
2% vs. 3% in lifetime
RR
.66
.66
.66
ARR
5%
25%
1%
NNT?
20
4
100
Quantifying the Benefit

NNT expression practice
“Need to treat 20 patients with statins for 10 years to prevent 1 MI”
“Need to treat 4 patients with chemo for 3 years to prevent 1 death”
“Need to treat 100 patients with sunscreen every day for their whole life to
prevent 1 basal cell”
Example 1

Randomized controlled trial of the
effects of hip replacement vs. screws on
re-operation in elderly patients with
displaced hip fractures.
Parker MH et al. Bone Joint Surg Br. 84(8):1150-1155.
Example 1
Re-operation
No Re-operation
Hip Replacement
12
217
229
Internal Fixation
with Screws
90
136
226
Parker MH et al. Bone Joint Surg Br. 84(8):1150-1155.
Example 1
Re-operation
No Re-operation
Risk
Hip Replacement
12
217
229
12/229 = 5.2%
Internal Fixation
with Screws
90
136
226
90/226 = 39.8%
Example 1
Re-operation
No Re-operation
Risk
Hip Replacement
12
217
229
12/229 = 5.2%
Internal Fixation
with Screws
90
136
226
90/226 = 39.8%
RR
= R1/R2
= 5.2% / 39.8%
= .13
RRR
= 1-RR
= 1-.13
= 87%
ARR
= R2 – R1
= 39.8% - 5.2%
= 34.6%
NNT
= 1/ARR
= 1/.346
=3
“Need to treat 3 patients with hip replacement instead of screws to prevent 1
from needing a re-do operation”
Example 2

JUPITER: Randomized controlled trial of
high dose rosuvastatin in patients with
LDL<130 and CRP>2.0
Ridker et al. NEJM 2008; 359:2195-207
Example 2
Ridker et al. NEJM 2008; 359:2195-207
Example 2
Ridker et al. NEJM 2008; 359:2195-207
Example 2
HR
= (R1/R2)
(from regression)
= .56
RRR
= 1-HR
= 1-.56
= 44%
ARR
= R2 – R1
= 1.36 - 0.77
= .59 / 100py*
= .0059 / py
NNT
= 1/ARR
= 1/.0059 = 100/.59
“Need to treat 169 patients for a year to prevent 1 CVD event”
Or better:
“Need to treat 85 patients for 2 years to prevent 1 CVD event”
(average treatment duration in trial was 1.9 years)
* py = person-years
= 169 pys
Example 4
Warfarin vs. placebo for atrial fibrillation
Warfarin Placebo
Risk of major bleed (/yr)
1.2%
0.7%
Ann Intern Med 1999; 131:492-501
Example 4
Warfarin vs. placebo for atrial fibrillation
RR
= R1/R2
= 1.2% / .7%
= 1.7
RR (flipped) = R2/R1
= .7% / 1.2%
= .59
RRR (flipped) = 1-RR
= 1 - .59
= 41%
ARR
= .7% - 1.2%
= -.5%
= R2 – R1
“ARI” – Absolute risk increase = 0.5%
NNT
= 1/ARR
= 1/-.5%
= -200
“NNH” – Number needed to harm = -NNT = 1/ARI = 200
“If you treat 200 Afib patients with warfarin, you will cause 1 major bleed”
Circling back to test utility…

Tests help determine:

If the RR applies


Treatment for a disease doesn’t help if you don’t have
the disease!
Interactions (RR is higher or lower than average)



Statins more effective if CRP is high?
Patients with gene XYZ more likely to have a side effect
Baseline risk

The higher the risk, the larger the ARR, the smaller the
NNT
Key Concepts



Test utility depends on how good the
treatment is
RR and p-values good for hypothesis
testing/statistics
ARR and NNT (and NNH) better for
interpreting clinical importance



ARR = risk difference
NNT = 1/NNT
Beware RRR and ambiguous language