Safety, Can You Paradigm? A Statistical Lament Janet Turk Wittes Statistics Collaborative Harms identified late •fenflurmine-phentermine (Fen Phen) •Rofecoxib (Vioxx) •Troglitazone (Rezulin) •HRT (Premarin and PremPro) •Celecoxib (Celebrex) •Telithromycin (Ketek) •Rosiglitazone (Avandia) •Antidepressants, anti-epileptics…. 2 Could we have identified these harms earlier? •Troglitazone (Rezulin) -removed from market in 2000 Lots of liver abnormalities Severe toxicities noted in 1997 Other equally effective drugs didn’t have same problems 3 Could we have identified these harms earlier? •Troglitazone (Rezulin) • Rofecoxib (Vioxx) -removed from market in 2000 -removed from market in 2004 Every study showed excess heart attack Attributed to benefit of naproxen 4 Could we have identified these harms earlier? •Troglitazone (Rezulin) – removed from market in 2000 •Rofecoxib (Vioxx) -removed from market in 2004 •HRT (Premarin/PremPro)-major label change 2006 Heart attacks in Puerto Rican girls on oral contraception -1960’s Men on estrogens had higher event rates – 1970’s 5 Could we have identified these harms earlier? •Troglitazone (Rezulin) – removed from market in 2000 •Rofecoxib (Vioxx) -removed from market in 2004 •HRT (Premarin and PremPro)-label change 2006 •Celecoxib (Celebrex) – paper published 2005 •Telithromycin (Ketek) – major label change 2007 6 December 2004 7 “CELEBRATE :: CELEBREX” How we statisticians help to save drugs •We find safety boring 8 For efficacy we think hard about… Outcomes Population to study Protocol Analysis of primary outcome Control of Type I error rate Other outcomes Missing data Sensitivity analyses 9 How we statisticians save drugs •Because we find safety boring…. We don’t look at preclinical and early Phase data We don’t ask about •Chemistry •Biology •What PK/PD studies show •Safety part of analysis plan is an afterthought 10 How the statistical a-police protect drugs •We test hypotheses •Put events in correct body system •Give precise definitions •No data dredging •Too many type 1 errors if we dredge 11 And we divide and… •conquer •obfuscate 12 e.g. Neuropathy Event T C Neuropathic pain 1 0 Neuropathy 1 0 Neuropathy NOS 5 2 Neuropathy peripheral 2 0 … 2 1 … … 13 e.g. Neuropathy Event T C Parathesia 3 2 Parathesia NOS 4 0 Parathesia other 0 1 Peripheral motor neuropathy 6 0 Peripheral sensory neuropathy 3 2 … … 14 True(ish) data from a coxib Cardiac disorders Respiratory Vascular disorders C 42 33 7 T 46 29 9 15 True(ish) data from a coxib Cardiac disorders •Angina •Angina aggravated •Angina unstable •… •Cardiac arrest •Cardiac failure congest •Coronary artery disease •… •Myocardial infarction C 42 2 0 0 T 46 2 2 3 0 2 4 1 0 7 5 10 16 True(ish) data from a coxib Respiratory 33 29 •Dyspnea 1 3 Vascular disorders 7 9 •Cerebral infarction •Pulmonary embolism •TIA 0 0 2 1 2 0 17 If you combined… No. of people with at least one serious thromboembolic event or evidence of heart failure Placebo Coxib 16 27 18 Other ways to save drugs Modified Daley’s Rule: Censor early and often 19 e.g., Rofecoxib- short follow-up 20 Through 36 months 21 With denominators (Bresalier et al. NEJM 2005 352:1092) (And see Adam Boyd’s poster!) 22 Known or suspected adverse events •Monitor them •Look at events, their (near) synonyms, labs Are they real? Are they too frequent? 23 Hierarchical multiplicity •Think of biology •Order hierarchy by decreasing Biological plausibility Objectivity •Look for monotone decreasing hazard ratio 24 Which dose of celecoxib do you want? 25 APC Study (Placebo vs high dose) Outcome n -----------------------------------------------CV death 6 +MI 19 +Stroke 26 +CHF 29 +Angina 34 +CV procedure 46 ----------------------------------------------Other CV 62 26 Adenoma Prevention with Celecoxib (APC) Study HR CV death 5.1 +MI 3.8 +Stroke 3.4 +CHF 3.2 +Angina 2.1 +CV procedure 1.7 ------------------------------------------------Other CV 1.1 27 APC Study CV death 5.1 ( 0.6, 43.2) +MI 3.8 ( 1.3, 11.4) +Stroke 3.4 ( 1.4, 8.3) +CHF 3.2 ( 1.4, 7.4) +Angina 2.1 ( 1.0, 4.3) +CV procedure 1.7 ( 1.0, 3.1) -----------------------------------------Other CV 1.1 ( 0.7, 1.8) 28 APC Study CV death 5.1 ( 0.6, 43.2) 0.14 +MI 3.8 ( 1.3, 11.4) 0.015 +Stroke 3.4 ( 1.4, 8.3) 0.007 +CHF 3.2 ( 1.4, 7.4) 0.006 +Angina 2.1 ( 1.0, 4.3) 0.05 +CV procedure 1.7 ( 1.0, 3.1) 0.05 ------------------------------------------------Other CV 1.1 ( 0.7, 1.8) 0.7 Solomon (2006). Circulation 114:1028 29 Unknown harms: usual approach •Respond by Agonizing Checking informed consent document Asking for more frequent looks Asking for more thorough analyses •Worry about falsely discovered harm 30 Sentinel events •Identify •Follow in the next patients •Invent formal statistical methods 31 Single sentinel event •Childhood vaccine •30 day follow-up for serious adverse events •1 death occurred •DSMB: did the vaccine cause the death? 32 Women’s Health Initiative •Early in the trial, DSMB noted: Increase in stroke Increase in pulmonary embolism Increase in myocardial infarction •Possible sentinel events Myocardial infarction The big meanies: stroke, PE, MI 33 Proposal 1. 2. Identify sentinel event (or cluster or rate) Monitor for subsequent occurrence(s) Have reasonable power Be statistically unbiased (exclude sentinel) Type 1 error rate may be large (~0.2) Lachenbruch, Wittes: 2007 34 Safety report sample –abnormal lab values •Time A B Total •Point [N= 150] [N= 148] [N= 298] •_______________________________________________________________ SCREENING 0 0 0 •RANDOM 0 0 0 •WEEK 2 0 0 0 •WEEK 3 0 0 0 •WEEK 4 0 0 0 •WEEK 5 0 0 0 •WEEK 6 0 0 0 •WEEK 7 0 0 0 •WEEK 8 0 0 0 •___________________________________________________________________ 35 But wait! You also get: • Time A B Total • Point [N= 150] [N= 148] [N= 298] •_______________________________________________________________ • SCREENING 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • RANDOM 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 2 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 3 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 4 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 5 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 6 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 7 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 8 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) •_______________________________________________________________ 36 And 150 pages of where’s Waldo • Time A B Total • Point [N= 150] [N= 148] [N= 298] •_______________________________________________________________ • SCREENING 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • RANDOM 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 2 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 3 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 4 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 5 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 6 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 7 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • WEEK 8 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) • EARLY TERM 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) UNSCHEDULED 0 (0.00 %) 0 (0.00 %) 0 (0.00 %) •_______________________________________________________________ 37 And if this isn’t enough… •Change from baseline where missing is counted as zero (change in HR=64????) •Values out of temporal order •Lots and lots of decimal places •P-values to 3 and 4 significant digits •Etc., etc. etc. 38 We need to change our habits •Current statistical approach One variable at a time Template applied to all studies No wonder the docs don’t ask us to work with them! •Simple change in attitude Safety parameters aren’t separable Focus first from biological insights and previous hints Then scan the other variables Then refocus 39 Conclusions •Worry about multiplicity, but not too much Listen to Joe Heyse’s talk this afternoon •Beware the censor-happy protocol and analysis •Don’t be too much the statistician •But don’t forget randomness 40