advertisement

Complete Least Squares and Variable Screening Eric M Reyes Under the Direction of Dennis D Boos and Leonard A Stefanski North Carolina State University Department of Statistics Rose-Hulman Institute of Technology Mathematics Seminar 15 Feb 2012 RHIT Seminar Outline 1 Motivation 2 Complete Least Squares CLS Objective Function CLS Estimator Related Estimators 3 Screening via CLS CLS Variable Orderings Simulation Studies 4 Discussion RHIT Seminar Motivation Complete Least Squares Screening via CLS Discussion Acknowledgments NIH Grants T32HL079986 and P01 CA142538 for funding support. RHIT Seminar (3) Motivation Complete Least Squares Screening via CLS Discussion Acknowledgments NIH Grants T32HL079986 and P01 CA142538 for funding support. RHIT Seminar (3) Motivation Complete Least Squares Screening via CLS Discussion Motivation RHIT Seminar (4) Motivation Complete Least Squares Screening via CLS Discussion Variable Selection What are the risk factors associated with heart failure? Which genetic biomarkers oﬀer early identification of individuals more likely to develop cancer? When triaging a stroke victim in the emergency room, which characteristics on their medical chart are predictive of recurrent stroke? RHIT Seminar (5) Motivation Complete Least Squares Screening via CLS Discussion Genetic Studies Polycystic Ovary Syndrome Endocrine disorder aﬀecting 10% of reproductive-aged women. Characterized by high androgen levels. Goal: identify genes associated with increased androgen levels. E [yi |x1,i , . . . , xp,i ] = x1,i β1 + x2,i β2 + x3,i β3 + · · · + xp,i βp Framingham Heart Study Conducted to identify risk factors for cardiovascular disease. Genetic and phenotypic data collected on ∼ 9000 subjects Just over 50% female. Genetic data includes 50,000 variables. RHIT Seminar (6) Motivation Complete Least Squares Screening via CLS Discussion Variable Screening Sure Independence Screening (SIS) [Fan, JRSS-B 2008] Order predictors by their correlation with the response. Retain top k predictors for variable selection. k chosen to be near the sample size (e.g. k = δn, δ ∈ (0, 1)). Assumes predictors are marginally related to response. RHIT Seminar (7) Motivation Complete Least Squares Screening via CLS Discussion Drawback of SIS ● ● 14 Speed Driven Over Posted Limit (MPH) ● ● ● ● ● ● ● ● 12 ● ● ● 8 6 4 ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ●● ● ● ●●●● ●● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●●● ●● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ●● ●● ● ● 10 ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 60 65 70 75 Height (inches) RHIT Seminar (8) Motivation Complete Least Squares Screening via CLS Discussion Drawback of SIS ● Speed Driven Over Posted Limit (MPH) ● ● ● ● ● ● ● 12 ● ● ● 8 6 4 ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ●● ● ● ●●●● ●● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●●● ●● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ●● ●● ● ● 10 ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Speed Driven Over Posted Limit (MPH) ● 14 ● ● ● 14 65 70 Height (inches) 75 ● ● ●● ● ● ● 12 ● ● ● ●● ● ● 10 ● 8 6 4 ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ●●●● ●● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ●●●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●●●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ●● ●● ● ● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ●●● ● ● ● ● ● ●● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● 60 ● 60 65 70 75 Height (inches) ● Female ● Male RHIT Seminar (8) Motivation Complete Least Squares Screening via CLS Discussion Drawback of SIS ● ● 3 ●● ● ● ● ● ● ● ● Response ● −1 −2 −3 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● 0 ● ● 2 1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●●●● ● ● ● ● ● ●● ● ● ● ● ● ●● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ●●●● ● ●●● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● 0.2 0.4 0.6 0.8 Dose RHIT Seminar (9) Motivation Complete Least Squares Screening via CLS Discussion Drawback of SIS ● ● 3 ●● ● ● ● ● −1 −2 −3 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●●●● ● ● ● ● ● ●● ● ● ● ● ● ●● ●●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ●●●● ● ●●● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● 1 0 −1 −2 −3 ● ● 2 ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Response Response ● ● ● ● ● 0 ● ● 2 ●● ● ● ● ● 3 ● 1 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.4 0.6 0.8 ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ●● ● ●● ● ●●● ● ● ●● ● ●● ● ●●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ●●●● ●● ● ●● ● ● ●●● ●● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ●●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● 0.2 ● ● ● ● ● ● ● ● ● 0.2 0.4 0.6 0.8 Dose Dose ● No Aspirin ● Aspirin RHIT Seminar (9) Motivation Complete Least Squares Screening via CLS Discussion Variable Screening Sure Independence Screening (SIS) [Fan, JRSS-B 2008] Order predictors by their correlation with the response. Retain top k predictors for variable selection. k chosen to be near the sample size (e.g. k = δn, δ ∈ (0, 1)). Assumes predictors are marginally related to response. RHIT Seminar (10) Motivation Complete Least Squares Screening via CLS Discussion Complete Least Squares (CLS) RHIT Seminar (11) Motivation Complete Least Squares Screening via CLS Discussion Notation Assume the linear model: y = Xβ + � y is an (n × 1) response vector. X is an (n × p) design matrix. Assume �i are i.i.d. such that E (�i ) = 0. V (�i ) = σ 2 , for unknown σ 2 . Assume y and X have been centered and scaled: y� 1 = 0, y� y = 1, X� 1 = 0, and X� X = R. R is a valid correlation matrix. RHIT Seminar (12) Motivation Complete Least Squares Screening via CLS Discussion “Good” Estimate of β Ordinary Least Squares (OLS) � β̂ OLS = X� X Best linear unbiased estimator. �−1 X� y “Noisy” when predictors are highly correlated. Not uniquely defined if p > n. RHIT Seminar (13) Motivation Complete Least Squares Screening via CLS Discussion “Good” Estimate of β LS Objective Functions for All Possible Models when p = 3 One-Variable Models: �y − x1 β1 �2 �y − x2 β2 �2 �y − x3 β3 �2 Two-Variable Models: �y − x1 β1 − x2 β2 �2 �y − x1 β1 − x3 β3 �2 �y − x2 β2 − x3 β3 �2 Three-Variable Models: �y − x1 β1 − x2 β2 − x3 β3 �2 RHIT Seminar (14) Motivation Complete Least Squares Screening via CLS Discussion “Good” Estimate of β Ordinary Least Squares (OLS) � β̂ OLS = X� X Best linear unbiased estimator. �−1 X� y “Noisy” when predictors are highly correlated. Not uniquely defined if p > n. Alternative: Use All Information Simultaneously Q(β) = �y − x1 β1 �2 + �y − x2 β2 �2 + �y − x3 β3 �2 + �y − x1 β1 − x2 β2 �2 + �y − x1 β1 − x3 β3 �2 + �y − x2 β2 − x3 β3 �2 + �y − x1 β1 − x2 β2 − x3 β3 �2 RHIT Seminar (15) Motivation Complete Least Squares Screening via CLS Discussion CLS Objective Function The general form of the CLS objective function is a weighted average of the LS objective functions for all possible models: Qp (β, ω) = ω1 p � j=1 + ω2 �y − xj βj �2 + � j<k + ω3 �y − xj βj − xk βk �2 � j<k<l �y − xj βj − xk βk − xl βl �2 + ···+ + ωp �y − Xβ�2 The model weights ω1 , ω2 , . . . , ωp ≥ 0 regulate the contribution of all models of a given size. RHIT Seminar (16) Motivation Complete Least Squares Screening via CLS Discussion CLS Objective Function The CLS objective function reduces to a simple form: Qp (β, ω) = (λ0 − pλ1 + (p − 1)λ2 ) y� y + λ2 �y − Xβ�2 + (λ1 − λ2 ) where λj = �p �p−j � k=1 ωk k−j p � k=1 �y − xk βk �2 for j = 0, 1, 2. RHIT Seminar (17) Motivation Complete Least Squares Screening via CLS Discussion CLS Estimator Theorem For a fixed set of model weights ω = (ω1 , ω2 , . . . , ωp )� such that ωk ≥ 0 for all k, the estimator � � � β CLS = τ X X + (1 − τ )DX� X minimizes the CLS objective function, where �−1 X� y DX� X = diag{X� X}, and �p−2� �p �p−1� �p τ = λ2 /λ1 = k=1 ωk k−2 / k=1 ωk k−1 Proof is similar to that of OLS. RHIT Seminar (18) Motivation Complete Least Squares Screening via CLS Discussion Choice of Model Weights � � � β CLS = τ X X + (1 − τ )DX� X �p �−1 X� y �p−2� λ2 k=1 ωk k−2 τ= = �p �p−1� λ1 k=1 ωk k−1 ωk �p−2� �k−2 � p−1 ωk k−1 k −1 = ≤ 1. p−1 ⇒ τ ∈ [0, 1] RHIT Seminar (19) Motivation Complete Least Squares Screening via CLS Discussion Choice of Model Weights � � � β CLS = τ X X + (1 − τ )DX� X �−1 X� y Ordinary Least Squares ωp = 1 and ωk = 0 for all k �= p τ =1 Univariate Marginal Models ω1 = 1 and ωk = 0 for all k �= 1 τ =0 Sum Across All Models ωk = 1 for all k τ = 1/2 RHIT Seminar (20) Motivation Complete Least Squares Screening via CLS Discussion Ridge Regression Penalized Objective The ridge estimator of β minimizes �y − Xβ�2 + ν�β�2 where ν is called a penalty parameter. Estimator � � β Ridge (ν) = X X + νIp � �−1 X� y RHIT Seminar (21) Motivation Complete Least Squares Screening via CLS Discussion Ridge Regression Connection to CLS � Ridge Estimator �−1 � � X X + νIp X y � CLS Estimator �−1 � � τ X X + (1 − τ )Ip X y � � Let τ = (ν + 1)−1 , then β CLS = (1 + ν)β Ridge (ν) Comparison Both estimators exist when p > n. CLS is an “inflated” Ridge estimator. Ridge shrinks coeﬃcients to 0, CLS “shrinks” to the marginal. ν chosen from the data, τ chosen by model weights. RHIT Seminar (22) Motivation Complete Least Squares Screening via CLS Discussion Ridge and CLS Trace CLS Ridge Standardized Coefficient 0.5 0.4 0.3 0.2 0.1 0.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 τ X1 X2 X3 RHIT Seminar (23) Motivation Complete Least Squares Screening via CLS Discussion Variable Screening via CLS RHIT Seminar (24) Motivation Complete Least Squares Screening via CLS Discussion Variable Screening Sure Independence Screening (SIS) [Fan, JRSS-B 2008] Order predictors by their correlation with the response. Retain top k predictors for variable selection. k chosen to be near the sample size (e.g. k = δn, δ ∈ (0, 1)). Assumes predictors are marginally related to response. RHIT Seminar (25) Motivation Complete Least Squares Screening via CLS Discussion CLS Sequence Standardize the response and covariates. Rank the variables by decreasing magnitude of the coeﬃcients. Compute the full CLS fit (τ = 1/2). Variable x1 x2 x3 x4 x5 x6 � |β| 0.364 3.785 2.491 1.027 0.073 0.365 Rank ϕ 5 1 2 3 6 4 RHIT Seminar (26) Motivation Complete Least Squares Screening via CLS Discussion Simulation Design Data Generation X: i-th row i.i.d. N(0, Σ) y = Xβ + � � ∼ N(0, I) βj = c (1.5)I(j≤3) (−1)uj , j ≤ 6 βj = 0 for j > 6 iid uj ∼ Ber (0.5) Parameters for Simulation p = 10n 100 replicate datasets (Σ)i,j = ρI(i�=j) c chosen such that ρ ∈ {0, 0.6} R 2 = β � Σβ/(β � Σβ + 1) = 0.6 RHIT Seminar (27) Motivation Complete Least Squares Screening via CLS Discussion Comparing Accuracy Accuracy A method is said to be “accurate” if it orders the variables correctly, with respect to the magnitude of the true parameter vector β (after standardization). RHIT Seminar (28) Motivation Complete Least Squares Screening via CLS Discussion Increasing Sample Size Independence Equicorrelation 0.9 0.7 0.6 0.5 0 30 0 25 0 20 0 15 0 10 50 0 30 0 25 0 20 0 15 10 0 0.4 50 Accuracy 0.8 Sample Size (p = 10n) CLS SIS RHIT Seminar (29) Motivation Complete Least Squares Screening via CLS Discussion Discussion and Summary RHIT Seminar (30) Motivation Complete Least Squares Screening via CLS Discussion Future Work Screening in GLM Many screening studies involve a binary endpoint. We have extended CLS to GLM framework. Weighted average of estimating equations. Algorithm is slow to converge. Distance Correlation Distance correlation is a measure of independence. It is not restricted to linear association. SIS has been extended to use with distance correlation. Idea: replace key pieces in CLS with distance correlation measures. � τ X� X + (1 − τ )I �−1 X� y RHIT Seminar (31) Motivation Complete Least Squares Screening via CLS Discussion Summary CLS is a new method of estimation, related to ridge regression. CLS estimator is a competitive screening technique. There may be advantages to its use in large samples. RHIT Seminar (32) Appendices References I � Fan J and Lv J. Sure Independence Screening for Ultrahigh Dimensional Feature Space. Journal of the Royal Statistical Society, Series B, 70:849-911, 2008. � Lipovetsky S. Enhanced Ridge Regressions. Mathematical and Computer Modelling, 51:338-348, 2010. � Wang H. Forward Regression for Ultra-High Dimensional Variable Screening. JASA, 104:1512-1524, 2009. RHIT Seminar (33)