Logistic regression models 1/12 Example 1: dogs look like their owners? I Some people believe that dogs look like their owners. Is this true? I To test the above hypothesis, The New York Times conducted a quiz online. A group of dogs and owners are photographed by Fred Conrad. I For each dog, four possible owners are given in the quiz. Please choose the owner for each dog. http: //www.nytimes.com/interactive/2015/02/16/ sports/westminster-dog-show-quiz.html?_r=0 2/12 Example 2: breast cancer data set I Consider the data set collected by Richardson et al. (2006) as an example. The study aims to find genes that are associated with the sporadic basal-like cancers (BLC), a distinct class of human breast cancers. I In this example, the response variable Yi is the types of the breast cancer. For instance, we use Yi = 0 to represent the non-BLC type and Yi = 1 to represent the BLC type. I The predictors in this example are the gene expression data or the SNPs data. For example, we could consider the gene CSF2RA as one of the candidate gene. 3/12 Logistic regression models I Consider Yi to be Bernoulli distributed response. For example, Yi could be failure or success, or could be different treatment groups. Assume Yi ∼ Bernoulli(pi ) and Yi is associated with the covariates Xi . I Model the conditional expectation of Yi . Recall that, in linear models, we assume that E(Yi |Xi ) = XiT β and in the non-linear models, E(Yi |Xi ) = f (Xi ; β). I In logistic regression model, assume that E(Yi |Xi ) = pi depends on Xi . Namely, E(Yi |Xi ) = p(Xi ) for 0 ≤ p(Xi ) ≤ 1. 4/12 Link functions In general, we assume that E(Yi |Xi ) = h(XiT β). Here h−1 (·) is the link function, which links E(Yi |Xi ) with a linear function of Xi . Three commonly used link functions: logit link, probit link and complementary log-log link. exp(z) 1+exp(z) , p then h−1 (p) = log( 1−p ). I (Logit link) If p = h(z) = I (Probit link)If p = Φ(z) where Φ(z) is the CDF function of a standard normal, then h−1 (p) = Φ−1 (p). I (Complementary log-log) If p = h(z) = 1 − exp{− exp(η)}, then h−1 (p) = log{− log(1 − p)}. 5/12 A logistic regression model I Response: Bernoulli distributed random variable Yi ∼ Bernoulli(pi ) i = 1, · · · , n. I Systematic component: ηi = I Link function: h(ηi ) = pi . Pp j=1 Xij βj . 6/12 Estimation of β The estimation of β can be obtained by the maximum likelihood method. The likelihood function for β is L(β) = n Y piYi (1 − pi )1−Yi . i=1 The log-likelihood function for β is `(β) = log L(β) = n X n Yi log( i=1 = n X i=1 Yi XiT β − n X X pi )+ log(1 − pi ) 1 − pi i=1 log{1 + exp(XiT β)}. i=1 7/12 MLE of β I The MLE of β is β = arg max `(β) β where `(β) is log-likelihood function of β. I We do not have closed form solution of β. But `(β) is a concave function of β, which is relatively easy to optimize. 8/12 Score function and Hessian matrix I The score function of β is n n n i=1 i=1 i=1 X Xi exp(X T β) X ∂`(β) X i = Xi Yi − = Xi (Yi − pi ). ∂β 1 + exp(XiT β) I The hessian matrix of β is n X exp(XiT β) exp(XiT β) ∂`(β) T = − X X {1 − } i i ∂β∂β T 1 + exp(XiT β) 1 + exp(XiT β) i=1 = −X T VX where V = diag{p1 (1 − p1 ), · · · , pn (1 − pn )} and X = (X1 , · · · , Xn )T . Here pi = exp(XiT β)/{1 + exp(XiT β)}. 9/12 Extension to Binomial distributed data Suppose we observe Binomial distributed response Si ∼ Binomial(ni , pi ), where ni is known. We would like to study the association between the response Si and some covariates Xi . A corresponding logistic regression model is Si ∼ Binomial(ni , pi ) p i = XiT β log 1 + pi for i = 1, · · · , m. 10/12 Estimation of β We can still apply the maximum likelihood method to estimate β. The likelihood function for β is L(β) = m Y ni Si i=1 piSi (1 − pi )ni −Si . The log-likelihood function for β is `(β) = log L(β) = C + m X m Si log( i=1 =C+ n X i=1 Si XiT β − m X X pi )+ ni log(1 − pi ) 1 − pi i=1 ni log{1 + exp(XiT β)}. i=1 where C is constant that has nothing to do with β. 11/12 Score function and Hessian matrix I The score function of β is m n m i=1 i=1 i=1 X ni Xi exp(X T β) X ∂`(β) X i = Xi Si − = Xi (Si − ni pi ). ∂β 1 + exp(XiT β) I The hessian matrix of β is m X exp(XiT β) exp(XiT β) ∂`(β) T = − n X X {1 − } i i i ∂β∂β T 1 + exp(XiT β) 1 + exp(XiT β) i=1 = −X T VX where V = diag{n1 p1 (1 − p1 ), · · · , nm pm (1 − pm )} and X = (X1 , · · · , Xm )T . Here pi = exp(XiT β)/{1 + exp(XiT β)}. 12/12