Binary Logistic Regression
with SPSS
Karl L. Wuensch
Dept of Psychology
East Carolina University
Download the Instructional
Document
• http://core.ecu.edu/psyc/wuenschk/SPSS/
SPSS-MV.htm .
• Click on Binary Logistic Regression .
• Save to desktop.
• Open the document.
When to Use Binary Logistic Regression
• The criterion variable is dichotomous.
• Predictor variables may be categorical or
continuous.
• If predictors are all continuous and nicely
distributed, may use discriminant function
analysis.
• If predictors are all categorical, may use
logit analysis.
Wuensch & Poteat, 1998
• Cats being used as research subjects.
• Stereotaxic surgery.
• Subjects pretend they are on university
research committee.
• Complaint filed by animal rights group.
• Vote to stop or continue the research.
Purpose of the Research
•
•
•
•
•
Cosmetic
Theory Testing
Meat Production
Veterinary
Medical
Predictor Variables
•
•
•
•
Gender
Ethical Idealism (9-point Likert)
Ethical Relativism (9-point Likert)
Purpose of the Research
Model 1: Decision = Gender
• Decision 0 = stop, 1 = continue
• Gender 0 = female, 1 = male
• Model is ….. logit =
 Yˆ 
  a  bX
ln ODDS   ln 
 1  Yˆ 


• Yˆ is the predicted probability of the event
which is coded with 1 (continue the research)
rather than with 0 (stop the research).
Iterative Maximum Likelihood
Procedure
• SPSS starts with arbitrary regression
coefficents.
• Tinkers with the regression coefficients to
find those which best reduce error.
• Converges on final model.
SPSS
• Bring the data into SPSS
• http://core.ecu.edu/psyc/wuenschk/SPSS/
Logistic.sav
• Analyze, Regression, Binary Logistic
• Decision  Dependent
• Gender  Covariate(s), OK
Look at the Output
Case Processing Summary
Unweighted Cases
Selected Cases
a
N
Included in Analysis
Mis sing Cases
Total
Unselected Cases
Total
a. If weight is in effect, see classification table for the total
number of cases.
• We have 315 cases.
Percent
315
100.0
0
.0
315
100.0
0
.0
315
100.0
Block 0 Model, Odds
• Look at Variables in the Equation.
• The model contains only the intercept
(constant, B0), a function of the marginal
distribution of the decisions.
Variables in the Equation
B
Step 0
Constant
-.379
S.E.
.115
Wald
10.919
 Yˆ
ln ODDS   ln 
 1  Yˆ

df
Sig.
1

   . 379


.001
Exp(B)
.684
Exponentiate Both Sides
• Exponentiate both sides of the equation:
• e-.379 = .684 = Exp(B0) = odds of deciding to
continue the research.
Yˆ
1  Yˆ
 Exp (  . 379 )  . 684 
128
187
• 128 voted to continue the research, 187 to stop
it.
Probabilities
•
•
•
•
•
Randomly select one participant.
P(votes continue) = 128/315 = 40.6%
P(votes stop) = 187/315 = 59.4%
Odds = 40.6/59.4 = .684
Repeatedly sample one participant and
guess how e will vote.
Humans vs. Goldfish
• Humans Match Probabilities
–
–
(suppose p = .7, q = .3)
.7(.7) + .3(.3) = .49 + .09 = .58
• Goldfish Maximize Probabilities
–
.7(1) = .70
• The goldfish win!
SPSS Model 0 vs. Goldfish
• Look at the Classification Table for Block 0.
Classification Table
a,b
Predicted
decision
Step 0
Observed
decision
stop
Percentage
Correct
continue
stop
187
0
100.0
continue
128
0
.0
Overall Percentage
59.4
a. Constant is included in the model.
b. The cut value is .500
• SPSS Predicts “STOP” for every participant.
• SPSS is as smart as a Goldfish here.
Block 1 Model
• Gender has now been added to the model.
• Model Summary: -2 Log Likelihood = how
poorly model fits the data.
Model Summary
Step
1
-2 Log
likelihood
399.913a
Cox & Snell
R Square
.078
Nagelkerke
R Square
.106
a. Estimation terminated at iteration number 3 because
parameter estimates changed by less than .001.
Block 1 Model
• For intercept only, -2LL = 425.666.
• Add gender and -2LL = 399.913.
• Omnibus Tests: Drop in -2LL = 25.653 =
Model 2.
• df = 1, p < .001.
Omnibus Tests of Model Coefficients
Chi-square
Step 1
df
Sig.
Step
25.653
1
.000
Block
25.653
1
.000
Model
25.653
1
.000
Variables in the Equation
• ln(odds) = -.847 + 1.217Gender
ODDS  e
a  b Gender
Variables in the Equation
B
Step
a
1
S.E.
Wald
df
Sig.
Exp(B)
gender
1.217
.245
24.757
1
.000
3.376
Constant
-.847
.154
30.152
1
.000
.429
a. Variable(s) entered on s tep 1: gender.
Odds, Women
ODDS
 e
 . 847  1 . 217 ( 0 )
 e
 . 847
 0 . 429
• A woman is only .429 as likely to decide to
continue the research as she is to decide
to stop it.
Odds, Men
ODDS  e
 . 847  1 . 217 ( 1)
e
. 37
 1 . 448
• A man is 1.448 times more likely to vote to
continue the research than to stop the research.
Odds Ratio
male _ odds
female _ odds

1 . 448
 3 . 376  e
1 . 217
. 429
• 1.217 was the B (slope) for Gender, 3.376 is the
Exp(B), that is, the exponentiated slope, the
odds ratio.
• Men are 3.376 times more likely to vote to
continue the research than are women.
Convert Odds to Probabilities
• For our women,
Yˆ 
ODDS
1  ODDS

0 . 429
 0 . 30
1 . 429
• For our men,
Yˆ 
ODDS
1  ODDS

1 . 448
2 . 448
 0 . 59
Classification
• Decision Rule: If Prob (event)  Cutoff,
then predict event will take place.
• By default, SPSS uses .5 as Cutoff.
• For every man, Prob(continue) = .59,
predict he will vote to continue.
• For every woman Prob(continue) = .30,
predict she will vote to stop it.
Overall Success Rate
• Look at the Classification Table
Classification Table
a
Predicted
decision
Step 1
Observed
decision
stop
stop
continue
continue
140
47
74.9
60
68
53.1
Overall Percentage
66.0
a. The cut value is .500
140  68
315
Percentage
Correct

208
315
• SPSS beat the Goldfish!
 66 %
Sensitivity
• P (correct prediction | event did occur)
• P (predict Continue | subject voted to Continue)
• Of all those who voted to continue the research,
for how many did we correctly predict that.
68
68  60

68
128
 53 %
Specificity
• P (correct prediction | event did not occur)
• P (predict Stop | subject voted to Stop)
• Of all those who voted to stop the research, for
how many did we correctly predict that.
140
140  47

140
187
 75 %
False Positive Rate
• P (incorrect prediction | predicted occurrence)
• P (subject voted to Stop | we predicted Continue)
• Of all those for whom we predicted a vote to Continue
the research, how often were we wrong.
47
47  68

47
115
 41 %
False Negative Rate
• P (incorrect prediction | predicted nonoccurrence)
• P (subject voted to Continue | we predicted Stop)
• Of all those for whom we predicted a vote to Stop the
research, how often were we wrong.
60
140  60

60
200
 30 %
Pearson 2
• Analyze, Descriptive Statistics, Crosstabs
• Gender  Rows; Decision  Columns
Crosstabs Statistics
• Statistics, Chi-Square, Continue
Crosstabs Cells
• Cells, Observed Counts, Row
Percentages
Crosstabs Output
• Continue, OK
• 59% & 30% match logistic’s predictions.
gender * decision Crosstabulation
decision
stop
gender
Female
Count
% within gender
Male
Count
% within gender
Total
Count
% within gender
continue
Total
140
60
200
70.0%
30.0%
100.0%
47
68
115
40.9%
59.1%
100.0%
187
128
315
59.4%
40.6%
100.0%
Crosstabs Output
• Likelihood Ratio 2 = 25.653, as with
logistic.
Chi-Square Tests
Value
Asymp. Sig.
(2-sided)
df
Pearson Chi-Square
25.685b
1
.000
Likelihood Ratio
25.653
1
.000
N of Valid Cases
315
a. Computed only for a 2x2 table
b. 0 cells (.0%) have expected count less than 5. The
minimum expected count is 46.73.
Model 2: Decision =
Idealism, Relativism, Gender
• Analyze, Regression, Binary Logistic
• Decision  Dependent
• Gender, Idealism, Relatvsm
Covariate(s)
• Click Options and check “HosmerLemeshow goodness of fit” and “CI for
exp(B) 95%.”
• Continue, OK.
Comparing Nested Models
• With only intercept and gender,
-2LL = 399.913.
• Adding idealism and relativism dropped
-2LL to 346.503, a drop of 53.41.
• 2(2) = 399.913 – 346.503 = 53.41, p = ?
Model Summary
Step
1
-2 Log
likelihood
346.503a
Cox & Snell
R Square
.222
Nagelkerke
R Square
.300
a. Estimation terminated at iteration number 4 because
parameter estimates changed by less than .001.
Obtain p
• Transform, Compute
• Target Variable = p
• Numeric Expression =
1 - CDF.CHISQ(53.41,2)
p=?
• OK
• Data Editor, Variable View
• Set Decimal Points to 5 for p
p < .0001
• Data Editor, Data View
• p = .00000
• Adding the ethical ideology variables
significantly improved the model.
Hosmer-Lemeshow
• Hø: predictions made by the model fit
perfectly with observed group
memberships
• Cases are arranged in order by their
predicted probability on the criterion.
• Then divided into ten bins with
approximately equal n.
• This gives ten rows in the table.
For each bin and each event, we have
number of observed cases and expected
number predicted from the model.
Contingency Table for Hosmer and Lemeshow Test
decision = s top
Observed
Step
1
decision = continue
Expected
Observed
Expected
Total
1
29
29.331
3
2.669
32
2
30
27.673
2
4.327
32
3
28
25.669
4
6.331
32
4
20
23.265
12
8.735
32
5
22
20.693
10
11.307
32
6
15
18.058
17
13.942
32
7
15
15.830
17
16.170
32
8
10
12.920
22
19.080
32
9
12
9.319
20
22.681
32
10
6
4.241
21
22.759
27
• Note expected freqs decline in first
column, rise in second.
• The nonsignificant chi-square is indicative
of good fit of data with linear model.
Hosmer and Lemeshow Test
Step
1
Chi-square
8.810
df
Sig.
8
.359
Hosmer-Lemeshow
• There are problems with this procedure.
• Not even Hosmer and Lemeshow
recommend it these days.
• Even with good fit the test may be
significant if sample sizes are large
• Even with poor fit the test may not be
significant if sample sizes are small.
Linearity of the Logit
• We have assumed that the log odds are
related to the predictors in a linear fashion.
• Use the Box-Tidwell test to evaluate this
assumption.
• For each continuous predictor, compute
the natural log.
• Include in the model interactions between
each predictor and its natural log.
Box-Tidwell
• If an interaction is significant, there is a
problem.
• For the troublesome predictor, try
including the square of that predictor.
• That is, add a polynomial component to
the model.
• See T-Test versus Binary Logistic
Regression
Variables in the Equation
B
S.E.
gender
1.147
idealism
1.130 1.921
Wald
.269 18.129
.346
df
Sig.
Exp(B)
1
.000 3.148
1
.556 3.097
relatvsm
1.656 2.637
.394
1
.530 5.240
idealism by
Step 1a
-.652
.690
.893
1
.345
.521
idealism_LN
relatvsm by
-.479
.949
.254
1
.614
.620
relatvsm_LN
Constant
-5.015 5.877
.728
1
.393
.007
a. Variable(s) entered on step 1: gender, idealism, relatvsm, idealism * idealism_LN
, relatvsm * relatvsm_LN .
No Problem Here.
Model 3: Decision =
Idealism, Relativism, Gender, Purpose
• Need 4 dummy variables to code the five
purposes.
• Consider the Medical group a reference
group.
• Dummy variables are: Cosmetic, Theory,
Meat, Veterin.
• 0 = not in this group, 1 = in this group.
Add the Dummy Variables
• Analyze, Regression, Binary Logistic
• Add to the Covariates: Cosmetic, Theory,
Meat, Veterin.
• OK
Block 0
• Look at “Variables not in the Equation.”
• “Score” is how much -2LL would drop if a
single variable were added to the model
with intercept only.
Variables not in the Equation
Score
Step
0
Variables
Overall Statistics
df
Sig.
gender
25.685
1
.000
idealism
47.679
1
.000
relatvsm
7.239
1
.007
cosmetic
.003
1
.955
theory
2.933
1
.087
meat
.556
1
.456
veterin
.013
1
.909
77.665
7
.000
Effect of Adding Purpose
• Our previous model had -2LL = 346.503.
• Adding Purpose dropped -2LL to 338.060.
Model Summary
Step
1
-2 Log
likelihood
338.060a
Cox & Snell
R Square
.243
Nagelkerke
R Square
.327
a. Estimation terminated at iteration number 5 because
parameter estimates changed by less than .001.
• 2(4) = 8.443, p = .0766.
• But I make planned comparisons (with medical
reference group) anyhow!
Classification Table
• YOU calculate the sensitivity, specificity,
false positive rate, and false negative rate.
Classification Table
a
Predicted
decision
Step 1
Observed
decision
stop
stop
continue
Overall Percentage
a. The cut value is .500
continue
Percentage
Correct
152
35
81.3
54
74
57.8
71.7
Answer Key
•
•
•
•
Sensitivity = 74/128 = 58%
Specificity = 152/187 = 81%
False Positive Rate = 35/109 = 32%
False Negative Rate = 54/206 = 26%
Wald Chi-Square
• A conservative test of the unique
contribution of each predictor.
• Presented in Variables in the Equation.
• Alternative: drop one predictor from the
model, observe the increase in -2LL, test
via 2.
Variables in the Equation
95.0% C.I.for EXP(B)
B
Step
a
1
Wald
df
Sig.
Exp(B)
Lower
Upper
gender
1.255
20.586
1
.000
3.508
2.040
6.033
idealism
-.701
37.891
1
.000
.496
.397
.620
relatvs m
.326
6.634
1
.010
1.386
1.081
1.777
cos metic
-.709
2.850
1
.091
.492
.216
1.121
theory
-1.160
7.346
1
.007
.314
.136
.725
meat
-.866
4.164
1
.041
.421
.183
.966
veterin
-.542
1.751
1
.186
.581
.260
1.298
Constant
2.279
4.867
1
.027
9.766
a. Variable(s) entered on s tep 1: gender, idealism, relatvs m, cosmetic, theory, meat, veterin.
Odds Ratios – Exp(B)
• Odds of approval more than cut in half (.496) for
each one point increase in Idealism.
• Odds of approval multiplied by 1.39 for each one
point increase in Relativism.
• Odds of approval if purpose is Theory Testing
are only .314 what they are for Medical
Research.
• Odds of approval if purpose is Agricultural
Research are only .421 what they are for
Medical research
Inverted Odds Ratios
• Some folks have problems with odds
ratios less than 1.
• Just invert the odds ratio.
• For example, 1/.421 = 2.38.
• That is, respondents were more than two
times more likely to approve the medical
research than the research designed to
feed to poor in the third world.
Classification Decision Rule
• Consider a screening test for Cancer.
• Which is the more serious error
– False Positive – test says you have cancer,
but you do not
– False Negative – test says you do not have
cancer but you do
• Want to reduce the False Negative rate?
Classification Decision Rule
• Analyze, Regression, Binary Logistic
• Options
• Classification Cutoff = .4, Continue, OK
Effect of Lowering Cutoff
• YOU calculate the Sensitivity, Specificity,
False Positive Rate, and False Negative
Rate for the model with the cutoff at .4.
• Fill in the table on page 15 of the handout.
Answer Key
Value When Cutoff =
.5
.4
Sensitivity
58%
75%
Specificity
81%
72%
False Positive Rate
32%
36%
False Negative Rate
26%
19%
Overall % Correct
72%
73%
SAS Rules
• See, on page 16 of the handout, how easy
SAS makes it to see the effect of changing
the cutoff.
• SAS classification tables remove bias
(using a jackknifed classification
procedure), SPSS does not have this
feature.
Presenting the Results
• See the handout.
Interaction Terms
• Center continuous variables
• Compute the interaction terms or
• Let Logistic compute them.
Deliberation and Physical
Attractiveness in a Mock Trial
• Subjects are mock jurors in a criminal trial.
• For half the defendant is plain, for the
other half physically attractive.
• Half recommend a verdict with no
deliberation, half deliberate first.
Get the Data
• Bring Logistic2x2x2.sav into SPSS.
• Each row is one cell in 2x2x2 contingency
table.
• Could do a logit analysis, but will do
logistic regression instead.
• Tell SPSS to weight cases by Freq. Data,
Weight Cases:
• Dependent = Guilty.
• Covariates = Delib, Plain.
• In left pane highlight Delib and Plain.
• Then click >a*b> to create the interaction
term.
• Under Options, ask for the HosmerLemeshow test and confidence intervals
on the odds ratios.
Significant Interaction
• The interaction is large and significant
(odds ratio of .030), so we shall ignore the
main effects.
Variables in the Equation
95.0% C.I.for EXP(B)
Wald
Step
a
1
df
Sig.
Exp(B)
Lower
Upper
Delib
3.697
1
.054
.338
.112
1.021
Plain
4.204
1
.040
3.134
1.052
9.339
Delib by Plain
8.075
1
.004
.030
.003
.338
.037
1
.847
1.077
Constant
a. Variable(s) entered on step 1: Delib, Plain, Delib * Plain .
• Use Crosstabs to test the conditional
effects of Plain at each level of Delib.
• Split file by Delib.
•
•
•
•
Analyze, Crosstabs.
Rows = Plain, Columns = Guilty.
Statistics, Chi-square, Continue.
Cells, Observed Counts and Column
Percentages.
• Continue, OK.
Rows = Plain, Columns = Guilty
• For those who did deliberate, the odds of a
guilty verdict are 1/29 when the defendant
was plain and 8/22 when she was
attractive, yielding a conditional odds ratio
of 0.09483 .
Plain * Guilty Crosstabulation
a
Guilty
No
Plain
Attrractive
Count
% within Plain
Plain
Count
% within Plain
Total
Count
% within Plain
a. Delib = Yes
Yes
Total
22
8
30
73.3%
26.7%
100.0%
29
1
30
96.7%
3.3%
100.0%
51
9
60
85.0%
15.0%
100.0%
• For those who did not deliberate, the odds
of a guilty verdict are 27/8 when the
defendant was plain and 14/13 when she
was attractive, yielding a conditional odds
ratio of 3.1339.
Plain * Guilty Crosstabulation
a
Guilty
No
Plain
Attrractive
Count
% within Plain
Plain
Count
% within Plain
Total
Count
% within Plain
a. Delib = No
Yes
Total
13
14
27
48.1%
51.9%
100.0%
8
27
35
22.9%
77.1%
100.0%
21
41
62
33.9%
66.1%
100.0%
Interaction Odds Ratio
• The interaction odds ratio is simply the ratio of
these conditional odds ratios – that is,
.09483/3.1339 = 0.030.
• Among those who did not deliberate, the plain
defendant was found guilty significantly more
often than the attractive defendant, 2(1, N = 62)
= 4.353, p = .037.
• Among those who did deliberate, the attractive
defendant was found guilty significantly more
often than the plain defendant, 2(1, N = 60) =
6.405, p = .011.
Interaction Between Continuous
and Dichotomous Predictor
Interaction Falls Short of
Significance
Standardizing Predictors
• Most helpful with continuous predictors.
• Especially when want to compare the
relative contributions of predictors in the
model.
• Also useful when the predictor is
measured in units that are not intrinsically
meaningful.
Predicting Retention in ECU’s
Engineering Program
Practice Your New Skills
• Try the exercises in the handout.