Neural Networks and Classical Linear
Regression
Szu Hua Huang
Jianjun Luo
Texas Tech University
4/9/2015
Contents
A. Neural Networks and Classical Linear
Regression
1.
Introduction Neural Networks
2.
Neural Networks VS Classical Linear Regression
B. Case study
Neural Network - Neuron
STRUCTURE OF A NEURAL NETWORK
Multilayer feedforward networks
A simple MLP (multilayer perceptron)
Hidden Layer
Input Layer
Output Layer
Weights(Regression Coefficients)
W13
Node
3
1
Back-propagation Model
Updating (Learning)
W36
W14
W15
W46
Input
4
W23
2
W56
W24
W25
5
Transfer Function, g(s)
6
Output
Data from True Function
Errors
Transfer Function
Neural Network VS Classical Regression Model
Neural Network
• Non-linear
Classical Regression
• Linear
• Normality
• Constant variability of error
terms
• Independence Assumption
Neural Network VS Classical Regression Model
Neural Network
• Weight estimates(regression
coefficient) do not tell you
the effect
• No guarantee the best linear
combination of parameter
estimates
Classical Regression
• Regression coefficient
shows the effect
Neural Network VS Classical Regression Model
Multi-collinearity
• No effect to Neural
Network ?
Outlier or Influential
• No effect to Neural
Network ?
•
• It does hurt the Classical
Regression
It does hurt the Classical
Regression
Overfitting
Neural Network
• Sensitive to the given data, too
much flexibility to the
underlying distribution of
data.
• Big Sample size can help to
solve the problem of
overfitting.
Classical Regression
• Not very sensitive to the
given data.
Assessment
Neural Network
• Optimization plot based on
the updated weight
estimates at each iteration of
the iterative grip search
routine.
• Using valid data
Classical Regression
• R square
Optimization plot
• 1. When the new
weights are only
incrementally different
from those of the
preceding iteration
ASE
training
0
• 2. When the
misclassification rate
reaches a required
threshold
validation
5
10
15
Iteration
20
• 3. When the limit on
the number of runs is
reached
Supplement-Other Optimization Algorithms:
• Newton
• Quasi-Newton method
• Levenberg-Marquardt
• Gauss-Newton Method
• etc
Stanford Open Course – Machine Learning
B: Case Study
• Dataset
• The School Children Data Set from Lewis & Taylor
“Introduction to Experimental Ecology” (1967)
• Includes 126 male records
• Variables:
1) Age (months)
2) height (inches)
3) weight (pounds)
• Purpose
• Predicting the weight of male school children based
on their age and height.
• Comparing neural networks with OLS
Exploration of the dataset
Classical Linear Regression Model
proc reg data=men;
model weight=height age;
output out=regout p=pred r=resid;
run;
Output of OLS
Neural Network Model
INPUT
COMBINATION
Standardization
Height
OUTPUT
HIDDEN
w1+ w2S_Height+ w3S_Age = H11
TRANSFORMATION
H11
tanh(H11)) =A
S_Height
Weight
COMBINATION
Age
S_Age
Standardization
w7+w8A+ w9B=Weight
TRANSFORMATION
COMBINATION
H12
w4+ w5S_Height+ w6S_Age = H12
tanh(H12) =B
18
Neural Network in SAS
• The SAS neural network procedure
– PROC NEURAL
• SAS Enterprise Miner
– A visual programming with a GUI interface
Neural Network Modeling using SAS Enterprise Miner
• To save time, I recorded the following video to show
how to build the Neural Network Model with SAS
Enterprise Miner.
• In case you are interested, I uploaded this video to
YouTube:
http://www.youtube.com/watch?v=Bb3K7xAcJbk&feature=youtu.be
Neural Network Weight Estimates
Variables
Variable Definition
age_H11
AGE : Input Layer Weights for 1st hidden unit
height_H11
HEIGHT: Input Layer Weights for 1 st hidden unit
age_H12
AGE : Input Layer Weights for 2nd hidden unit
height_H12
HEIGHT: Input Layer Weights for 2ndhidden unit
BIAS_H11
Input-to-Hidden Layer Bias for 1st hidden unit
BIAS_H12
Input-to-Hidden Layer Bias for 2nd hidden unit
H11_weight
Hidden-to-Target Layer Weight for 1st hidden unit
H12_weight
Hidden-to-Target Layer Weight for 2nd hidden unit
BIAS_weight Hidden-to-Target Layer Bias
Weights
ŵ3
ŵ2
ŵ6
ŵ5
ŵ1
ŵ4
ŵ8
ŵ9
ŵ7
Weight
Estimate
1.731949
1.394462
-0.225332
1.164043
-5.645913
0.548446
29.703755
21.240443
125.950303
Neural network or classical linear regression?
Comparing Neural Network and Classical
Linear Regression predicted values
Output: Observed and Predicted Values of Male's Weight against Age
References
1. Eric Roberts. Neural networks. Available online at: http://www-csfaculty.stanford.edu/~eroberts/courses/soco/projects/neural-networks/
2. Jim Georges, 2009. Applied analytics using SAS Enterprise Miner 6.1 Course Notes. SAS
Institute Inc.
3. Lewis, T. and Taylor, L.R. 1967. Introduction to Experimental Ecology, Academic Press, Inc.
4. Randall Matignon, 2005. Neural Network Modeling using SAS Enterprise Miner.
AuthorHouse
5. SAS Institute, 1999. SAS/STAT User’s Guide Version 8. Available online at:
http://ciser.cornell.edu/sasdoc/saspdf/common/mainpdf.htm
6. Sue Walsh, 2002. Applying Data Mining Techniques Using Enterprise Miner Course Notes.
SAS Institute Inc.
7. Wikipedia. Neural network. Available online at: http://en.wikipedia.org/wiki/Neural_network
Thank You!