Review of Leture 13
• Data ontamination
0.8
Expeted Error
• Validation
0.7
D
−
Eout gm
∗
0.6
(N )
Dtrain
−
Eval gm
∗
0.5
(N − K )
5
g
Dval
(K )
Dval
15
Validation Set Size,
K
25
slightly ontaminated
• Cross validation
z
Eval(g )
g
Eval(g −)
estimates
Eout(g)
D
}|
{
D1 D2 D3 D4 D5 D6 D7 D8 D9 D10
train validate
train
10-fold ross validation
Learning From Data
Yaser S. Abu-Mostafa
California Institute of Tehnology
Leture 14: Support
Vetor Mahines
Sponsored by Calteh's Provost Oe, E&AS Division, and IST
•
Thursday, May 17, 2012
Outline
•
Maximizing the margin
•
The solution
•
Nonlinear transforms
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
2/20
Better linear separation
Hi
Hi
Hi
Linearly separable data
Dierent separating lines
Whih is best?
Hi
Hi
Hi
Two questions:
1. Why is bigger margin better?
2. Whih
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
w
maximizes the margin?
3/20
Remember the growth funtion?
All dihotomies with any line:
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
4/20
Dihotomies with fat margin
Fat margins imply fewer dihotomies
AM
L
infinity
0.866
0.5
0.397
infinity
0.866
0.5
0.397
Creator: Yaser Abu-Mostafa - LFD Leture 14
5/20
Finding
Let
xn
w
with large margin
be the nearest data point to the plane
wTx = 0.
How far is it?
2 preliminary tehnialities:
1.
Normalize
w:
|wTxn| = 1
2.
Pull out
w0:
w = (w1, · · · , wd)
The plane is now
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
apart from
w Tx + b = 0
b
(no
x0 )
6/20
Computing the distane
The distane between
⊥ to
The vetor
w
is
Take
x′
and
x′′
xn
the plane in the
X
wTx + b = 0
′
and
′′
where
|wTxn + b| = 1
spae:Hi
on the plane
wTx′ + b = 0
T
and the plane
xn
w
x’
wTx′′ + b = 0
=⇒ w (x − x ) = 0
x’’
Hi
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
7/20
and the distane is
Distane between
Take any point
Projetion of
x
AM
L
and the plane:
on the plane
xn − x
w
ŵ =
=⇒
kwk
distane
xn
on
w
...
Hi
xn
w
x
T
distane = ŵ (xn − x)
Hi
1
1 T
1 T
T T
=
w xn − w x =
w xn + b − w x − b =
kwk
kwk
kwk
Creator: Yaser Abu-Mostafa - LFD Leture 14
8/20
The optimization problem
Maximize
1
kwk
subjet to
min
n=1,2,...,N
|wTxn + b| = 1
Notie:
Minimize
subjet to
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
|wTxn + b| = yn (wTxn + b)
1 T
ww
2
T
yn (w xn + b) ≥ 1
for
n = 1, 2, . . . , N
9/20
Outline
•
Maximizing the margin
•
The solution
•
Nonlinear transforms
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
10/20
Constrained optimization
Minimize
subjet to
1 T
ww
2
yn (wTxn + b) ≥ 1
for
n = 1, 2, . . . , N
w ∈ Rd, b ∈ R
Lagrange?
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
inequality onstraints
=⇒
KKT
11/20
We saw this before
Remember regularization?
Minimize
E (w) =
in
(Zw − y)T(Zw − y)
w
lin
normal
w
in
normal to onstraint
optimize
onstrain
E
w Tw
Regularization:
SVM:
AM
L
in
wTw ≤ C
subjet to:
∇E
1
N
E = onst.
in
wTw
Creator: Yaser Abu-Mostafa - LFD Leture 14
E
∇E
in
wtw = C
in
12/20
Lagrange formulation
Minimize
1 T
L(w, b, α) = w w −
2
w.r.t.
w
and
b
and
N
X
αn(yn (wTxn + b) −1)
n=1
maximize w.r.t. eah
∇wL = w −
αn ≥ 0
N
X
αnynxn = 0
n=1
∂L
= −
∂b
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
N
X
αn y n = 0
n=1
13/20
Substituting
w =
N
X
αnynxn
N
X
and
1 T
L(w, b, α) = w w −
2
in the Lagrangian
N
X
αn (yn (wTxn+b) −1 )
n=1
N
X
N X
N
X
1
L(α) =
αn −
ynym αnαm xnT xm
2 n=1 m=1
n=1
we get
AM
L
αn y n = 0
n=1
n=1
Maximize w.r.t. to
...
α
subjet to
Creator: Yaser Abu-Mostafa - LFD Leture 14
αn ≥ 0
for
n = 1, · · · , N
and
PN
n=1 αnyn
=0
14/20
The solution - quadrati programming

min
α
subjet to

1 T
α 
2

|
y1y1 x1Tx1 y1y2 x1Tx2
y2y1 x2Tx1 y2y2 x2Tx2
...
...
yN y1 xNTx1 yN y2 xNTx2
{z
quadrati oeients
T
y
| α{z= 0}


T
)α
 α + |(−1
{z
}

linear
}
linear onstraint
0
|{z}
lower bounds
AM
L
. . . y1yN x1TxN
. . . y2yN x2TxN
...
...
. . . yN yN xNTxN

Creator: Yaser Abu-Mostafa - LFD Leture 14
≤
α
≤
∞
|{z}
upper bounds
15/20
QP hands us
Solution:
α = α1 , · · · , αN
=⇒ w =
N
X
αnynxn
α
E = onst.
in
w
lin
n=1
KKT ondition:
normal
For
n = 1, · · · , N
w
αn (yn (wTxn + b) − 1) = 0
∇E
We saw this before!
αn > 0 =⇒ xn
AM
L
is a
in
support vetor
Creator: Yaser Abu-Mostafa - LFD Leture 14
wtw = C
16/20
Support vetors
Hi
Closest
xn's
to the plane: ahieve the margin
=⇒ yn (wTxn + b) = 1
X
w =
xn
Solve for
is SV
b
αnynxn
using any SV:
yn (wTxn + b) = 1
Hi
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
17/20
Outline
•
Maximizing the margin
•
The solution
•
Nonlinear transforms
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
18/20
z
instead of
x
N
X
N X
N
X
1
L(α) =
αn −
ynym αnαm zTnzm
2
n=1
n=1 m=1
PSfrag replaements
1
1
0
X PSfrag
−→replaements
Z 0.5
−1
−1
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
0
0
1
0
0.5
1
19/20
Support vetors in
X
spae
Hi
Support vetors live in
In
X
Z
spae
spae, pre-images of support vetors
The margin is maintained in
Z
spae
Generalization result
E[Eout]
≤
E [#
of SV's
]
N −1
Hi
AM
L
Creator: Yaser Abu-Mostafa - LFD Leture 14
20/20