Lecture 21 Bounds on the generalization error of voting classifiers. 18.465 We continue to prove the lemma from Lecture 20: �d Lemma 21.1. Let Fd = convd H = { i=1 λi hi , hi ∈ H} and fix δ ∈ (0, 1]. Then � �� � �� dV log nδ t Eϕδ − ϕ¯δ ≥ 1 − e−t . ≤K + P ∀f ∈ Fd , √ n n Eϕδ Proof. We showed that log D(ϕδ (yFd ) , ε/2, dx,y ) ≤ KdV log 2 . εδ By the result of Lecture 16, n 1� k Eϕδ (yf (x)) − ϕδ (yi f (xi )) ≤ √ n i=1 n � √ Eϕδ � log 1/2 D(ϕδ (yFd (x)) , ε)dε + 0 tEϕδ n with probability at least 1 − e−t . We have � √Eϕδ � √Eϕδ � k k 2 1/2 √ log D(ϕδ (yFd (x)) , ε)dε ≤ √ dV log dε εδ n 0 n 0 � � √ k 2 δ Eϕδ /2 √ 1 =√ dV log dx x nδ 0 � k 2√ δ� 2 ≤√ dV 2 Eϕδ log √ 2 nδ δ Eϕδ = x, ε = 2x . Without loss of generality, assume Eϕδ ≥ 1/n. � δ Otherwise, we’re doing better than in Lemma: √EE ≤ logn n ⇒ E ≤ logn n . Hence, � √ � √Eϕδ k dV Eϕδ 2 n 1/2 √ log D(ϕδ (yFd (x)) , ε)dε ≤ K log n δ n 0 � dV Eϕδ n ≤K log n δ where we have made a change of variables 2 εδ So, with probability at least 1 − e−t , n 1� Eϕδ (yf (x)) − ϕδ (yi f (xi )) ≤ K n i=1 � dV Eϕδ (yf (x)) n log + n δ � tEϕδ (yf (x)) n which concludes the proof. � The above lemma gives a result for a fixed d ≥ 1 and δ ∈ (0, 1]. To obtain a uniform result, it’s enough to consider δ ∈ Δ = {2−k , k ≥ 1} and d ∈ {1, 2, . . .}. For a fixed δ and d, use the Lemma above with tδ,d defined by e−tδ,d = e−t d26πδ2 . Then � P ∀f ∈ Fd , . . . + and ⎛ P⎝ � d,δ ∀f ∈ Fd , . . . + � tδ,d n � � ≥ 1 − e−tδ,d = 1 − e−t 6δ d2 π 2 �⎞ � tδ,d ⎠ 6δ ≥1− e−t 2 2 = 1 − e−t . n d π δ,d 53 Lecture 21 Bounds on the generalization error of voting classifiers. Since tδ,d = t + log 18.465 d2 π 2 6δ , � ⎞ ⎛� 2 2 t + log d 6δπ dV log nδ Eϕδ − ϕ¯δ ⎠ ∀f ∈ Fd , √ + ≤K⎝ n n Eϕδ �� � � � dV log nδ d2 π 2 t ≤K + log + n n 6δ �� � � n dV log t δ ≤ K� + n n � 2 2 dV log n δ since log d 6δπ , the penalty for union-bound, is much smaller than . n Recall the bound on the misclassification error n 1 � P (yf (x) ≤ 0) ≤ I(yi f (xi ) ≤ δ) + n i=1 If � � n 1� Eϕδ (yf (x)) − ϕδ (yi f (xi )) . n i=1 �n Eϕδ − n1 i=1 ϕδ √ ≤ ε, Eϕδ then n Eϕδ − ε � Eϕδ − 1� ϕδ ≤ 0. n i=1 Hence, � � �� � n � ε ε 2 1� Eϕδ ≤ + � + ϕδ 2 2 n i=1 Eϕδ ≤ 2 � ε �2 2 n +2 1� ϕδ . n i=1 The bound becomes ⎛ ⎞ n ⎜ 1 � dV n t⎟ ⎜ ⎟ P (yf (x) ≤ 0) ≤ K ⎜ I(yi f (xi ) ≤ δ) + log + ⎟ n δ n ⎝ n i=1 ⎠ � �� � (*) where K is a rough constant. (*) not satisfactory because in boosting the bound should get better when the number of functions grows. We prove a better bound in the next lecture. 54