Basis Expansion and Regularization Prof. Liqing Zhang Dept. Computer Science & Engineering, Shanghai Jiaotong University Outline • Piece-wise Polynomials and Splines • Wavelet Smoothing • Smoothing Splines • Automatic Selection of the Smoothing Parameters • Nonparametric Logistic Regression • Multidimensional Splines • Regularization and Reproducing Kernel Hilbert Spaces 2015/4/13 Basis Expansion and Regularization 2 Piece-wise Polynomials and Splines • Linear basis expansion N f ( x) m hm ( x) m 1 • Some basis functions that are widely used hm ( x) x hm ( x) x p hm ( x) log(x) hm ( x) sin(mx); cos(mx) 2015/4/13 Basis Expansion and Regularization 3 Regularization • Three approaches for controlling the complexity of the model. f ( x) h ( x) – Restriction N – Selection – Regularization: k k k 1 m y k hk ( x) k 1 M min i 1 M min i 1 M m i 1 k 1 (i ) yi k hk ( xi ) 2 m 2 2 yi k hk ( xi ) J ( ) k 1 J ( ) ? 2 2015/4/13 Basis Expansion and Regularization 4 Piecewise Polynomials and Splines h1 X I X 1 , h2 X I 1 X 2 h3 X I 2 X ; h1 X I X 1 , h2 X I 1 X 2 h3 X I 2 X , hm 3 X hm X X ; h1 X 1, h2 X X , h3 X X 1 , h4 X X 2 2015/4/13 Basis Expansion and Regularization 5 Piecewise Cubic Polynomials • Increasing orders of continuity at the knots. • A cubic spline with knots at 1 and 2 : 1, X , X 2 , X 3 , X 1 3 , X 2 3 ; • Cubic spline truncated power basis 2015/4/13 Basis Expansion and Regularization 6 Piecewise Cubic Polynomials • An order-M spline with knots , j=1,…,K is a piecewise-polynomial of order M, and has continuous derivatives up to order M-2. • A cubic spline has M=4. • Truncated power basis set: hj ( X ) X j 1 , j 1, , M hM l ( X ) ( X l ) M 1 , l 1,, M 2015/4/13 Basis Expansion and Regularization 7 Natural cubic spline • Natural cubic spline adds additional constraints, namely that the function is linear beyond the boundary knots. Natural boundary constraints Linear Cubic Cubic 1 2015/4/13 Linear 2 3 Basis Expansion and Regularization 8 B-spline • The augmented knot sequenceτ: 1 2 M 0 ; jM j , j 1,, K ; K 1 K M 1 K M 2 K 2 M • Bi,m(x), the i-th B-spline basis function of order m for the knot-sequenceτ, m≤M. 1 i x i 1 Bi ,1 ( x) i 1,, K 2M m 0 others x i im x Bi ,m ( x) Bi.m 1 ( x) Bi 1,m 1 ( x) i m 1 i i m i 1 2015/4/13 Basis Expansion and Regularization 9 B-spline • The sequence of B-spline up to order 4 with ten knots evenly spaced from 0 to 1 • The B-spline have local support; they are nonzero on an interval spanned by M+1 knots. 2015/4/13 Basis Expansion and Regularization 10 Smoothing Splines • Base on the spline basisN method: m f ( x) k hk ( x) • So y k hk ( x) , is the noise. k 1 k 1 • Minimize the penalized residual sum of squares N RSS( f , ) yi f ( xi ) f ' ' (t ) dt 2 2 i 1 is a fixed smoothing parameter 0 : f can be any function that interpolates the data : the simple least squares line fit 2015/4/13 Basis Expansion and Regularization 11 Smoothing Splines N • The solution is a natural spline: f ( x) N j ( x) j j 1 • Then the criterion reduces to: T RSS( , ) y N y N T N – where N {N j ( xi ) }; Nij N i" (t ) N "j (t )dt • So the solution: ˆ ( N T N N )1 N T y • The fitted smoothing spline: N fˆ ( x) N j ( x)ˆ j j 1 2015/4/13 Basis Expansion and Regularization 12 Smoothing Splines 脊骨BMD—骨质密度 2015/4/13 • Function of age, that response the relative change in bone mineral density measured at the spline in adolescents • Separate smoothing splines fit the males and females, 0.00022 • 12 degrees of freedom Basis Expansion and Regularization 13 Smoothing Matrix • fˆ the N-vector of fitted values fˆ N ( N T N N )1 N T y S y • The finite linear operator S — the smoother matrix • Compare with the linear operator in the LS-fitting: M cubic-spline basis functions, knot sequenceξ T T fˆ B ( B B ) 1 B y H y, B is N M matrix • Similarities and differences: – Both are symmetric, positive semidefinite matrices – H H H idempotent(幂等的) ; S S S shrinking – rank: r (S ) N , r ( H ) M 2015/4/13 Basis Expansion and Regularization 14 Smoothing Matrix • Effective degrees of freedom of a smoothing spline df trace(S ) • S in the Reinsch form: S (I K )1 2 T ˆ f S y min y f f Kf • Since , solution: f • S is symmetric and has a real eigen-decomposition N S k ( )uk ukT k 1 k ( ) 1 1 d k – d k is the corresponding eigenvalue of K 2015/4/13 Basis Expansion and Regularization 15 • Smoothing spline fit of ozone(臭氧) concentration versus Daggot pressure gradient. • Smoothing parameter df=5 and df=10. • The 3rd to 6th eigenvectors of the spline smoothing matrices 2015/4/13 Basis Expansion and Regularization 16 • The smoother matrix for a smoothing spline is nearly banded, indicating an equivalent kernel with local support. 2015/4/13 Basis Expansion and Regularization 17 Bias-Variance Tradeoff • Example: Y f (X ) sin(12( X 0.2)) f (X ) X 0.2 • For fˆ S y, then cov(fˆ ) S cov(y)ST S ST • The diagonal contains the pointwise variances at the training xi • Bias is given by Bias( fˆ ) f E( fˆ ) f S f • f is the (unknown) vector of evaluations of the true f 2015/4/13 Basis Expansion and Regularization 18 Bias-Variance Tradeoff • df=5, bias high, standard error band narrow • df=9, bias slight, variance not increased appreciably • df=15, over learning, standard error widen 2015/4/13 Basis Expansion and Regularization 19 Bias-Variance Tradeoff • The integrated squared prediction error (EPE) The EPE and CV combines both bias and variancecurves in a single have the a summary: EPE( fˆ ) E (Y fˆ ( X ))2 similar shape. And, Var(Y ) E Bias ( fˆ ( Xoverall )) Varthe ( fˆ (CV X )) curve is 2 ˆ MSE( f ) approximately • N fold (leave one) cross-validation: N unbiased as an i 2 ˆ ˆ CV ( f ) ( y f ( xi ))estimate of the i 1 EPE curve 2 N ˆ (x ) y f i i i 1 1 S (i, i ) 2015/4/13 Basis Expansion and Regularization 20 Logistic Regression • Logistic regression with a single quantitative P r(Y 1 | X x) input X log f ( x) P r(Y 0 | X x) e f ( x) P r(Y 1 | X x) 1 e f ( x) • The penalized log-likelihood criterion N 1 l ( f ; ) yi log p( xi ) (1 yi ) log(1 p( xi )) { f " (t )}2 dt 2 i 1 N yi f ( xi ) log(1 e i 1 2015/4/13 f ( xi ) 1 ) { f " (t )}2 dt 2 Basis Expansion and Regularization 21 Multidimensional Splines • Tensor product basis – The M1×M2 dimensional tensor product basis g jk ( X ) h1 j ( X1 )h2k ( X 2 ), j 1,, M1, k 1,, M 2 – h1 j ( X1 ), basis function for coordinate X1 – h2k ( X 2 ), basis function for coordinate X2 M1 M 2 g ( X ) jk g jk ( X ) j 1 k 1 2015/4/13 Basis Expansion and Regularization 22 Tenor product basis of B-splines, some selected pairs 2015/4/13 Basis Expansion and Regularization 23 Multidimensional Splines • High dimension smoothing Splines N min f 2 y f ( x ) J f , i i i 1 xi IRd – J is an appropriate penalty function 2 f ( x) 2 2 f ( x) 2 2 f ( x) 2 2 dx1dx2 J f 2 2 2 IR x1 x1x2 x2 a smooth two-dimensional surface, a thin-plate spline. • The solution has the form N f ( x) 0 x j h j ( x) T j 1 2015/4/13 Basis Expansion and Regularization 24 Multidimensional Splines • The decision boundary of an additive logistic regression model. Using natural splines in each of two coordinates. • df = 1 +(4-1) + (41) = 7 2015/4/13 Basis Expansion and Regularization 25 Multidimensional Splines • The results of using a tensor product of natural spline basis in each coordinate. • df = 4 x 4 = 16 2015/4/13 Basis Expansion and Regularization 26 Multidimensional Splines • A thin-plate spline fit to the heart disease data. • The data points are indicated, as well as the lattice of points used as knots. 2015/4/13 Basis Expansion and Regularization 27 Reproducing Kernel Hilbert space • A regularization problems has the form: N min L( yi , f ( xi )) J ( f ) f H i 1 J( f ) ~ 2 f ( s) ds ~ G( s) – L(y,f(x)) is a loss-function. – J(f) is a penalty functional, and H is a space of functions on which J(f) is defined. • The solution K N k 1 i 1 f ( x) kk ( X ) i G( X xi ) – k span the null space of the penalty functional J 2015/4/13 Basis Expansion and Regularization 28 Spaces of Functions Generated by Kernel • Important subclass are generated by the positive kernel K(x,y). • The corresponding space of functions Hk is called reproducing kernel Hilbert space. • Suppose thatK has an eigen-expansion K ( x, y) ii ( x)i ( y), i 0, 2 i 1 i i 1 • Elements of H have an expansion f ( x) cii ( x), i 1 2015/4/13 f 2 Hk ˆ ci2 / i i 1 Basis Expansion and Regularization 29 Spaces of Functions Generated by Kernel • The regularization problem become N 2 min L( yi , f ( xi )) f H k f H k i 1 N 2 min L( yi , c j j ( xi )) c j / j C j 1 i 1 j 1 j 1 • The finite-dimension solution(Wahba,1990) N f ( x) i K ( x, xi ) i 1 • Reproducing properties of kernel function K (, xi ), f H K f ( xi ), K (, xi ), K (, x j ) K ( xi , x j ) 2015/4/13 Basis Expansion and Regularization 30 Spaces of Functions Generated by Kernel • N f ( x) i K ( x, xi ) i 1 • The penalty functional N N J ( f ) K ( xi , x j ) i j i 1 j 1 • The regularization function reduces to a finite-dimensional criterion min L( y, K ) T K , K K ( xi , x j ) – K is NxN matrix, the ij-th entry K(xi, xj) 2015/4/13 Basis Expansion and Regularization 31 RKHS • Penalized least squares min ( y K ) ( y K ) K T T • The solution of : ˆ ( K I )T y • The fitted values: N fˆ ( x) ˆ k K ( x, xk ) k 1 • The vector of N fitted value is given by fˆ Kˆ K ( K I ) 1 y ( I K 1 ) 1 y 2015/4/13 Basis Expansion and Regularization 32 Example of RKHS • Polynomial regression – Suppose h( x) : IR p IRM M huge – Given x1, x2 ,, xN , with M N , H {h j ( xi )} – Loss function: R( ) ( y H )T ( y H ) T L( )polynomial – The penalty 0 H T ( yregression: Hˆ ) ˆ 0 2 M M N minM yi T m ˆhm ( xi ) ˆ m2 { m }1 HH ( y H ) H 0 i 1 m 1 m 1 The solution H: ( HH T I ) 1 HH T y {HH } : h( x ), h( x ) K ( x , x ) fˆ ( x) h( x)T ˆ i K (i x, xi ),j ˆ ( Ki j I ) 1 y TN 2015/4/13 i 1 Basis Expansion and Regularization 33 Penalized Polynomial Regression pd • Kernel: K ( x, x) (1 x, x ) has M d eigen-functions d • E.g. d=2, p=2: K ( x, x) (1 x1 x1 x2 x2 )2 h( x )T h( x ) 2 2 1 2 x1 x1 2 x2 x2 ( x1 x1 ) ( x2 x2 ) 2 x1 x1x2 x2 h( x ) (1, 2 x1 , 2 x2 , x12 , x22 , 2 x1 x2 ) • The penalty polynomial regression: 2 M M 2 minM yi m hm ( xi ) m { m }1 i 1 m 1 m 1 N 2015/4/13 Basis Expansion and Regularization 34 RBF kernel & SVM kernel • Gaussian Radial Basis2 Functions x y / 2 2 K ( x, y ) e ; h( x j ) K ( x, x j ); j 1,...,M • Support Vector Machines N f ( x ) 0 j K ( x, x j ) j 1 N T min 1 yi f ( xi ) K 0 , i 1 2015/4/13 Basis Expansion and Regularization 35 Wavelet smoothing • Another type of bases——Wavelet bases • Wavelet bases are generated by translations and dilations of a single scaling function (x). • If ( x) I ( x [0 1]) ,then 0,k ( x) ( x k ) generates an orthonormal basis for functions with jumps at the integers. • 0,k ( x) form a space called reference space V0 • The dilations 1,k ( x) 2 (2 x k ) form an orthonormal basis for a space V1 V0 • Generally, we have V1 V0 V1 2015/4/13 Basis Expansion and Regularization 36 j ,k ( x) 2 j / 2 (2 j x k ) V0 V1 V2 V j 1 V j W j W j 信号的细节,正交于V j ( x) (2 x) (2 x 1) W0上的小波基 j ,k ( x) 2 j / 2 (2 j x k ) W j 上的小波基 2015/4/13 Basis Expansion and Regularization 37 2015/4/13 Basis Expansion and Regularization 38 Wavelet smoothing • The L2 space dividing V j 1 V j W j V j 1 W j 1 W j V0 W0 W1 W j V 2V1 V0 W3 W 2 V1 W1 W0 V1 /V0 V2 • Mother wavelet ( x) (2 x) (2 x k ) generate function 0,k ( x k ) form an orthonormal basis for W0. Likewise j ,k 2 j / 2 (2 j x k ) form a basis for Wj. 2015/4/13 Basis Expansion and Regularization 39 Wavelet smoothing • Wavelet basis on W0 : ( x) (2 x) (2 x 1) • Wavelet basis on Wj : j ,k ( x) 2 j / 2 (2 j x k ) • The symmlet-p wavelet: – A support of 2p-1 consecutive intervals. – p vanishing moments: j ( x ) x dx 0, 2015/4/13 j 1, , p Basis Expansion and Regularization 40 2015/4/13 Basis Expansion and Regularization 41 2015/4/13 Basis Expansion and Regularization 42 2015/4/13 Basis Expansion and Regularization 43 Adaptive Wavelet Filtering • Wavelet transform: y* W T y – y: response vector, W: NxN orthonormal wavelet basis matrix • Stein Unbiased Risk Estimation (SURE) min y W – The solution: 2 2 2 1 ˆ j sign( y j )(| y j | ) – Fitted function is given by inverse wavelet transform: fˆ Wˆ , LS coefficients truncated to 0 – Simple choice for : 2 log N 2015/4/13 Basis Expansion and Regularization 44 2015/4/13 Basis Expansion and Regularization 45