Robust Nonparametric Regression
by Controlling Sparsity
Gonzalo Mateos and Georgios B. Giannakis
ECE Department, University of Minnesota
Acknowledgments: NSF grants no. CCF-0830480, 1016605
EECS-0824007, 1002180
May 24, 2011
1
Nonparametric regression
Given
, function estimation allows predicting
Estimate unknown
from a training data set
If one trusts data more than any parametric model
Then go nonparametric regression:
lives in a (possibly
-dimensional) space of “smooth’’ functions
Ill-posed problem
Workaround: regularization [Tikhonov’77], [Wahba’90]
RKHS
with reproducing kernel
and norm
Our focus
Nonparametric regression robust against outliers
Robustness by controlling sparsity
2
Our work in context
Noteworthy applications
Load curve data cleansing [Chen et al’10]
Spline-based PSD cartography [Bazerque et al’09]
Robust nonparametric regression
Huber’s function [Zhu et al’08]
No systematic way to select thresholds
Robustness and sparsity in linear (parametric) regression
Huber’s M-type estimator as Lasso [Fuchs‘99]; contamination model
Bayesian framework [Jin-Rao‘10][Mitra et al’10]; rigid choice of
3
Variational LTS
Least-trimmed squares (LTS) regression [Rousseeuw’87]
Variational (V)LTS counterpart
(VLTS)
is the -th order statistic among
residuals discarded
Q: How should we go about minimizing
?
(VLTS) is nonconvex; existence of minimizer(s)?
A: Try all
subsamples of size , solve, and pick the best
Simple but intractable beyond small problems
4
Modeling outliers
Outlier variables
Nominal data obey
s.t.
outlier
otherwise
; outliers something else
Remarks
Both
and
are unknown
If outliers sporadic, then vector is sparse!
Natural (but intractable) nonconvex estimator
5
VLTS as sparse regression
Lagrangian form
(P0)
Tuning parameter
controls sparsity in
Proposition 1: If
solves (P0) with
then
solves (VLTS) too.
number of outliers
chosen s.t.
,
The equivalence
Formally justifies the regression model and its estimator (P0)
Ties sparse regression with robust estimation
6
Just relax!
(P0) is NP-hard
relax
(P1)
(P1) convex, and thus efficiently solved
Role of sparsity controlling
is central
Q: Does (P1) yield robust estimates ?
A: Yap! Huber estimator is a special case
where
7
Alternating minimization
(P1) jointly convex in
AM solver
(P1)
Remarks
Single Cholesky factorization of
Soft-thresholding
Reveals the intertwining between
Outlier identification
Function estimation with outlier compensated data
8
Lassoing outliers
Alternative to AM
Proposition 2: Minimizers
solve Lasso [Tibshirani’94]
of (P1) are fully determined by
w/
as
and
, with
Enables effective methods to select
Lasso solvers return entire robustification path (RP)
Cross-validation (CV) fails with multiple outliers [Hampel’86]
9
Robustification paths
LARS returns whole RP [Efron’03]
Same cost of a single LS fit (
)
Coeffs.
Lasso path of solutions is piecewise linear
Lasso is simple in the scalar case
Coordinate descent is fast! [Friedman ‘07]
Exploits warm starts, sparsity
Other solvers: SpaRSA [Wright et al’09], SPAMS [Mairal et al’10]
Leverage these solvers
values of
For each ,
consider 2-D grid
values of
10
Selecting
and
Relies on RP and knowledge on the data model
Number of outliers known: from RP, obtain range of
Discard outliers (known), and use CV to determine
s.t.
Variance of the nominal noise known: from RP, for each
grid, obtain an entry of the
sample variance matrix
The best
.
on the
as
are s.t.
Variance of the nominal noise unknown: replace
above with a
robust estimate
, e.g., median absolute deviation (MAD)
11
Nonconvex regularization
Nonconvex penalty terms approximate
better in (P0)
Options: SCAD [Fan-Li’01], or sum-of-logs [Candes et al’08]
Iterative linearization-minimization of
around
Remarks
Initialize with
, use
and
Bias reduction (cf. adaptive Lasso [Zou’06])
12
Robust thin-plate splines
Specialize to thin-plate splines [Duchon’77], [Wahba’80]
Smoothing penalty only a seminorm in
Solution:
Radial basis function
Augment w/ member of the nullspace of
Given , unknowns
found in closed form
Still, Proposition 2 holds for appropriate
13
Simulation setup
Training set
: noisy samples of Gaussian mixture
examples,
i.i.d.
Outliers:
Nominal:
True function
i.i.d. for
w/
i.i.d. (
known)
Data
14
Robustification paths
Grid parameters:
grid:
grid:
Paths obtained using SpaRSA [Wright et al’09]
Outlier
Inlier
15
Results
True function
Robust predictions
Nonrobust predictions
Refined predictions
Effectiveness in rejecting outliers is apparent
16
Generalization capability
In all cases, 100% outlier identification success rate
Figures of merit
Training error:
Test error:
Nonconvex refinement leads to consistently lower
17
Load curve data cleansing
Load curve: electric power consumption recorded periodically
Reliable data: key to realize smart grid vision
B-splines for load curve prediction and denoising [Chen et al ’10]
Deviation from nominal models (outliers)
Faulty meters, communication errors
Unscheduled maintenance, strikes, sporting events
Uruguay’s aggregate power consumption (MW)
18
Real data tests
Robust predictions
Nonrobust predictions
Refined predictions
19
Concluding summary
Robust nonparametric regression
VLTS as
-(pseudo)norm regularized regression (NP-hard)
Convex relaxation
variational M-type estimator
Lasso
Controlling sparsity amounts to controlling number of outliers
Sparsity controlling role of
is central
Selection of
using the Lasso robustification paths
Different options dictated by available knowledge on the data model
Refinement via nonconvex penalty terms
Bias reduction and improved generalization capability
Real data tests for load curve cleansing
20