1
ON A DESCENT ALGORITHM FOR THE
DISCRETE BEST 1 LINEAR SPLINE
APPROXIMATION PROBLEM
S.S. Papakonstantinou1 & I.C. Demetriou2
Abstract - Let there be given error-contaminated
measurements of function values at strictly ascending
abscissae. We address the problem of calculating a best
1
linear spline approximation to these data, whose
f ( xi ) . We assume that i  f ( xi )   i , where  i
is a random number. We also assume that there
are some gross errors in the data due to blunders.
knots are predefined on the abscissae. We establish
We seek a best
conditions that allow the development of a special
descent algorithm that takes into account the structure
 xi , i  ,
of the problem.
prescribed knots that form a subset of the
1
linear spline s ( x ) to the points
i  1, 2,..., n , defined on a number of
abcsissae. Therefore we develop a method that
Index Terms - data fitting, divided difference,
approximation,
least
absolute
deviation,
1
linear
minimizes the sum of the moduli of the errors
n
F ( s )   φi  si ,
programming, smoothing, spline
(1.1)
i 1
subject to the stated condition, where s is the
vector in
I. INTRODUCTION
We address the problem of providing a linear
spline approximation to measured values of a
function f ( x) . Specifically, the data are the pairs
 xi , i  ,
i  1, 2,..., n , where the abscissae xi ,
i  1, 2,..., n ,
x1  x2 
satisfy
the
inequalities
 xn , and  i is the measurement
Department of Economics, University of Athens,
8 Pesmazoglou Street, Athens 10559, Greece. E-mails:
sotpapak@econ.uoa.gr (SSP), demetri@econ.uoa.gr (ICD).
with components
si  s( xi ) ,
i  1, 2,..., n .
Let  j : j  0,1,..., m , where m  n , be the
knots of s ( x ) , which by definition form a subset
of the abscissae xi : i  1, 2,..., n , and where we
define 0  x1 and  m  xn . We can visualize the
required linear spline by joining the point
( j , s ( j ))
1, 2
n
to ( j 1 , s( j 1 )) with a line for
j  0,1,..., m  1 . Thus, the points
intervals 0 , 1  ,
 xi , si 
on the
,  m2 ,  m1  and m1 , m  are
2
collinear. Generally, if K is the set of the data
efficient
indices that define the knots, then an alternative
methods. Thus we require conditions that allow
expression of the required linear spline s ( x ) is
the development of such a method. In Section II,
that its components satisfy the constraints
we consider a transformation that reduces the
s  xi 1 , xi , xi 1   0 , i 1, n \ K ,
(1.2)
than
general
linear
programming
equality-constrained problem to an equivalent
unconstrained one with fewer variables. In
where
Section III we provide theoretical support for a
s  xi 1 
s  xi 1 , xi , xi 1  

 xi 1  xi  xi 1  xi 1 
descent method that is outlined in Section IV.
s  xi 
s  xi 1 

 xi  xi 1  xi  xi 1   xi 1  xi 1  xi 1  xi 
(1.3)
II. LINEAR B-SPLINES TRANSFORMATION
Let S be the n  A dimensional space defined
is the i th second divided difference of s ( x ) . For
example, if n  100 , m  3 and 0  x1 , 1  x35 ,
by the linear equality constraints (1.4), where A
 2  x86 and 3  x100 , then s  xi 1 , xi , xi 1   0 is
is the number of elements of A . We consider a
obtained, for i  2,...,34,36,...,85,87,...,99 .
in order to reduce the equality constrained
The discussion so far allows restating the
calculation
of
follows.
fewer variables (Demetriou 2004). We wonder
A  1, n \ K and we calculate si , i  1, 2,..., n , by
whether a basis can be found for S that will
minimizing (1.1) subject to the constraints
simplify the calculations. Indeed, if i  A , then
s  xi 1 , xi , xi 1   0 , i  A .
We
problem to an equivalent unconstrained one with
let
s( x)
as
transformation that makes use of these constraints
(1.4)
y[ xi 1 , xi , xi 1 ]  0
the condition
implies the
In view of the linearity of the constraint functions
collinearity of the points ( xi 1 , yi 1 ), ( xi , yi ) and
with respect to si , i  1, 2,..., n , and the form of
( xi 1 , yi 1 ) ,
the objective function (1.1), the calculation of s
{2,
may be expressed as a linear programming
i is the index of an interior knot of s ( x ) . It
problem (Hudley 1954, Wagner 1959). However
follows that the number of knots (including the
this will result in a very inefficient way for
end
obtaining the solution, due to the matrix
m  n | A | 1 , we denote the interior knots of
s( x)
if
i
is
an
integer
in
, n  1} \ A such that y[ xi 1 , xi , xi 1 ]  0 , then
points)
calculations required by the simplex method.
Therefore, because several thousand data points
but
by
is
n | A | .
Further,
k : k  1, 2,..., m 1 ,
we
set
arranged in
occur in many smoothing calculations, it is useful
ascending order, and as usual we set 0  x1 and
to develop a special technique that is more
 m  xn . A basis for the set of linear splines with
3
the given knots defined on {xi : i  1, 2,..., n} is
1967) that could be applied to our problem. Still,
derived from the functions {Bk : k  0,1,..., m} ,
we would like to develop a special method that
where Bk is the linear B-spline defined by (see
makes use of the structure of the problem.
Schumaker 1981: p.127)
III. LINEAR PROGRAMMING FORMULATION
( x   k 1 ) /( k   k 1 ),  k 1  x   k

Bk ( x)  ( k 1  x) /( k 1   k ),  k  x   k 1
0, otherwise,

The
linear
programming
formulation
of
minimizing (2.3) is as follows (see, for example,
Powell 1981). We replace i  si by ui  vi where
where we set  1   0 and  m1   m .
ui  max i  yi ,0 ,
Now the constraints
y  xi 1 , xi , xi 1   0, i  A ,
(2.1)
are automatically satisfied by s ( x ) and the
vi  max i  yi ,0 .
(3.1)
i  si  ui  vi
(3.2)
So
number of the parameters to be found is only
m  1  n  A . Therefore s ( x ) is written in the
and the problem becomes
form
minimize:
m
s ( x)   σ j B j ( x)
(2.2)
n
i
m
i
i 1
j 0
j
B j ( xi )
(3.3)
subject to
n
n
i
i
i 1
m
m
   s( x )     
i 1
i
i 1
and our problem is equivalent to finding the least
value of
n
 (u  v )     
j 0
i
j 0
j
B j ( xi )
(2.3)
vi  i    j B j ( xi )  ui , i  1, 2,..., n,
i 0
ui  0, vi  0, i  1, 2,..., n.
where
 j , j  0,..., m ,
(2.4)
One may of course apply any linear programming
are the variables of the minimization calculation.
Thus we have reduced the equality-constrained
problem of minimizing (1.1) subject to (1.4) to an
unconstrained calculation of m  1 variables. This
transformation is highly useful because there
exist general algorithms on the subject (for
algorithm to solve this problem, but we should
note that the use of a general linear programming
procedure would ignore some properties of our
best
1
fit. For example, it is well known (Dodge
1987) that the best
1
line fit is defined by means
example, Abdelmalek 1974, Barrodale & Roberts
of two interpolation points. Similarly, a best
1973, Barrodale & Roberts 1978, Bartels, Conn
from a linear subspace to the data satisfies certain
& Sinclair 1978, Osborne & Watson 1971, Usow
interpolation conditions.
1
fit
4
*
Then s is a best
Theorem 1 (for example, Rice 1964)
*
Let s be a best
1
1
fit if-f
n
 sign(  s )s   s ,
fit to  defined by the linear
i 1
i
iI
i  I  1, 2,..., n ,
subspace (2.1). Then it satisfies the interpolation
conditions
*
i
i
i





(3.7)
for all s defined on the subspace (2.1). ■
si*  i , i  I  1, 2,..., n ,
Therefore a best
where
I  dim[linear subspace defined by (2.1)].
1
fit may be calculated by
*
seeking a set I that allows an optimal s to be
Thus
obtained by interpolation to the points of I .
I  A  n. ■
(3.4)
Thus, we need a systematic method of search for
interpolation points and also it is necessary to test
For example, if we require the best
1
line fit to
whether a trial set of interpolation points gives a
best fit. However, condition (3.7) is not suitable
 , then (3.3) becomes
for use, because it has to be satisfied for every
n
minn
a , b , u
, v
n
 (u  v )
i 1
i
s  S , so it requires checking an infinite number
i
of approximations for a solution. Therefore, the
subject to
following theorem may be valuable in testing
ui  vi  i  axi  b , i  1, 2,..., n ,
whether a trial approximation is optimal.
ui  0, vi  0 , i  1, 2,..., n .
It follows that a best
1
line fit to i , i  1,..., n
Theorem 3 (Powell & Roberts 1980)
Let a trial set of interpolation conditions
requires
si*  i , i  I  1, 2,..., n .
I  n  A  n  (n  2)  2
interpolation points, where the dimension of the
linear subspace defined by (2.1) is exactly 2,
because A  2,3,..., n  1 in (2.1).
Then s * is a best

i1, n \ I
1
fit if-f
sign(i  si* ) j ( xi )  1 , j  0,1,..., m ,
where we make use of the cardinal functions of
our subspace
Theorem 2 (for example, Rice 1964)
*
Let s be a trial
1
fit from S to  that satisfies
j
( xi )   ji , j  0,1,..., m . ■
the interpolation conditions
si*  i , i  I  1, 2,..., n .
(3.6)
This theorem states that we can find out
whether a set of interpolation points gives a best
5
fit by testing only m  1 inequalities. Still, we
hold, then terminate because the conditions of
have
suitable
Theorem 3 are satisfied. Otherwise, there is a
interpolation points. Therefore our problem may
descent direction in the solution subspace, which
be expressed as follows: Starting from any guess
means that (3.3) may be reduced by replacing an
interpolation points, move in the solution
interpolation condition by another one. The
subspace (2.1) until the least value of (3.3) is
immediate question is on how to locate the new
found.
interpolation point. Then, the following theorem
the
problem
of
generating
is highly useful for this task.
IV. OUTLINE OF THE METHOD
Let an initial set of m  1 interpolation points
defined by (3.6). Then set
Theorem 4 (see, for example, Watson 1980)
Revisit our problem: minimize
n


ui  max 0, i    j B j ( xi )  
j 0


 , i  1, 2,...,n
m


vi  max 0,   j B j ( xi )  i  
 j 0

i 1
i   j B j ( xi ) , i  I  1, 2,..., n.
Let si* be the values of s ( x ) due to (2.2) after we
replace  j by ˆ j . If

i1, n \ I
sign(i  si* ) j ( xi )  1 , j  0,1,..., m ,
1
(4.1)
Bm ( x1 ) 

Bm ( x2 ) 
.


Bm ( xn ) 
Then  , in view of (2.2), is associated to a best
1
fit if-f there is w in
n
such that
BT w  0 ,
Calculate
m


ui  max 0, i   ˆ j B j ( xi )  
j 0


 , i  1, 2,...,n.
m


vi  max 0,  ˆ j B j ( xi )  i  
 j 0

B j ( xi )    B ,
si  i , i  I  1, 2,..., n .
m
and let ˆ j , j  0,1,..., m , be the solution.
j
Let
Solve the linear system of equations
j 0
j 0
 B0 ( x1 ) B1 ( x1 )

B ( x ) B1 ( x2 )
B 0 2


 B0 ( xn ) B1 ( xn )
m
i   j B j ( xi ) , i  I  1, 2,..., n ,
i
where B is the matrix
and
j 0
m
   
m
where
m


wi  1 , wi  sign i   j B j ( xi )  , i  I . ■
j 0


We see that only I components of w are
undefined, which are obtained by solving the
vector equation
BT w  0 .
6
If there exists an integer k such that wk  1 , then
it is possible to reduce the objective function (4.1)
by giving up the interpolation condition sk  k .
n
Define the search direction d in
by
dk  1, di  0, i  I , i  k ,
 w   B
iI
i
T
i
i
iI
An equivalent expression for this search direction
is to give up sk  k , so I is replaced by I \ k
i
T
i
i
n
 B
T
i
i
n
 B
m
1    j B j ( xi ), i  k ,
i 1
j 0
iI
ik
0

   BiT d    wi i  BiT    BiT d 
iI
ik
T
i
i
   BiT d  wk k  BkT    BkT d  
   BiT d 
  B    d 
m
0    j B j ( xi ), i  I \ k .

 wk k  BkT     wk BkT d 
i 1
ik
and we obtain



 wk k  BkT   


0
 w   B
d  x j 1 , x j , x j 1   0, j  A .

   BiT d    wi i  BiT    BiT d 
1
.
j 0
One more alternative expression is to give up
sk  k , so I is replaced by I \ k and
Therefore, starting from any guess set of
interpolation points, we have provided conditions
BkT d  1,
that allow the derivation of a new set of
BiT d  0, i  I , i  k ,
interpolation points that gives a lower value for
d  x j 1 , x j , x j 1   0, j  A,
the objective function. Termination of this
where BkT  [ Bk ( x1 ), Bk ( x2 ),..., Bk ( xn )]T . Then the
objective function
n
i 1
i
j 0
the finite number of choices of interpolation
points.
m
   
algorithm occurs at the required optimum, due to
j
B j ( xi )    B
1
along the direction    d obtains a strictly
We have presented an algorithm that provides a
smaller value. Indeed,
n
best
 (u  v ) 
  B 
 w   B
    wi i  BiT      wi BiT d 
i
i 1
iI
i
i
i
T
i
1
linear spline approximation to a finite
number of data points, where the knots of this
1
iI
V. DISCUSSION
iI
0
spline form a subset of the abscissae. The
algorithm starts from a guess set of interpolation
points
and
by
following
suitable
descent
directions terminates at an optimum in a finite
number of iterations. The algorithm may be
7
employed in intermediate stages of processes that
make use of linear splines calculations.
Powell, M.J.D. (1981) Approximation Theory and
Methods.
Cambridge
University
Press,
Cambridge, England.
References
Powell, M.J.D. and F.D.K. Roberts (1980): A
Abdelmalek, N.N. (1974): On the discrete linear
1
approximation and
1
solutions
of
overdetermined linear equations. J. Approx.
Theory, 11, 38-53
discrete characterization theorem for the
linear approximation problem. Journal of
Approximation Theory, 30, 173-179
Rice,
Barrodale I. and F.D.K. Roberts (1973): An
improved algorithm for discrete
1
linear
approximation. SIAM J. Numer. Anal., 10,
J.R.
(1964):
The
Approximation
Barrodale I. and F.D.K. Roberts (1978): An
efficient algorithm for discrete
Functions. Vol. 1, Addison-Wesley, Reading,
Massachusetts.
Schumaker, L.L. (1981): Spline Functions: Basic
1
linear
Usow, K.H. (1967): On
Computation
for
1
approximation II:
discrete
functions
approximation with linear constraints. SIAM J.
discretization effects.
Numer. Anal., 15, 603-611.
Numerical Analysis, Vol 4 (2), 233-244.
Bartels, R.H., Conn, A.R. and Sinclair, J.W.
Minimization
techniques
for
piecewise differentiable functions: The
1
solution to an overdetermined linear system.
Demetriou, I.C. (2004): Least squares convex data
smoothing.
Computational
Optimization and Applications, 29, 197-217.
Dodge, Y. (Ed) (1987): Special Issue on
Statistical Data Analysis Based on the
1
Norm and Related Methods, North-Holland.
Hudley,
G.
(1954):
Linear
Programming.
Addison-Wesley Co., London.
Osborne, M.R. and G.A. Watson (1971): On an
algorithm for nonlinear
Comp. J., 14, 184-188.
1
SIAM,
Journal
and
of
Wagner, H.M. (1959): Linear programming
techniques for regression analysis. J. Amer.
Stat. Assoc., 54, 206-212.
Watson, G.A. (1980): Approximation Theory and
Numerical Methods. J. Wiley & Sons,
SIAM J. Numer. Anal.,15, 224-241.
concave
of
Theory, J. Wiley & Sons, NY.
839-848.
(1978):
1
approximation.
Chichester.