Project Proposal
A Recursive Method for the Solution of the Linear Least Squares
Formulation – algorithm and performance in connection with the PLX
instruction set
Shyam Bharat
Claus Benjaminsen
The least squares method of parameter estimation seeks to minimize the squared
error between the observed data sequence and the assumed signal model, which is some
function of the parameter to be estimated. In particular, the linear least squares
formulation represents the case where the signal model is a linear function of the
parameter to be estimated. The salient feature of the least squares method is that no
probabilistic assumptions are made about the data – only a signal model is assumed. This
method is generally used in situations where a precise statistical characterization of the
data is unknown.
The linear least squares estimation (LLSE) method ultimately boils down to
solving a set of linear equations. As will be seen from the equations that follow, the
solution to the LLSE involves the computation of the inverse of a matrix. This matrix
happens to be the autocorrelation matrix of the assumed signal model matrix. Now, the
dimensions of this autocorrelation matrix are P x P, where ‘P’ is the number of samples
in the observation vector. This computation of the matrix inverse is straight-forward for
small values of ‘p’ but for large values of ‘p’, it becomes increasingly computationally
intensive. In digital signal processing (DSP) applications, the computation is generally
required to be done in real-time for a continuous succession of samples. In other words,
for DSP applications, p  ∞. It is practically impossible to compute the inverse of a
matrix consisting of infinite entries! Hence, it is essential to find an alternate way of
solving the system of linear equations to yield the unknown parameters. This problem is a
fairly well-researched one and there exists an alternate method of solving for the
unknown parameters. This alternate method basically involves updating a so-called
weight vector in time, based on its value at the previous time instant. The meaning of this
statement will become clearer with the aid of equations presented below.
The function to be minimized with respect to the parameter at the present instant
is given by:
n
J(w(n)) =

nk
| en (k ) |2
k=1
where en(k) = d(k) – wH(n)u(k)
In matrix form, this function can be written as:
,0<λ<1
J(w(n)) = [d* - uHw(n)]HΛ[d – uHw(n)]
where Λ = diag{λn-1, λn-2, … , λ, 1}
The solution to this equation is given by:
w(n) = (u Λ uH)-1 u Λ d
or w(n) = Φ(n)-1ө(n)
where:
n
 (n)    n  k u (k )u (k )
H
k 1
n
 (n)    n  k u (k )d *(k )
k 1
As can be observed from the above equations, the path to the solution involves the
computation of the inverse of a matrix, for every value of ‘n’. The method of recursive
least squares seeks to circumvent this computationally intensive step by instead
calculating w(n) from w(n-1), it’s value at the previous time instant.
Φ(n) and ө(n) can be rewritten in recursive form as follows:
Φ(n) = λΦ(n-1) + u(n)uH(n)
ө(n) = λө(n-1) + u(n)d*(n)
The matrix inversion lemma shown below is applied to Φ(n)
1
( A  UV ) 1  A 
H
1
H
1
A UV A
H
1
1V A U
Therefore,
1
1
 (n)   1  (n  1) 
 2  1 (n  1)u (n)u H (n) 1 (n  1)
H
1
1   1 u (n) (n  1)u (n)
At this point, the following 2 new terms are defined:
P(n) = Φ-1(n) , where P(n) has dimensions M x M
k ( n) 
 1 P(n  1)u (n)
1   1 u (n) P(n  1)u (n)
H
It can be seen that:
, where k(n) is an M x 1 gain vector
k(n) = P(n)u(n)
If we proceed with the math, we ultimately arrive at the following recursion formula for
w(n):
w(n) = w(n-1) + k(n)[d*(n) – uH(n)w(n-1)]
The last term (in square brackets) is denoted as α*(n), where α(n) = d(n) – wH(n-1)u(n) is
called the ‘innovation’ or ‘apriori estimation error’. This differs from e(n) which is the
‘aposteriori estimation error’.
To update w(n-1) to w(n), we need k(n) and α(n)
Based on the above formulation, shown below is an algorithm for solving the least
squares equation recursively:
Initialization: P(0) = δ-1I
w(0) = 0
where δ is small and positive
For n = 1,2,3,….
x(n) = λ-1P(n-1)u(n)
k(n) = [1 + uH(n)x(n)]-1x(n)
α(n) = d(n) – wH(n-1)u(n)
w(n) = w(n-1) + k(n)α*(n)
P(n) = λ-1P(n-1) – k(n)xH(n)
In order to successfully implement this algorithm the expressions written in the
loop above needs to be analyzed. In this project we will therefore investigate what kind of
computation is required in the algorithm, and whether anything can be simplified? What
are the different dependencies among the variables and the sequential statements? Can
any of the dependencies be removed, is there anything to gain by rewriting into single
assignment format, can the computations be done in parallel?
The next thing we will look into is the numerical properties of the algorithm. Is
the algorithm stable and/or bounded, or can it become unstable and unbounded? What are
the requirements which insure that it works correctly? How does the values grow during
the calculations, what should be the word length of inputs, intermediate results and
outputs? Is it helpful to scale any of the variables used in the computations and what kind
of arithmetic should be used signed/unsigned – saturation/wrap around?
Finally we will look into how to actually implement the algorithm using the PLX
instruction set. How should the assembly code be written, what kind of instructions are
required? Are all necessary instructions available or would other non supported
instructions be helpful? How should the data be stored and updated? What is the speed of
the implementations, how many inputs can be processed per second? Can the algorithm
be executed in real time?
References:
Simon Haykin; Adaptive Filter Theory, Prentice Hall, 2002
Fixed and Floating Point Error Analysis of QRD-RLS and STAR-RLS Adaptive Filters,
Kalavai J. Raghunath; Keshab K. Parhi
Acoustics, Speech, and Signal Processing, 1994. ICASSP-94.,
Volume iii, 19-22 April 1994 Page(s):III/81 - III/84 vol.3