Total Least Squares Estimation of Dynamical Systems
Manoranjan Majji∗ and John L. Junkins
†
Texas A & M University, College Station, TX, 77843-3141.
The Total least squares error criterion is considered for estimation problems. Exact necessary conditions for the total error criterion have been derived. Several solution methodologies are presented to solve the modified normal equations obtained from the necessary
conditions. The results are applied on the parameter identification of a novel morphing
wing developed at Texas A & M University (Static Problem). Subsequently, a filter is
derived, with a multilinear measurement model, whose minimum variance estimator is
shown to also minimize the total least squares type cost function. The filter thus derived
is compared with a classical Kalman filter on a numerical example.
I.
Introduction
The least squares error criterion was invented by Carl Friedrich Gauss which till to this day remains the
most widely used in many diverse areas owing to its computational and statistical properties. During the
turn of this century, this criterion was generalized to include uncertainty in the basis functions by Adcock.1
Consequently, a new theory was developed called the “errors in variables” theory.2 Golub and Van loan3
applied this to numerical mathematics problems. Sabine Van Huffel and Van de Valle4 applied the theory
developed by Gleser to parameter estimation problems and brought the theory to systems science.
I.A.
The Total Least Squares Error Criterion
The total least squares error criterion is based on generalization of the least squares error criterion. It aims
at changing the basis by the slightest possible (as small as possible) to capture as much of the measurement
vector as possible.
I.A.1.
Paper Outline
The paper is presented as follows. First section introduces the total least squares error criterion and derives
the necessary conditions in a direct manner, obtaining a modest set of nonlinear equations that are a
modification of the least squares solution. Geometrical insights in to the problem are presented and a
comparison is made with existing literature. Methods to solve the equations thus obtained are discussed in
the next section. Two novel approaches are presented along with the celebrated SVD solution by Golub et.
al,3 Van Huffel,4 and the well known eigenvalue problem proposed by Villegas.5 We have to note the strong
correlation of the concepts developed herein to the Minimum Model error estimation proposed by Mook and
Junkins.6 Subjecting the error criterion to dynamical systems is the topic of section (VI). Subsequent section
(VI) compares the performance of the filter derived in this section with the classical Kalman estimator.
II.
Total Least Squares
As pointed out in the introduction, the least squares error criterion for minimization of residual error
does not apply when linear equations are involved with an uncertainty in the basis function. Therefore, if we
only have access to measurements Ã,of basis functions and ỹ of the range vector, in the linear error model
ỹ ≈ Ãx + v
∗ Graduate
(1)
Student, Department of Aerospace Engineering, 3141 TAMU, and Student Member, AIAA.
Professor, Holder of Royce E. Wisenbaker Chair in Engineering, Regents Professor, Department of Aerospace
Engineering, Fellow, AIAA.
† Distinguished
1 of 19
American Institute of Aeronautics and Astronautics
where à = A + VA and VA is a matrix of random variables ∼ N (0, σ) and v is the vector of random variables
∼ N (0, σ). We do not require the errors to have same statistics, however for simplicity of the derivations,
we would like to impose these conditions. Straight-forward Generalizations to arbitrary statistics can be
performed with arbitrary weighting as shown by Van Huffel4 and Glesler.2 Though the simplicity of our
approach in obtaining these results makes this generalization obvious and natural, we choose not to present
the general method in favor of clarity. However we do summarize the results obtained for general weight
matrices in a separate section. So the Total Least Squares error criterion minimizes the cost associated with
estimation errors in basis defined by ∆ := Ã − Â, r = ỹ − ŷ
1
1
tr(∆T ∆) + rT r
2
2
J=
(2)
the necessary conditions for a minimum of these equations are given by
∂J
∂ Â
= 0,
∂J
=0
∂ x̂
(3)
leading to the equations (using matrix derivative identities7 ),
∆(I + x̂x̂T ) = (Ãx̂ − ỹ)x̂T
(4)
which, (using the Morrison Sherman Woodbury Matrix Inversion Lemma,7 ) is equivalent to
 = à −
(Ã − ỹ)xT
= Ã + ex̂T
(1 + x̂T x̂)
(5)
Ãx̂)
where e := (ỹ−
1+x̂T x̂ , leading to the fact that the optimal correction in the data matrix, is the rank one
T
correction ex̂ . The second necessity that ∂J
∂ x̂ = 0 yields
(ÂT Â)x̂ = ÂT ỹ
which, by using the expression  = à + ex̂T yields,
"
#
(ỹ − Ãx̂)T (ỹ − Ãx̂)I
T
à à −
x̂ = ÃT ỹ
(1 + x̂T x̂)
(6)
(7)
which we call the modified normal equations. We found that these equations are same as the ones derived
by Van Huffel.4 Substituting the necessary conditions in to the cost function, we get the optimal cost to be
J1 =
(ỹ − Ãx̂)T ∗ (ỹ − Ãx̂)
(1 + x̂T x̂)
(8)
The solution to the necessary conditions derived here is a nonlinear problem but the solution that minimizes
J
corresponding to the smallest eigenvalue of the symmetric positive definite form T T T =
"1 is the eigenvector
#
ÃT Ã ÃT ỹ
, T being defined as T = [Ã . . . ỹ]. Defining an associated vector z = [x̂T − 1]T the minimum
ỹ T Ã ỹ T ỹ
value of J1 hence becomes (after making necessary substitutions of the necessary conditions),
J1 =
zT T T T z
(ỹ − Ãx̂)T (ỹ − Ãx̂)
=
(1 + x̂T x̂)
zT z
(9)
whose extremals are eigenvalues associated with the quadratic form T T T , called the Rayleigh quotient.8 The
associated eigenvectors are the solutions of the problem and in this problem, the smallest eigenvalue and the
corresponding eigenvector are the solutions.
2 of 19
American Institute of Aeronautics and Astronautics
II.A.
Geometry of the problem
By a careful analysis of the necessary conditions for a solution to the total least squares problem, we can
make some observation on the geometry of the problem. Consider the space of rectangular matrices with
fixed order n × m together with the inner product definition < A, B >: tr(AT B). The norm derived from
this inner product definition satisfies the polarization identity, as shown below
kP + Qk2 + kP − Qk2 = tr((P + Q)T (P + Q)) + tr((P − Q)T (P − Q))
= 2tr(P T P ) + 2tr(QT Q) = 2(kAk2 + kBk2 )
(10)
(11)
A result in analysis9 states that a norm satisfies the polarization identity is if and only if it is derived from an
inner product and is unique. This allows us to define orthogonality in matrix spaces. Now consider the inner
products ŷ T (ỹ − Âx̂) and tr(ÂT ∆). These are the inner products of the best estimates with the residual
object in the corresponding space. Then,
ŷ T (ỹ − Âx̂) = x̂T ÂT (ỹ − Âx̂) = 0
(12)
T
(13)
T
T
T
tr(Â ∆) = −tr(Â ex̂ ) = −Â e = 0
In the above expressions, the identity ÂT e = 0 has been used. This identity directly follows from the
necessary conditions ÂT (Âx̂ − ỹ) = ÂT (Ãx̂ − ỹ − ex̂T x̂) = ÂT e = 0. Therefore, the total least squares
problem enforces a geometry and performs an orthogonal regression in both range space and the space of
the basis functions.
III.
Weighted Total Least Squares
Researchers have claimed equivalence of the weighted total least squares problems to an error criterion
given by Golub and Van loan,8 and Van Huffel.4 However there is not sufficient freedom nor insightful
conclusions from the resulting advantages unless there is a symbolic expression for the estimate. In this
section we show that our technique yields a more general necessary condition that gives a lot of design
freedom to alter the weights. So the cost function with appropriate weights is given by
Jw = tr(∆T P ∆) + rT Qr
(14)
where P, Q are arbitrary positive definite weight matrices. They allow the designer to control the magnitudes
w
of correction of the range and basis function tolerance levels. The first order necessary conditions ∂J
= 0,
∂ Â
∂Jw
∂ x̂ = 0 yield
P ∆ + Q∆x̂x̂T = Q(Ãx̂ − ỹ)x̂T
T
−1
x̂ = (Â QÂ)
T
 Qỹ
(15)
(16)
Clearly, there is no obvious way of determining the “best” correction for giving an expression for Â. But
indeed there is. We choose
 = à + P −1 (Q−1 + P −1 (x̂T x̂))−1 (ỹ − Ãx̂)x̂T
(17)
We can easily verify that this expression for  satisfies the first necessary condition. Upon substitution
in to the second condition, we get nothing similar to the nice eigenvalue problem. We observe that if the
range measurements are weighted more Q >> P , then the best correction to  is applied and we get
T
and when P >> Q, we receive no correction in ÂP →inf = Ã, which is the least
ÂQ→inf = Ã + (ỹ−x̂Ãx̂)x̂
T x̂
T
Ãx̂)x̂
squares solution. In between, when there is equal uncertainty in both, we have P ∼ Q, Â = Ã + α (ỹ−
1+x̂T x̂ ,
which is the eigenvalue problem presented above. This design freedom via weighted total least squares
solution is not present in the literature to the best knowledge of the authors. However the pay off is the lack
of algorithms to solve this problem. Therefore we propose some new methods to solve the problem besides
the eigenvalue iterations that may be extensible to solve the more general problem.
IV.
Solution Methodologies
The necessary conditions being the eigenspace computations of a symmetric quadratic form can be
computed stably using the Singular Value Decomposition.8 Therefore we have the first algorithm.
3 of 19
American Institute of Aeronautics and Astronautics
IV.A.
SVD Method
Since the matrix is symmetric, the smallest left singular vector is the solution to the problem (We recall that
the eigenvectors and singular vectors span the same spaces in the case of a symmetric matrix).
SVD Approach.
Step 1 Compute SVD and save V , in T T T = U ΣV T
Step 2 xSV D = −V (1 : n, n + 1)/(V (n + 1, n + 1))
Step 3 xSV D is the required solution.
This algorithm is fairly robust but computationally expensive.
IV.B.
Eigenvalue Problem
The solution, which also is the eigenvector corresponding to the smallest eigenvalue can be computed by
inverse iterations.8
Rayleigh quotient iteration.
Step 1 Start with z = [x̂T − 1]T . x̂ = (ÃT Ã)−1 ÃT ỹ.
Step 2 Solve (T T T − λI)v (k) = v (k−1) for k = 1, 2, ....
Step 3 x̂T LS = −v (n) (1 : n, 1)/v (n) (n + 1, 1) is the converged solution.
The algorithm has cubic convergence and we can get 10 digits of accuracy in 3 iterations.
IV.C.
Davidenko’s Homotopy Method
Structure of the necessary conditions clearly indicates that their solution is “close” to the solution of the
normal equations. This motivates us to explore the perturbation methods10 to solve this problem. This
method sees importance in the light of the expressions for the necessary conditions for the weighted total
least squares formulation developed by the authors, where the eigenvalue problem is not “obvious” from the
nonlinear equations (obtained in section III).
Davidenko’s Method.
Step 1 Start with least squares solution G(x̂) = (ÃT Ã)x̂ − ÃT ỹ = 0.
Step 2 Let F (x̂) = (ÃT Ã − λ(x̂)I)x̂ − ÃT ỹ = 0 (Or the weighted version of it).
Step 3 Consider the Homotopy H(z) = tF (z) + (1 − t)G(z), ∀t ∈ [0, 1].
∂H −1 ∂H
Step 4 Integrate dz
[ ∂t ]
dt = −[ ∂z ]
Step 5 x̂ = z(1) is the required estimate.
The accuracy of the solution depends on the numerical integrator and also is reasonably slow to compute.
IV.D.
A QUEST type algorithm
QUEST is an attitude estimation algorithm, proposed by Shuster11 which determines the “Best” attitude
matrix for vector measurements. This algorithm receives attention owing to the possibility of its recursive
implementation.12 Our recursive (rather accumulative) formulations of the TLS problem has strong correlation with the REQUEST methodology. We will also derive some additional benefit from this algorithm
and it is presented next. It is amazing to find that the exact developments carry forward to a general
dimension from the three dimensional case involving QUEST computations. The result is summarized and
the algorithm is presented next. In the eigenvalue problem,
"
# "
#
"
#
ÃT Ã ÃT ỹ
x̂
x̂
=λ
(18)
=
−1
−1
ỹ T Ã ỹ T ỹ
(S − λI)x̂ = z
T
−1
λ = α + z (λI − S)
z
(19)
(20)
where, S := ÃT Ã and z := ÃT ỹ. The fact that (S − λI)−1 can be expanded in lower powers (due to Cayley
Hamilton theorem13 ) enables us to compute the x̂ algebraically.
4 of 19
American Institute of Aeronautics and Astronautics
Generalized QUEST type algorithm.
Step 1 Compute the characteristic polynomial associated with the matrix T T T .
Step 2 With λ0 = 0, as the initial guess, compute the smallest eigenvalue (Newtons root solver).
Step 3 Calculate x̂ = (S − λI)−1 z
V.
Static Parameter Estimation Application : Morphing Wing
The algorithms presented above were used in the identification of sensitivity coefficients of a novel morphing wing developed at Texas A& M University (fig.1). The twisting wing actuator being developed, was
amenable to quasi-static aerodynamic models. As an alternative approach, we wanted to develop alternative
models directly from the input output data of the wind tunnel tests. The experimental setup and aerodynamic models, along with the specifications of the tests performed are discussed in an accompanying paper.14
Figure 1. Morphing Wing : Experimental Setup
V.A.
Discussion
The idea of using the Total least squares method as opposed to least squares method in fitting the data
obtained, was to have a better approximation of the data in regions where physics based models (any strictly
linear models) fail. Least squares approximation is known to “filter” the data in such regions (especially
where the wing stalls) and therefore yields poor models of the physics. On the other hand, the Total least
squares algorithms, possessing more knobs to turn would indeed model the physics to arbitrary extent (fig.
??) (user could control this by playing with the weights). This objective could only partially be realized
because the current TLS approximation deals with “equal” magnitudes of uncertainty and thereby staying
close to the least squares estimates. Upon careful observation, the better approximation of TLS is revealed
by the plots 2, 3. This would be potentially increased by incorporation of weights.
V.B.
Model Validation
Once the fit was completed, a time varying test result was obtained and the prediction from least squares
and the TLS method are compared in the following plots. The time varying test was performed with a small
5 of 19
American Institute of Aeronautics and Astronautics
Lift Coefficient, CL
Lift Coefficient, CL
TLS Data Fit
2
1
Data
TLS fit
LS fit
0
−1
−4
−2
0
2
4
6
Angle of Attack
Residual Errors
8
10
12
8
10
12
0.5
0
−0.5
−4
TLS fit − Baseline test
LS fit
−2
0
2
4
6
Angle of Attack
Figure 2. Baseline (no twist) test : Linear Model and Residuals
stalling and large stalling time periods and compared with the outputs of the models. It is observed that
the approximation of the model fails in region where near stall conditions prevail (figs. 5, 4).
VI.
Applications to Dynamic Systems: The Total Least Squares Kalman
Filter
Having discussed methods of determining the best estimates of a static problem, we would now be
interested in extending the methodology for applications to dynamical systems. In other words, we would
be interested in applying the constraints of differential equations to the optimization problem and obtain
associated filters. The associated filter is developed by considering the following problem. Consider the
discrete time dynamical system given by
xk+1 = Ak xk + Bk uk + Gk wk
(21)
where uk is the control input to the system and wk is the random forcing function most popularly known as
the process noise. The measurements of linear combinations of the states being given by
ỹk = H̃k + vk
(22)
In contrast to the classical Kalman filtering framework, in this case, the true measurement sensitivity matrix,
Hk is unknown. But, its measurements are available at every update time step, being given by,
H̃k = Hk + Ekm
(23)
The state process and measurement noise vectors are assumed to be zero mean Gaussian random vectors, with
the covariances defined by wk ∼ N (0, Rksp ) and vk ∼ N (0, Rksm ). The matrix measurement noise corrupting
the measurement sensitivity matrix are also assumed to be zero mean Gaussian random variables. However,
to simplify the developments, each row of this matrix is assumed to be an independent random vector,
identically distributed with all the other rows. That is, the statistics of the measurement noise matrix given
6 of 19
American Institute of Aeronautics and Astronautics
Lift Coefficient, CL
Lift Coefficient, CL
TLS Data Fit − Tip Section Twists
1.5
1
0.5
0
−5
0
5
Tip Section Twist Angle
Residual Errors
10
Data
TLS fit
LS fit
15
0
5
Tip Section Twist Angle
10
15
0.5
0
TLS Residuals
−0.5 LS Residuals
−5
Figure 3. Tip Section Twist test: Fit of Model and Residuals






:= 




e1
e2
..
.






m
by Ek
 are defined by the statistics of each row,given by ei ∼ N (0, Ri ), uncorrelated with other
ei 

.. 
. 
em
rows and uncorrelated in time. This implies, that,
E V ec EkmT = 0m×n
(24)


T
T
T
E(e1 e1 ) E(e1 e2 ) · · · E(e1 em )


h
i
T
 E(eT2 e1 ) E(eT2 e2 ) · · · E(eT2 em ) 
mT T
mT


V ec Ek
= 
E V ec Ek
(25)
..
..
..

.


.
···
.
E(eTm e1 ) E(eTm e2 )

R1

 R2,1
= 
 ..
 .
Rm,1
=
R1,2
R2
..
.
Rm,2
···
···
· · · E(eTm em )

R1,m
R2,m
..
.
···
· · · Rm,m
EH
Rk m





(26)
(27)
The process noise statistics involved with the evolution of the truth model of the measurement sensitivity
matrix, given by
Hk+1 = Hk + Ekp
EH
V ec EkmT ∼ N (0, Rk p )
(28)
(29)
are similarly defined and the second moment matrix (first moment being zero) of all the elements arranged
EH
vectorially (exactly similar to above developments) is accordingly denoted by Rk p , as defined above. With
the appropriate noise statistics defined as above, the problem is to produce a state estimate, x, by processing
7 of 19
American Institute of Aeronautics and Astronautics
TLS Data Fit testing − Time Varying Input
1
Data
TLS fit
LS fit
0.9
Lift Coefficient, CL
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
10
20
30
40
50
time (sec)
60
70
80
Figure 4. Model Validation : time varying input with large stalling periods
the vector measurements ỹk , that arrive at each update time step. To solve this problem, we propose an
estimator of the form,
x̂k+1 = Ak x̂k + Bk uk
(30)
Ĥk+1 = Ĥk
(31)
The estimated output is assumed to be computed from the updated measurement sensitivity matrix and
therefore attains the following structure.
ŷk = Ĥk+ x̂−
k
(32)
Kalman updates are performed on both state and the measurement sensitivity matrix updates and are given
by
−
x̂+
k = x̂k + Kk [ỹk − ŷk ]
h
i
Ĥk+ = Ĥk− + Lk H̃k − Ĥk−
(33)
(34)
±
Defining the estimation error to be given by δx±
k x − x̂k and the measurement sensitivity matrix estimation
±
±
error δHk = Hk − Ĥk , the innovation error, ỹk − ŷk can be expressed as
ỹk − ŷk
=
H̃k xk + vk − Ĥk+ x̂−
k
=
Hk xk + Ekm xk + vk − Hk+ xk − Ĥk+ xk
m
δHk+ xk + Ĥk+ δx−
k + Ek xk + vk
=
(35)
+
Ĥk+ xk
−
Ĥk+ x̂−
k
(36)
(37)
With the above definitions, the estimation errors in the state update equation is given by
δx+
k
=
xk − x̂+
k
(38)
=
xk −
x̂−
k
(39)
=
δx−
k
− Kk [ỹk − ŷk ]
h
i
+ −
+
−
− Kk δHk+ δx−
k + δHk x̂k + Ĥk δxk + Ek xk + vk
(40)
The corresponding matrix update error of the measurement sensitivity matrix is given by
δHk+
=
Hk − Ĥk+
=
Hk − Ĥk− − Lk H̃k − Ĥk−
=
(Im×m −
(41)
Lk ) δHk−
−
Lk Ekm
8 of 19
American Institute of Aeronautics and Astronautics
(42)
(43)
TLS Data Fit testing − Time Varying Input while stalling
1.2
Data
TLS fit
LS fit
Lift Coefficient, CL
1
0.8
0.6
0.4
0.2
0
10
20
30
40
time (sec)
50
60
70
Figure 5. Model Validation : time varying inputs with small stalling periods
This enables us to write the state estimation error covariance update equation of the form,
Pks+ = E δk+ δk+T
(44)
h
i h
iT
+
+
−
+
+
−
= E δx−
δx−
k − Kk δHk xk + Ĥk δxk + Ek xk + vk
k − Kk δHk xk + Ĥk δxk + Ek xk + vk (45)
To simplify the expression, we make use of the conditional expectation identity from probability theory that
E [X] = E (E [X|Y ]), for all random variables X, Y . Using this property and the additional property that
the closed loop of the discrete Kalman filter is stable, leading to unbiased estimates of the state and the
measurement sensitivity matrix, (i.e., δHk± = 0m×n , δx±
k = 0), together with the zero mean and uncorrelated
nature of the measurement and process noise terms, E(wk ) = 0, E(vk ) = 0, E(Ekm ) = 0m×n , the following
terms in the state covariance update equation vanish
E vk δx−T
=0
(46)
k
+ − −T + − −T
+
E δHk x̂k δxk
= E E δHk x̂k δxk |δHk = 0
(47)
+ − −T + − −T
+
+ s−
E δHk δxk δxk
= E δHk δxk δxk |δHk = E δHk Pk
=0
(48)
−T
−T
m
m
E Ek xk δxk
= E (Ek ) E xk δxk
=0
(49)
Similarly,
h
i
T
−
+
−
m
Kk E δHk+ (δx−
+
x̂
)
+
Ĥ
δx
+
E
x
+
v
[Ekm xk + vk ] = 0
k
k
k
k
k
k
k
h
i
h
i
h
i
−T
+
−T
+
+
E δHk+ δx−
= E E δHk+ δx−
= E δHk+ Pks− Ĥk+ = 0
k δxk Ĥk
k δxk Ĥk |δHk
−T
+T
−T
+T
+
E δHk+ δx−
= E E δHk+ δx−
= E δHk+ Pks− δHk+T
k δxk δHk
k δxk δHk |δHk
(50)
(51)
(52)
This simplifies the covariance update equation to take the form,
Pks+
∆k
= Pks− − Kk Ĥk+ Pk− − Pk− Ĥk+T KkT + Kk ∆k KkT
h
i
−T
+T
−T
+T
= E δHk+ δx−
+ δHk+ x̂−
+ Ĥk+ Pks− Ĥk+T + Ekm xk xTk EkmT + Rksm
k δxk δHk
k x̂k δHk
(53)
(54)
−
The expression for ∆k further simplifies using the fact that xk = x̂−
k + δk and the additional abbreviation,
s−
− −T
Ξk = Pk + x̂k x̂k , in to
∆k = Ĥk+ Pks− Ĥk+T + Rksm + E δHk+ Ξk δHk+T + Ekm Ξk EkmT
(55)
= Ĥk+ Pks− Ĥk+T + Rksm + ∆1k
9 of 19
American Institute of Aeronautics and Astronautics
(56)
Now, to determine the best estimates of the state, we determine the optimal gains, Lk , Kk such that the
error covariance of the updated estimation error is minimized. Considering the scalar performance index,
J = trace Pks+
(57)
= trace Pks− − Kk Ĥk+ Pks− − Pks− Ĥk+T KkT + Kk ∆k KkT
(58)
Necessary conditions for minima,
∂J
∂J
∂Kk , ∂Lk
= 0 lead to the gain equations,
∂J
= −Pks− Ĥk+T + Kk ∆k = 0
∂Kk
(59)
h
i−1
Kk = Pks− Ĥh+T Ĥk+ Pks− Ĥk+T Rksm + ∆1k
(60)
∂J
∂trace(∆2k )
=
=0
∂Lk
∂Lk
(61)
implying
The other necessary condition is rather unique. Notice that the
only terms containing Lk to optimize are
contained in trace Kk ∆2k KkT (where ∆2k = E δHk+ Ξk δHk+T . Hence the necessary condition becomes,
Given ∆2k , we can express it as a function of the propagated measurement sensitivity error covariance elements
and the gain Lk to be determined as follows
∆2k = E δHk+ Ξk δHk+T
(62)
−
m
T
T
−T
mT
= (Im×m − Lk ) E δHk Ξk δHk (Im×m − Lk ) + Lk E Ek Ξk Ek
Lk
(63)
From the above, the second necessary condition becomes,
∂trace Kk ∆2k KkT
= 2KkT Kk −E δHk− ΞδHk−T + Lk E δHk− ΞδHk−T + E Ekm ΞEkmT
=0
∂Lk
(64)
leading to the gain equation
Lk = E δHk− ΞδHk−T
−1
−
E δHk ΞδHk−T + E Ekm ΞEkmT
(65)
However, to facilitate these gain computations, it turns out, we need to compute the covariance associated
with all the elements of the measurement sensitivity matrix. In what
follows, we
will derive the relations
used in the computations of the weighted covariances of the form E δHk− ΞδHk−T . The required covariance
propagation and update equations of the measurement sensitivity matrix estimation error are first derived
using the full covariance matrix (of all mn elements) given by
h
T i
PkH± := E V ec δHk±T V ec δHk±T
(66)
h
i
where, if δHk±T = h1 · · · hm and hi is the ith row of the matrix δHk± , the V ec operator operates on
a matrix (of dimensions say m × n) and produces a vector of length mn. Accordingly,


hT1
 . 

V ec(δHk±T ) = 
 .. 
hTm
Consequently, the expression for the covariance is given by the mn × mn matrix,
h
T i
PkH± = E V ec δHk±T V ec δHk±T


E(hT1 h1 ) E(hT1 h2 ) · · · E(hT1 hm )


 E(hT2 h1 ) E(hT2 h2 ) · · · E(hT2 hm ) 


= 
..
..
..
..

.


.
.
.
E(hTm h1 )
···
· · · E(hTm hm )
10 of 19
American Institute of Aeronautics and Astronautics
(67)
(68)
an expression much similar to equation 24, which denoted the statistics of the matrix. At this stage, we
would point out that there was no assumption so far on the nature of the random variables. We assumed that
the rows were i.i.d. Gaussian, but clearly, as we are tracking the evolutions of all possible correlations of the
rows, (E(HiT hj )), this is not required, provided, we know the correlations apriori. Writing the measurement
sensitivity update error equation 41 another way,
δHk+T = δHk−T (Im×m − Lk )T − EkmT LTk
(69)
T
Taking the V ec operator on both sides and using the identity, V ec (ACB) = B ⊗ A V ecC, we get,
V ec δHk+T
=
[(Im×m Lk ) ⊗ In×n ] V ec δHk−T − (Lk ⊗ In×n ) V ec EkmT
(70)
=
Φ+
kV
(71)
ec
δHk−T
+
Ψ+
kV
ec
EkmT
This
with the definition
of the measurement sensitivity estimation error covariance, PkH± :=
together
±T
±T T
E (V ec δHk )(V ec δHk ) leads to the measurement sensitivity covariance update equation,
EH
H− +T
+T
m
PkH+ = Φ+
Φk + Ψ +
k Pk
k Rk Ψk
(72)
where the expression for the measurement noise statistics from equation (27), has been used. Similarly, the
measurement sensitivity estimation error propagation vector is given by
V ec δHk−T = V ec δHk+T + V ec Ekm
(73)
EH
Rk p
(74)
PkH− = PkH+ +
Using the above derived propagation and update equations, a filter can be constructed. An example demonstrating the same and comparing the results with a classical Kalman Filter under three different circumstances
is presented in the next section.
VII.
Numerical Simulation
We now consider a simple example to evaluate the performance of the newly developed filter and the classical Kalman filter. The problem is a linear oscillator where only the position is available for measurements.
The dynamics of the plant are given by
"
#
"
#
"
#
0
1
0
0
ẋ =
x+
u+
w
(75)
−2 −0.5
1
1
h
i
y = Cx = 1 0 x
(76)
The measurement model is given by
ỹ = C̃x + v
(77)
EkmH
(78)
C̃ = C +
The filter is required to generate position and velocity estimates. Since the filter has to be compared with the
classical Kalman filter, we implement the filter and base our Kalman gain calculation based on the measured
measurement sensitivity matrix C̃. The tuning parameters for the filter implementations are summarized
for the three different cases in table (1).
VIII.
VIII.A.
Conclusion and Future Directions
Conclusions
The least squares error criterion is generalized to account for errors in both range space and the basis
functions in a measurement model. This was shown to lead to nonlinear necessary conditions. The necessary
conditions were then realized as solutions to eigenvalue problem associated with the measurement matrix and
the vector. A novel weighted total least squares criterion was presented and associated necessary conditions
were derived. Several methods to solve this problem including two novel methods were presented. This
was applied on a parameter identification problem of a morphing wing model developed at Texas A& M
University.
11 of 19
American Institute of Aeronautics and Astronautics
Table 1. Continuous Discrete Constrained Attitude Filter
Case Number
Process Noise (StateE(wwT ) = Rksp )
Measurement Noise (State E(vv T ) = Rksm )
1
10 I2×2
10−4
2
10 I2×2
10−1
3
10 I2×2
10−1
Process Noise (Meas. Sensitivity)Rk p
10−4 I 2×2
10−4 I 2×2
10−4 I 2×2
0 × I 2×2
101 I 2×2
102 I 2×2
[1, 1]T
[−1, 1]T
figure (6)
figure (7)
figure (8)
NA
NA
10−4 I 2×2
101 I 2×2
102 I 2×2
[1, 1]T
[−1, 1]T
figure (9)
figure (10)
figure (11)
figure (12)
figure (13)
10−4 I 2×2
101 I 2×2
102 I 2×2
[1, 1]T
[−1, 1]T
figure (14)
figure (15)
figure (16)
figure (17)
figure (18)
−6
EH
EH
Rk m
Measurement Noise (Meas. Sensitivity)
PkH (0)
Pkx (0)
Ĥ(0)
x̂(0)
Estimates Vs. Truth
Absolute Value of Estimation Errors
Estimation Errors and Covariance Bounds
Estimation Errors and Covariance Bounds (Magnified)
Parameter Estimation Errors
VIII.B.
−4
−2
Future Directions
Most importantly because of the large degree of design choice, the method tunes itself to fit the measurements
as close as possible. While in some problems, this models the physics not essentially modeled by linear least
squares, in filtering problems this is not always desirable as some kind of signal reconstruction is anticipated.
Therefore, work is in progress in the direction of modifying this error criterion so as to reduce the huge over
parameterization. Smoother formulations incorporated for dynamical state estimation are being considered
for incorporation. As mentioned in the paper, work is also in progress to develop algorithms for weighted
total least squares criterion whose necessary conditions do not form the eigenvalue problem. Static problems
were dealt with in the above discussion. Extension to dynamical system state estimation is expected to
improved filters where there is uncertainty in the models of measurement and plant dynamics. This is being
investigated and researched currently. The developments so far make optimistic gestures towards this goal.
Acknowledgments
The authors wish to acknowledge the support of Texas Institute of Intelligent Bio Nano Materials and
Structures for Aerospace Vehicles funded by NASA Cooperative Agreement No. NCC-1-02038.
References
1 Adcock,
R. J., “A Problem in Least Squares,” The Analyst, Vol. 5, Jan. – Feb. 1878, pp. 53–54.
L. J., “Estimation in a multivariable “errors in variables” regression model: large sample results,” Annals of
Statistics, Vol. 9, 1981, pp. 24–44.
3 Golub, G. H. and Loan, C. F. V., “An Analysis of the Total Least Squares Problem,” SIAM Journal of Numerical
Analysis, Vol. 17, 1980, pp. 883–893.
4 Huffel, S. V. and Vandewalle, J., The Total Least Squares Problem: Computational Aspects and Analysis, SIAM Publications, Philadelphia, 1991.
5 Villegas, C., “Maximum Likelihood Estimation of a Linear Functional Relationship,” The Annals of Mathematical Statistics, Vol. 32, No. 4, December, 1961, pp. 1048–1062.
6 Mook, D. J. and Junkins, J. L., “Minimum Model Error Estimation for Poorly Modeled Dynamic Systems,” Journal of
Guidance Control and Dynamics, Vol. 3, No. 4, 1988, pp. 367–375.
7 Crassidis, J. L. and Junkins, J. L., Optimal Estimation of Dynamic Systems, Chapman and Hall/CRC Press, Boca
Raton, FL, 2004.
8 Golub, G. H. and Loan, C. F. V., Matrix Computations, The Johns Hopkins University Press, Baltimore, MD, 3rd ed.,
1996.
9 Riesz, F. and Nagy, B. S., Functional Analysis, Reprinted by Dover Publications, Mineola, NY, 1990.
10 Davidenko, D. F., “A new method of solution of a system of nonlinear equations,” Dokl. Akad. Nauk SSSR, Vol. 88,
1953, pp. 601.
2 Glesler,
12 of 19
American Institute of Aeronautics and Astronautics
Comparison of the Kalman filters in multilinear uncertainty environment
Position
2
truth
Discrete Kalman Filter estimate
Total Kalman Filter estimate
1
0
−1
0
1
2
3
4
5
time
6
7
8
9
10
1
2
3
4
5
time
6
7
8
9
10
2
Velocity
1
0
−1
−2
0
Figure 6. Case1: Estimates Vs Truth
11 Schuster, M. D. and Oh, S. D., “Three axis attitude determination from Vector Observations,” Journal of Guidance and
Control , Vol. 4, Jan. – Feb. 1981, pp. 70 – 77.
12 Bar-Itzhack, T. Y., “REQUEST : A recursive QUEST Algorithm for Squential Attitude Determination,” Journal of
Guidance Control and Dynamics, Vol. 19, No. 5, Sep. – Oct. 1996, pp. 1034–1038.
13 Junkins, J. L. and Kim, Y., Introduction to Dynamics and Control of Flexible Structures, AIAA Education Series,
Washington, DC, 1991.
14 Majji, M., Rediniotis, O. K., and Junkins, J. L., “Design of a Morphing Wing : Modeling and Experiments,” Submittal,
AIAA Guidance Navigation and Control Conference, 2007.
13 of 19
American Institute of Aeronautics and Astronautics
Position Estimation Error
Estimation Error in Two Filters
Estimation Error −DKF
Estimation Error − TDKF
0
10
−5
10
0
1
2
3
4
5
time
6
7
8
9
10
1
2
3
4
5
time
6
7
8
9
10
0
Velocity
10
−5
10
0
Figure 7. Case1: Estimation error
Velocity Estimation Errors
Position Estimation Errors
Estimation Error and Error Covariance Comparison
15
Estimation Error KF
Error Covariance KF
Estimation Error TLSKF
Error Covariance TLSKF
10
5
0
−5
−10
−15
0
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
7
8
9
10
15
10
5
0
−5
−10
−15
0
Figure 8. Case1: Estimation error and covariance bounds
14 of 19
American Institute of Aeronautics and Astronautics
Comparison of the Kalman filters in multilinear uncertainty environment
3
truth
Discrete Kalman Filter estimate
Total Kalman Filter estimate
Position
2
1
0
−1
−2
0
1
2
3
4
5
time
6
7
8
9
10
1
2
3
4
5
time
6
7
8
9
10
2
Velocity
1
0
−1
−2
−3
0
Figure 9. Case2: Estimates Vs Truth
Position Estimation Error
Estimation Error in Two Filters
Estimation Error −DKF
Estimation Error − TDKF
0
10
−5
10
0
1
2
3
4
5
time
6
7
8
9
10
1
2
3
4
5
time
6
7
8
9
10
0
Velocity
10
−5
10
0
Figure 10. Case2: Estimation error (Log Scale)
15 of 19
American Institute of Aeronautics and Astronautics
Velocity Estimation Errors
Position Estimation Errors
Estimation Error and Error Covariance Comparison
50
Estimation Error KF
Error Covariance KF
Estimation Error TLSKF
Error Covariance TLSKF
0
−50
0
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
7
8
9
10
50
0
−50
0
Figure 11. Case2: Estimation error and covariance bounds
Position Estimation Errors
Estimation Error and Error Covariance Comparison
Estimation Error KF
Error Covariance KF
Estimation Error TLSKF
Error Covariance TLSKF
2
1
0
−1
Velocity Estimation Errors
2.5
2.55
2.6
2.65
2.7
2.75
2.8
2.85
2.9
2.95
1
0
−1
−2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
Figure 12. Case2: Estimation error and covariance bounds Magnified View
16 of 19
American Institute of Aeronautics and Astronautics
Measurement Sensitivity Matrix Parameter Estimation Errors
0
10
H(1,1)
H(2,1)
−1
Parameter Estimation Errors
10
−2
10
−3
10
−4
10
−5
10
−6
10
0
1
2
3
4
5
time
6
7
8
9
10
Figure 13. Case2: Measurement Sensitivity Matrix Element Estimation Errors
Comparison of the Kalman filters in multilinear uncertainty environment
3
truth
Discrete Kalman Filter estimate
Total Kalman Filter estimate
Position
2
1
0
−1
−2
0
1
2
3
4
5
time
6
7
8
9
10
1
2
3
4
5
time
6
7
8
9
10
4
Velocity
2
0
−2
−4
0
Figure 14. Case3: Estimates Vs Truth
17 of 19
American Institute of Aeronautics and Astronautics
Position Estimation Error
Estimation Error in Two Filters
Estimation Error −DKF
Estimation Error − TDKF
0
10
−2
10
0
1
2
3
4
5
time
6
7
8
9
10
1
2
3
4
5
time
6
7
8
9
10
0
Velocity
10
−2
10
0
Figure 15. Case3: Estimation error (Log Scale)
Velocity Estimation Errors
Position Estimation Errors
Estimation Error and Error Covariance Comparison
50
Estimation Error KF
Error Covariance KF
Estimation Error TLSKF
Error Covariance TLSKF
0
−50
0
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
7
8
9
10
50
0
−50
0
Figure 16. Case3: Estimation error and covariance bounds
18 of 19
American Institute of Aeronautics and Astronautics
Position Estimation Errors
Estimation Error and Error Covariance Comparison
1.5
1
0.5
0
−0.5
−1
Velocity Estimation Errors
2
2.1
2.2
2.3
2.4
2.5
Estimation Error KF
Error Covariance KF
2.7 Estimation
2.8
2.9 TLSKF
Error
Error Covariance TLSKF
2.6
5
0
−5
2
2.2
2.4
2.6
2.8
3
3.2
Figure 17. Case3: Estimation error and covariance bounds
Measurement Sensitivity Matrix Parameter Estimation Errors
0
10
H(1,1)
H(2,1)
−1
Parameter Estimation Errors
10
−2
10
−3
10
−4
10
−5
10
−6
10
0
1
2
3
4
5
time
6
7
8
9
Figure 18. Case3: Measurement Sensitivity Matrix Estimation Errors
19 of 19
American Institute of Aeronautics and Astronautics
10