arXiv:0910.0902v3 [cs.LG] 22 Dec 2009
Reduced-Rank Hidden Markov Models
Sajid M. Siddiqi
Robotics Institute
Carnegie Mellon University
Pittsburgh, PA 15213
siddiqi@cs.cmu.edu
Byron Boots
Computer Science Department
Carnegie Mellon University
Pittsburgh, PA 15213
beb@cs.cmu.edu
Geoffrey J. Gordon
Machine Learning Department
Carnegie-Mellon University
Pittsburgh, PA 15213
ggordon@cs.cmu.edu
Abstract
We introduce the Reduced-Rank Hidden Markov Model (RR-HMM), a generalization of HMMs that
can model smooth state evolution as in Linear Dynamical Systems (LDSs) as well as non-log-concave
predictive distributions as in continuous-observation HMMs. RR-HMMs assume an m-dimensional latent
state and n discrete observations, with a transition matrix of rank k ≤ m. This implies the dynamics
evolve in a k-dimensional subspace, while the shape of the set of predictive distributions is determined
by m. Latent state belief is represented with a k-dimensional state vector and inference is carried out
entirely in Rk , making RR-HMMs as computationally efficient as k-state HMMs yet more expressive.
To learn RR-HMMs, we relax the assumptions of a recently proposed spectral learning algorithm for
HMMs [1] and apply it to learn k-dimensional observable representations of rank-k RR-HMMs. The
algorithm is consistent and free of local optima, and we extend its performance guarantees to cover the
RR-HMM case. We show how this algorithm can be used in conjunction with a kernel density estimator to
efficiently model high-dimensional multivariate continuous data. We also relax the assumption that single
observations are sufficient to disambiguate state, and extend the algorithm accordingly. Experiments on
synthetic data and a toy video, as well as on a difficult robot vision modeling problem, yield accurate
models that compare favorably with standard alternatives in simulation quality and prediction capability.
1
Introduction
Models of stochastic discrete-time dynamical systems have important applications in a wide range of fields. Hidden Markov
Models (HMMs) [2] and Gaussian Linear Dynamical Systems (LDSs) [3] are two examples of latent variable models
of dynamical systems, which assume that sequential data points are noisy, incomplete observations of a latent state that
evolves over time. HMMs model this latent state as a discrete variable, and represent belief as a discrete distribution over
states. LDSs on the other hand model the latent state as a set of real-valued variables, are restricted to linear transition and
observation functions, and employ a Gaussian belief distribution. The distributional assumptions of HMMs and LDSs also
result in important differences in the evolution of their belief over time. The discrete state of HMMs is good for modeling
systems with mutually exclusive states that can have completely different observation signatures. The joint predictive
distribution over observations is allowed to be non-log-concave when predicting or simulating the future, leading to what
we call competitive inhibition between states (see Figure 3 below for an example). Competitive inhibition denotes the
ability of a model’s predictive distribution to place probability mass on observations while disallowing mixtures of those
observations. Conversely, the Gaussian joint predictive distribution over observations in LDSs is log-concave, and thus does
not exhibit competitive inhibition. However, LDSs naturally model smooth state evolution, which HMMs are particularly
bad at. The dichotomy between the two models hinders our ability to compactly model systems that exhibit both competitive
inhibition and smooth state evolution.
We present the Reduced-Rank Hidden Markov Model (RR-HMM), a smoothly evolving dynamical model with the ability to
represent nonconvex predictive distributions by relating discrete-state and continuous-state models. HMMs can approximate
smooth state evolution by tiling the state space with a very large number of low-observation-variance discrete states with a
specific transition structure. However, inference and learning in such a model is highly inefficient due to the large number
of parameters, and due to the fact that existing HMM learning algorithms, such as Expectation Maximization (EM) [2],
are prone to local minima. RR-HMMs allow us to reap many of the benefits of large-state-space HMMs without incurring
the associated inefficiency during inference and learning. Indeed, we show that all inference operations in the RR-HMM
1
lt-1
lt
ht-1
ht
ht+1
xt-1
xt
xt+1
lt+1
B.
S
R
n
R
m
R
k
xt .
ht .
simplex
simplex
lt.
C. 0.6
O
R
Pr(observation 2)
A.
RR-HMM
(m=10,rank 3)
HMM
(m=3)
0.5
0.4
0.5
0.6
Pr(observation 1)
0.4
Figure 1: (A) The graphical model representation of an RR-HMM. lt denotes the k-dimensional state vector, ht the mdimensional discrete state, and xt the discrete observation. The distributions over ht and lt+1 are deterministic functions of
lt . (B) An illustration of different RR-HMM parameters and the spaces and random variables they act on. (C) Projection of
sets of predictive distributions of a rank 3 RR-HMM with 10 states, and a 3-state full-rank HMM with similar parameters.
can be carried out in the low-dimensional space where the dynamics evolve, decoupling their computational cost from the
number of hidden states. This makes rank-k RR-HMMs (with any number of states) as computationally efficient as kstate HMMs, but much more expressive. Though the RR-HMM is in itself novel, its low-dimensional Rk representation is
related to existing models such as Predictive State Representations (PSRs) [4], Observable Operator Models (OOMs) [5],
generalized HMMs [6], and weighted automata [7, 8], as well as the the representation of LDSs learned using Subspace
Identification [9]. These and other related models and algorithms are discussed further in Section 5.
To learn RR-HMMs from data, we adapt a recently proposed spectral learning algorithm by Hsu, Kakade and Zhang [1]
(henceforth referred to as HKZ) that learns observable representations of HMMs using matrix decomposition and regression
on empirically estimated observation probability matrices of past and future observations. An observable representation of
an HMM allows us to model sequences with a series of operators without knowing the underlying stochastic transition and
observation matrices. The HKZ algorithm is free of local optima and asymptotically unbiased, with a finite-sample bound
on L1 error in joint probability estimates from the resulting model. However, the original algorithm and its bounds assume
(1) that the transition model is full-rank and (2) that single observations are informative about the entire latent state, i.e.
1-step observability. We show how to generalize the HKZ bounds to the low-rank transition matrix case and derive tighter
bounds that depend on k instead of m, allowing us to learn rank-k RR-HMMs of arbitrarily large m in O(N k 2 ) time, where
N is the number of samples. We also describe and test a method for circumventing the 1-step observability condition by
combining observations to make them more informative. A version of this learning algorithm can learn general PSRs [10]
though our error bounds don’t yet generalize to this case.
Experiments show that our learning algorithm can recover the underlying RR-HMM in a variety of synthetic domains.
We also demonstrate that RR-HMMs are able to compactly model smooth evolution and competitive inhibition in a clock
pendulum video, as well as in real-world mobile robot vision data captured in an office building. Robot vision data (and,
in fact, most real-world multivariate time series data) exhibits smoothly evolving dynamics requiring multimodal predictive
beliefs, for which RR-HMMs are particularly suited. We compare performance of RR-HMMs to LDSs and HMMs on
simulation and prediction tasks. Proofs and details regarding examples are in the Appendix.
2
Reduced-rank Hidden Markov Models
Let ht ∈ 1, . . . , m denote the discrete hidden states of an HMM at time t, and xt ∈ 1, . . . , n denote the discrete observations.
Let T ∈ Rm×m be the state transition probability matrix with Tij = Pr[ht+1 = i | ht = j]. O ∈ Rn×m is the observation
probability matrix such that Oij = Pr[xt = i | ht = j], and ~π ∈ Rm is the initial state distribution with ~πi = Pr[h1 = i].
Let ~ht ∈ Rm denote the system’s belief, i.e. a distribution over hidden states at time t. If we use ei to denote the ith
column of the identity matrix, then ~ht is equivalent to the conditional expectation of eht , with the conditioning variables
clear from context. In addition to the standard HMM notation, assume T has rank k and let T = RS where R ∈ Rm×k
and S ∈ Rk×m . This implies that the dynamics of the system can be expressed in Rk rather than Rm . By convention, we
think of S as projecting the m-dimensional state distribution vector to a k-dimensional state vector, and R as expanding
this low-dimensional state back to an m-dimensional state distribution vector while propagating it forward in time. One
possible choice for R and S is to use any k independent columns of T as the columns of R, and let the columns of S contain
the coefficients required to reconstruct T from R, though other choices are possible (e.g. using SVD). Also assume for now
that m ≤ n (we relax this assumption in Section 3.4). We denote the k-dimensional projection of the hidden state vector ~ht
2
as ~lt , which is simply a vector of real numbers rather than a stochastic vector. We assume the initial state distribution lies
in the low dimensional space as well, i.e. ~π = R~πl for some vector ~πl ∈ Rk . Figure 1(A) illustrates the graphical model
corresponding to an RR-HMM. Figure 1(B) illustrates some of the different RR-HMM parameters and the spaces they act
on.
To see how the probability of a sequence can be computed using these parameters, define
Ax = RS diag(Ox,1 , ..., Ox,m )
so that Ax ∈ Rm×m , and define
Wx = S diag(Ox,1 , ..., Ox,m )R
k×k
so that Wx ∈ R
. Also let W =
using either{Ax } or {Wx } as
P
x
Wx = SR. With these definitions, the joint probability of x1 , ..., xt , can written
Pr[x1 , ..., xt ] = ~1T
π
m Axt . . . Ax1 ~
T
~
= 1m RWxt . . . Wx1 ~πl
(2.1a)
(2.1b)
The latter parametrization casts a rank-k RR-HMM as a k-dimensional PSR or transformed PSR [11]. Inference can be
carried out in O(N k 2 ) time in this representation. However, since every HMM is trivially a PSR, this leads to the question
of how expressive rank-k RR-HMMs are in comparison to k-state full-rank HMMs. The following example is instructive.
2.1
Expressivity of RR-HMMs
We describe a rank-k RR-HMM whose set of possible predictive distributions is easy to visualize and describe. Consider
the following rank 3 RR-HMM with 10 states and 4 observations. The observation probabilities in each state are of the form
Oi,· = [ pi qi pi (1 − qi ) (1 − pi )qi (1 − pi )(1 − qi )]
for some 0 ≤ pi , qi ≤ 1, which can be interpreted as 4 discrete observations, factored as two binary components which are
independent given the state. T and pi , qi are chosen to place the vertices of the set of possible predictive distributions on
evenly spaced points along a circle in (p, q)-space:
Tij = (1/2m) [2 + sin (2πi/m) sin (2πj/m) + cos (2πi/m) cos (2πj/m)]
pi = [sin(2πi/m) + 1] /2
qi = [cos(2πi/m) + 1] /2
We plot the marginal probability of each component of the observation, ranging across all achievable values of the latent
state vector for the m = 10 case (Figure 1(C)), yielding a 10-sided polygon as the projection of the set of possible predictive
distributions. These distributions are the columns of T T O. We also plot the corresponding marginals for the m = 3 fullrank HMM case to yield a triangular set. More generally, from a k-state HMM, we can get at most a k-sided polygon for
the set of possible predictive distributions.
The above example illustrates that rank-k RR-HMMs with m states can model sets of predictive distributions which fullrank HMMs with less than m states cannot express. However, as we shall see, inference in rank-k RR-HMMs of arbitrary
m is as efficient as inference in k-state full-rank HMMs. This implies that the additional degrees of freedom in the RRHMM’s low-dimensional parameters and state vectors buy it considerable expressive power. Since RR-HMMs are also
related to PSRs as pointed out in the previous section, and since our learning algorithm will be shown to be consistent for
estimating PSRs (though we have finite-sample guarantees only for the RR-HMM case), it is also instructive to examine the
expressivity of PSRs in general. We refer the reader to Jaeger (2000) [5] and James et. al. (2004) [12] for more on this.
3
Learning Reduced-Rank HMMs
In a full-rank HMM, the maximum likelihood solution for the parameters {T, O} can be found through iterative techniques
such as expectation maximization (EM) [13]. EM, however, is prone to local optima and does not address the model
selection problem. HMM model selection algorithms that avoid local minima (e.g. [14]) are better but still not guaranteed
to return anything close to optimal as data increases, and is slow beyond a certain state space magnitude. Moreover, in
learning RR-HMMs we face the additional challenge of learning the factors of its low-rank transition matrix. We could use
EM to estimate T followed by (or combined with) matrix factorization algorithms such as Singular Value Decomposition
(SVD) [15] or Non-negative Matrix Factorization (NMF) [16]. This approach has several drawbacks. For example, if the
noisy estimate of a low-rank transition matrix is not low-rank itself, SVD could cause negative numbers to appear in the
reconstructed transition matrix. Also, algorithms for NMF are only locally optimal, and NMF is overly restrictive in that it
3
constrains its factor matrices to be non-negative, which is unnecessary for our application since low-rank transition matrices
may have negative numbers in their factors R and S.
An alternative approach, which we adopt, is to learn an asymptotically unbiased observable representation of an RR-HMM
directly using SVD of a probability matrix relating past and future observations. This idea has roots in subspace identification [9, 17] and multiplicity automata [7, 8, 6] as well as the PSR/OOM literature [5, 18] and was recently formulated in
a paper by Hsu, Kakade and Zhang [1] for full-rank HMMs. We use their algorithm, extending its theoretical guarantees
for the low-rank HMM case where the rank of the transition matrix T is k ≤ m. Computationally, the only difference in
our base algorithm (before Section 3.4) is that we learn a rank k representation instead of rank m. This allows us learn
much more compact representations of possibly large-state-space real-world HMMs, and greatly increases the applicability
of the original algorithm. Even when the underlying HMM is not low-rank, we can examine the singular values to tune
the complexity of the underlying RR-HMM, thus providing a natural method for model selection. We present the main
definitions, the algorithm and its performance bounds below. Detailed versions of the supporting proofs and lemmas can be
found in the Appendix.
3.1
The Algorithm
The learning algorithm depends on the following vector and matrix quantities that comprise properties of single observations, pairs of observations and triples:
[P1 ]i = Pr[x1 = i]
[P2,1 ]i,j = Pr[x2 = i, x1 = j]
[P3,x,1 ] = Pr[x3 = i, x2 = x, x1 = j] for x = 1, . . . , n
P1 ∈ Rn is a vector, P2,1 ∈ Rn×n and P3,x,1 ∈ Rn×n are matrices. These quantities are closely related to matrices
computed in algorithms for learning OOMs [5], PSRs [18] and LDSs using subspace identification (Subspace ID) [9]. They
can be expressed in terms of HMM parameters (for proofs see the Appendix: Lemmas 8 and 9 in Section 7.1):
T
P~1T = ~1T
m T diag(π)O
P2,1 = OT diag(π)OT
P3,x,1 = OAx T diag(π)OT
Note that P2,1 and P3,x,1 both contain a factor of T and hence are both of rank k for a rank-k RR-HMM. This property
will be important for recovering an estimate of the RR-HMM parameters from these matrices. The primary intuition is
that, when projected onto an appropriate linear subspace, P3,x,1 is linearly related to P2,1 through a product of RR-HMM
parameters. This allows us to devise an algorithm that
1. estimates P2,1 and P3,x,1 from data,
2. projects them to an appropriate linear subspace computed using SVD,
3. uses linear regression to estimate the RR-HMM parameters (up to a similarity transform) from these projections.
Specifically, the algorithm attempts to learn an observable representation of the RR-HMM using a matrix U ∈ Rn×k such
that U T OR is invertible. An observable representation is defined as follows.
Definition 1 The observable representation is defined to be the parameters b1 , b∞ , {Bx }nx=1 such that:
~b1 = U T P1
~b∞ = (P T U )+ P1
2,1
T
(3.1a)
(3.1b)
T
Bx = (U P3,x,1 )(U P2,1 )
+
for x = 1, . . . , n
(3.1c)
For the RR-HMM, note that the dimensionality of the parameters is determined by k, not m: b1 ∈ Rk , b∞ ∈ Rk and
∀x Bx ∈ Rk×k . Though these definitions seem arbitrary at first sight, the observable representation of the RR-HMM
is closely related to the true parameters of the RR-HMM in the following manner (see Lemma 9 in the Appendix for the
proof):
1. ~b1 = (U T OR)πl = (U T O)π,
2. ~bT = 1T R(U T OR)−1 ,
∞
m
4
3. For all x = 1, . . . , n : Bx = (U T OR)Wx (U T OR)−1
Hence Bx is a similarity transform of the RR-HMM parameter matrix Wx = S diag(Ox,· )R (which, as we saw earlier,
allows us to perform RR-HMM inference), and ~b1 and ~b∞ are the corresponding linear transformations of the RR-HMM
initial state distribution and the RR-HMM normalization vector. Note that (U T OR) must be invertible for these relationships
to hold. Together, the parameters ~b1 ,~b∞ and Bx for all x comprise the observable representation of the RR-HMM. Our
learning algorithm will estimate these parameters from data. The algorithm for estimating rank-k RR-HMMs is equivalent
to the spectral HMM learning algorithm of HKZ [1] for learning k-state HMMs. Our relaxation of their conditions (e.g.
HKZ assume a full-rank transition matrix, without which their bounds are vacuous), and our performance guarantees for
learning rank-k RR-HMMs, show that the algorithm learns a much larger class of k-dimensional models than the class of
k-state HMMs.
L EARN -RR-HMM(k, N ) The learning algorithm takes as input the desired rank k of the underlying RR-HMM rather
than the number of states m. Alternatively, given a singular value threshold the algorithm can choose the rank of the HMM
by examining the singular values of P2,1 in Step 2. It assumes that we are given N independently sampled observation
triples (x1 , x2 , x3 ) from the HMM. In practice, we can use a single long sequence of observations as long as we discount
the bound on the number of samples based on the mixing rate of the HMM (i.e. (1 − the second eigenvalue of T )), in which
case π must correspond to the stationary distribution of the HMM to allow estimation of P~1 . The algorithm results in an
bx for x = 1, . . . , n. The steps are
estimated observable representation of the RR-HMM, with parameters bb1 , bb∞ , and B
briefly summarized here for reference:
1. Compute empirical estimates Pb1 , Pb2,1 , Pb3,x,1 of P~1 , P2,1 , P3,x,1 (for x = 1, ..., n).
b , the matrix of left singular vectors corresponding to the k largest singular values.
2. Use SVD on Pb2,1 to compute U
3. Compute model parameter estimates:
b T Pb1 ,
(a) bb1 = U
b )+ Pb1 ,
(b) bb∞ = (PbT U
2,1
b T Pb2,1 )+ (for x = 1, . . . , n)
bx = U
b T Pb3,x,1 (U
(c) B
We now examine how we can perform inference in the RR-HMM using the observable representation. For this, we will
need to define the internal state ~bt . Just as the parameter ~b1 is a linear transform of the initial RR-HMM belief state, ~bt is a
linear transform of the belief state of the RR-HMM at time t (Lemma 10 in Section 7.1 of the Appendix):
~bt = (U T OR)~lt (x1:t−1 ) = (U T O)~ht (x1:t−1 )
This internal state ~bt can be updated to condition on observations and evolve over time, just as we can update ~lt for RRHMMs and ~ht for regular HMMs.
3.2
Inference in the Observable Representation
Given a set of observable parameters, we can predict the probability of a sequence, update the internal state bbt to perform
filtering and predict conditional probabilities as follows (see Lemma 10 in the Appendix for proof):
c 1 , . . . , xt ] = bbT B
b
b b
• Predict sequence probability: Pr[x
∞ xt . . . Bx1 b1
• Internal state update: bbt+1 =
bx b
B
t bt
b
b b
bT
∞ Bxt bt
c t | x1:t−1 ] =
• Conditional probability of xt given x1:t−1 : Pr[x
b
bx b
bT B
bt
P ∞bT tb b
x b ∞ Bx b t
Estimated parameters can, in theory, lead to negative probability estimates. These are most harmful when they cause
P bT b b
b b
the normalizers bbT
∞ Bxt bt or
x b∞ Bx bt to be negative. However, in our experiments, the latter was never negative and
the former was very rarely negative; and, using real-valued observations with KDE (as in Section 3.5) makes negative
normalizers even less likely, since in this case the normalizer is a weighted sum of several estimated probabilities. In
practice we recommend thresholding the normalizers with a small positive number, and not trusting probability estimates
for a few steps if the normalizers fall below the threshold.
5
Note that the inference operations occur entirely in Rk . We mentioned earlier that parameterizing RR-HMM parameters as
Wx for all observations x casts it as a PSR of k dimensions. In fact the learning and inference algorithms for RR-HMMs
proposed here have no dependence on the number of states m whatsoever, though other learning algorithms for RR-HMMs
can depend on m (e.g. if they learn R and S directly). The RR-HMM formulation is intuitively appealing due to the idea
of a large discrete state space with low-rank transitions, but this approach is also a provably consistent learning algorithm
for PSRs in general, with finite-sample performance guarantees for the case where the PSR is an RR-HMM. Since PSRs
are provably more expressive and compact than finite-state HMMs [5, 12], this indicates that we can learn a more powerful
class of models than HMMs using this algorithm.
3.3
Theoretical Guarantees
The following finite sample bound on the estimated model generalizes analogous results from HKZ to the case of low-rank
T . Theorem 2 bounds the L1 error in joint probability estimates from the learned model. This bound shows the consistency
of the algorithm in learning a correct observable representation of the underlying RR-HMM, without ever needing to recover
the high-dimensional parameters R, S, O of the latent representation. Note that our error bounds are independent of m, the
number of hidden states; instead, they depend on k, the rank of the transition matrix, which can be much smaller than m.
Since HKZ explicitly assumes a full-rank HMM transition matrix, and their bounds become vacuous otherwise, generalizing
their framework involves relaxing this condition, generalizing the theoretical guarantees of HKZ and deriving proofs for
these guarantees.
Define σk (M ) to denote the k th largest singular value of matrix M . The sample complexity bounds depend polynomially
on 1/σk (P2,1 ) and 1/σk (OR). The larger σk (P2,1 ) is, the more well-separated are the dynamics from noise. The larger
σk (OR) is, the more informative the observation is regarding state. For both these quantities, the larger the magnitude, the
fewer samples we need to learn a good model. The bounds also depend on a term n0 (), which is the minimum number
of observations that account for (1 − ) of the total probability mass, i.e. the number of “important” observations. Recall
that N is the number of independently sampled observation triples which comprise the training data, though as mentioned
earlier we can also learn from a single long training sequence.
The theorem holds under mild conditions. Some of these are the same as (or relaxations of) conditions in HKZ, namely
that the prior ~π is nonzero everywhere, and a number of matrices of interest (R, S, O, (U T OR)) are of rank at least k for
invertibility reasons. The other conditions are uniquepto the low-rank setting, namely that S diag(~π )OT has rank at least k,
R has at least one column whose L2 norm is at most k/m, and the L1 norm of R is at most 1. The first of these conditions
implies that the column space of S and the row space of O have some degree of overlap. The other two are satisfied, in the
case of HMMs, by thinking of R as containing k linearly independent probability distributions along its columns (including
a near-uniform column) and of S as containing the coefficients needed to obtain T from those columns. Alternatively, the
conditions can be satisfied for an arbitrary R by scaling down entries of R and scaling up entries of S accordingly. However,
this increases 1/σk (OR), and hence we pay a price by increasing the number of samples needed to attain a particular error
bound. See the Appendix (Section 7.1) for formal statements of these conditions.
Theorem 2 [Generalization of HKZ Theorem 6] There exists a constant C > 0 such that the following √
holds. Pick any
0 ≤ , η ≤ 1 and t ≥ 1. Assume the HMM obeys Conditions 3,4,5,6 and 7. Let ε = σk (OR)σk (P2,1 )/(4t k). Assume
t2
k
k · n0 (ε)
N ≥C· 2 ·
+
· log(1/η)
σk (OR)2 σk (P2,1 )4
σk (OR)2 σk (P2,1 )2
With probability ≥ 1 − η, the model returned by L EARN RR-HMM(k, N ) satisfies
X
c 1 , . . . , xt ]| ≤ | Pr[x1 , . . . , xt ] − Pr[x
x1 ,...,xt
For the proof, see the Appendix (Section 7.4).
3.4
Learning with Observation Sequences as Features
The probability matrix P2,1 relates one past timestep to one future timestep, under the assumption that the vector of observation probabilities at a single step is sufficient to disambiguate state (n ≥ m and rank(O) = m). In system identification
theory, this corresponds to assuming 1-step observability [9]. This assumption is unduly restrictive for many real-world
dynamical systems of interest. More complex sufficient statistics of past and future may need to be modeled, such as the
block Hankel matrix formulations for subspace methods [9, 17] to identify linear systems that are not 1-step observable.
6
For RR-HMMs, this corresponds to the case where n < m and/or rank(O) < m. Similar to the Hankel matrix formulation,
we can stack multiple observation vectors such that each augmented observation comprises data from several, possibly
consecutive, timesteps. The observations in the augmented observation vectors are assumed to be non-overlapping, i.e. all
observations in the new observation vector at time t + 1 have larger time indices than observations in the new observation
vector at time t. This corresponds to assuming past sequences and future sequences spanning multiple timesteps as events
that characterize the dynamical system, causing P~1 ,P2,1 and P3,x,1 to be larger. Note that the x in P3,x,1 still denotes a
single observation, whereas the other indices in P~1 , P2,1 and P3,x,1 are now associated with events. For example, if we
stack n consecutive observations, P3,x,1 [i, j] equals the probability of seeing the ith n-length sequence, followed by the
single observation x, followed by the j th n-length sequence. Empirically estimating this matrix consists of scanning for the
appropriate subsequences i and j separated by observation symbol x, and normalizing to obtain the occurrence probability.
P2,1 and P3,x,1 become larger matrices if we use a larger set of events in the past and future. However, stacking observations
does not complicate the dynamics: it can be shown that the rank of P2,1 and P3,x,1 cannot exceed k (see Section 8 in the
Appendix for a proof sketch). Since our learning algorithm relies on an SVD of P2,1 , this means that augmenting the
observations does not increase the rank of the HMM we are trying to recover. Also, since P3,x,1 is still an observation
probability matrix with respect to a single unstacked observation x in the middle, the number of observable operators we
need remains constant. Our complexity bounds successfully generalize to this case, since they only rely on P~1 , P2,1 and
P3,x,1 being matrices of probabilities summing to 1 (for the former two) or to Pr[x2 = x] (for the latter), as they are here.
The extension given above for learning HMMs with ambiguous observations differs from the approach suggested by HKZ,
which simply substitutes observations with overlapping tuples of observations (e.g. P2,1 (j, i) = Pr[x3 = j2 , x2 = j1 , x2 =
i2 , x1 = i1 ]). There are two potential problems with the HKZ approach. First, the number of observable operators increases
exponentially with the length of each tuple: there is one observable operator per tuple, instead of one per observation.
Second, P2,1 cannot be decomposed into a product of matrices that includes T , and consequently no longer has rank equal
to the rank of the HMM being modeled. Thus, the learning algorithm could require much more data to recover a correct
model if we use the HKZ approach.
3.5
Learning with Real-Valued Observations
The default RR-HMM formulation assumes discrete observations. However, since the model formulation converts the
discrete observations into n-dimensional probability vectors, and the filtering, smoothing and learning algorithms we discuss
all do the same, it is straightforward to model multivariate continuous data with Kernel Density Estimation [19].
This affects the learning algorithm and inference procedure as follows.
Assume for ease of notation
that the training data consists of N sets of three consecutive continuous observation vectors each, i.e.,
{h~x1,1 , ~x1,2 , ~x1,3 i, h~x2,1 , ~x2,2 , ~x2,3 i, . . . , h~xN,1 , ~xN,2 , ~xN,3 i}, though in practice we could be learning from a single long
sequence (or several). Also assume for now that each observation vector contains a single raw observation, though this
technique can easily be combined with the more sophisticated sequence-based learning and feature-based learning methods
described above. Pick a kernel function K(·) and n kernel centers ~c1 . . . ~cn . (In general we can use different kernels and
centers for different feature vectors.) Let λ be a bandwidth parameter that goes to zero at the appropriate rate in the limit.
~ j iN , hψ
~j iN , hξ~j iN and hζ~j iN , and normalize each to sum to 1:
First compute n × 1 feature vectors hφ
j=1
j=1
j=1
j=1
~ j ]i ∝ K(~xj,1 − ~ci ) [ψ
~j ]i ∝ K(~xj,2 − ~ci )
[φ
[ξ~j ]i ∝ K(~xj,3 − ~ci ) [ζ~j ]i ∝ K ((~xj,2 − ~ci )/λ)
Note that for the second observation (in hζ~j iN
j=1 ) we scale the kernel function by the bandwidth. Then, estimate the vector
P~1 and matrices P2,1 and P3,x,1 (for ~x = ~c1 , . . . , ~cn ) from data:
N
1 X~
Pb1 =
φj
N j=1
N
1 X ~ ~T
Pb2,1 =
ψj φj
N j=1
N
1 X ~ ~ ~T
For x = c1 , . . . , cn : Pb3,x,1 =
[ζj ]x ξj φj
N j=1
We compute n ‘base’ observable operators Bc1 , . . . , Bcn from the estimated probability matrices, as well as vectors ~b1 and
~b∞ , using algorithm L EARN -RR-HMM (Section 3.1). Given these parameters, filtering for a sequence h~x1 , . . . , ~xτ i now
7
A.
B.
RR-HMM
True
10000
100000
eigenvalues
1
2-step-Obs. HMM
True
10000
100000
1
C.
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
1
2
3
0
1
2
3
True
10000
100000
1
0.8
0
2-step-Obs. RR-HMM
0
1
2
3
4
Figure 2: Learning discrete RR-HMMs. The three figures depict the actual eigenvalues of three different RR-HMM transition matrices, and the eigenvalues (95% error bars) of the sum of RR-HMM observable operators estimated with 10, 000
and 100, 000 training observations. (A) A 3-state, 3-observation, rank 2 RR-HMM. (B) A full-rank, 3-state, 2-observation
HMM. (C) A 4-state, 2-observation, rank 3 RR-HMM.
proceeds as follows:
For t = 1, . . . , τ :
Compute and normalize[~σt ]i ∝ K ((~xt − ~ci )/λ) .
n
X
[~σt ]j Bcj
Bσt =
j=1
~bt+1 =
Bσt~bt
~b∞ Bσ ~bt
t
Our theoretical results carry over to the KDE case with modifications described in the RR-HMM document. Essentially, the
bound still holds for predicting functions of ~σ1 , ~σ2 , . . . , ~σt , though we do not yet have results connecting this bound to the
error in estimating probabilities of raw observations.
4
Experimental Results
We designed several experiments to evaluate the properties of RR-HMMs and the learning algorithm both on synthetic
and on real-world data. The first set of experiments (Section 4.1) tests the ability of the spectral learning algorithm to
recover the correct RR-HMM. The second experiment (Section 4.2) evaluates the representational capacity of the RRHMM by learning a model of a video that requires both competitive inhibition and smooth state evolution. The third set of
experiments (Section 4.3) tests the model’s ability to learn, filter, predict, and simulate video captured from a robot moving
in an indoor office environment.
4.1
Learning Synthetic RR-HMMs
First we evaluate the unbiasedness of the spectral learning algorithm for RR-HMMs on 3 synthetic examples. In each case,
we build an RR-HMM, sample observations from the model,
Pand estimate the model with the spectral learning algorithm
described in Section 3. We compare the eigenvalues of B = x Bx in the learned model to the eigenvalues of the transition
matrix T of the true model. B is a similarity transform of S · R which therefore has the same non-zero eigenvalues as
T = RS, so we expect the estimated eigenvalues to converge to the true eigenvalues with enough data. This is a necessary
condition for unbiasedness but not a sufficient one. See Section 9 in Appendix for parameters of HMMs used in the examples
below.
Example 1: An RR-HMM We examine an HMM with m = 3 hidden states, n = 3 observations, a full-rank observation
matrix and a k = 2 rank transition matrix. Figure 2(A) plots the true and estimated eigenvalues for increasing size of
dataset, along with error bars, suggesting that we recover the true dynamic model.
8
A. HMM
B. Stable LDS
C. RR-HMM
4000
−0.17
0
0
−0.1
−0.2
0.15
0.05
0.06
−4000
1
4
0.08
x 10
−0.2
0
0
4
x 10
1
−0.23
0.2
0
0
0.15
−1
Figure 3: The clock video texture simulated by a HMM, a stable LDS, and a RR-HMM. (A) The clock modeled by a
10-state HMM. The manifold consists of the top 3 principal components of predicted observations during simulation. The
generated frames are coherent but motion in the video is jerky. (B) The clock modeled by a 10-dimensional LDS. The
manifold indicates the trajectory of the model in state space during simulation. Motion in the video is smooth but frames
degenerate to superpositions. (C) The clock modeled by a rank 10 RR-HMM. The manifold consists of the trajectory of the
model in the low dimensional subspace of the state space during simulation. Both the motion and the frames are correct.
Example 2: A 2-step-Observable HMM We examine an HMM with m = 3 hidden states, n = 2 observations, and a
full-rank transition matrix (see Appendix for parameters). This HMM violates the m ≤ n condition. The parameters of
this HMM cannot be estimated with the original learning algorithm, since a single observation does not provide enough
information to disambiguate state. By stacking 2 consecutive observations (see Section 3.4), however, the spectral learning
algorithm can be applied successfully (Figure 2(B)).
Example 3: A 2-step-Observable RR-HMM We examine an HMM with m = 4 hidden states, n = 2 observations,
and a k = 3 rank transition matrix (see Appendix for parameters). In this example, the HMM is low rank and multiple
observations are required to disambiguate state. Again, stacking two consecutive observations in conjunction with the
spectral learning algorithm is enough to recover good RR-HMM parameter estimates (Figure 2(C)).
4.2
Competitive Inhibition and Smooth State Evolution in Video
We model a clock pendulum video consisting of 55 frames (with a period of ∼ 22 frames) as a 10-state HMM, a 10dimensional LDS, and a rank 10 RR-HMM with 4 stacked observations. Note that we could easily learn models with
more than 10 latent states/dimensions; we limited the dimensionality in order to demonstrate the relative expressive power
of the different models. For the HMM, we convert the continuous data to discrete observations by 1-NN on 25 kernel
centers sampled sequentially from the training data. We trained the resulting discrete HMM using EM. We learned the LDS
directly from the video using subspace ID with stability constraints [20] using a Hankel matrix of 10 stacked observations.
We trained the RR-HMM by stacking 4 observations, choosing an approximate rank of 10 dimensions, and learning 25
observable operators corresponding to 25 Gaussian kernel centers. We simulate a series of 500 observations from the model
and compare the manifolds underlying the simulated observations and frames from the simulated videos (Figure 3). The
small number of states in the HMM is not sufficient to capture the smooth evolution of the clock: the simulated video is
characterized by realistic looking frames, but exhibits jerky irregular motion. For the LDS, although the 10-dimensional
subspace captures smooth evolution of the simulated video, the system quickly degenerates and individual frames of video
are modeled poorly (resulting in superpositions of pendulums in generated frames). For the RR-HMM, the simulated video
benefits from both smooth state evolution and competitive inhibition. The manifold in the 10-dimensional subspace is
smooth and structured and the video is realistic. The results demonstrate that the RR-HMM has the benefits of smooth state
evolution and compact state space of a LDS and the benefit of competitive inhibition of a HMM.
4.3
Filtering, Prediction, and Simulation with Robot Vision Data
We compare HMMs, LDSs, and RR-HMMs on the problem of modeling video data from a mobile robot in an indoor
environment. A video of 2000 frames was collected at 6 Hz from a Point Grey Bumblebee2 stereo camera mounted on a
Botrics Obot d100 mobile robot platform circling a stationary obstacle (Figure 4(A)) and 1500 frames were used as training
9
Path
Environment
B.
Avg. Prediction Err.
A. Example Images
x 10 6
8.5
7.5
6.5
Mean
Last
RR-HMM
4.5
LDS
HMM
3.5
0 10 20 30 40 50 60 70 80 90 100
5.5
Prediction Horizon
Figure 4: (A) Sample images from the robot’s camera. The figure below depicts the hallway environment with a central
obstacle (black) and the path that the robot took through the environment while collecting data (the red counter-clockwise
ellipse) (B) Squared error for prediction (1, . . . , 100 steps out in future) with different estimated models and baselines,
averaged over different initial filtering durations (1, . . . , 250).
data for each model. Each frame from the training data was reduced to 100 dimensions via SVD on single observations.
Using this training data, we trained an RR-HMM (k = 50, n = 1500) using spectral learning with sequences of 20
continuous observations (Section 3.4) and KDE with Gaussian kernels (Section 3.5) with 1500 centers, a 50-dimensional
LDS using Subspace ID with Hankel matrices of 20 timesteps, and a 50-state HMM with 1500 discrete observations using
EM run until convergence. For each model, we performed filtering for different extents t1 = 100, 101, . . . , 250, then
predicted an image which was a further t2 steps in the future, for t2 = 1, 2 . . . , 100. The squared error of this prediction
in pixel space was recorded, and averaged over all the different filtering extents t1 to obtain means which are plotted in
Figure 4(B). As baselines, we plot the error obtained by using the mean of filtered data as a predictor (‘Mean’), and the
error obtained by using the last filtered observation (‘Last’).
Both baselines perform worse than any of the more complex algorithms (though as expected, the ‘Last’ predictor is a good
one-step predictor), indicating that this is a nontrivial prediction problem. The LDS does well initially (due to smoothness),
and the HMM does well in the longer run (due to competitive inhibition), while the RR-HMM performs as well or better at
both time scales since it models both the smooth state evolution and competitive inhibition in its predictive distribution. In
particular, the RR-HMM yields significantly lower prediction error consistently for the duration of the prediction horizon
(100 timesteps, i.e. 16 seconds).
5
5.1
Related Work
Predictive State Representations
Predictive State Representations (PSRs) [4, 18] and Observable Operator Models (OOMs) [5] model sequence probabilities as a product of observable operator matrices. This idea, as well as the idea of learning such models using linear
algebra techniques, originates in the literature on multiplicity automata and weighted automata [7, 8, 6]. Despite recent
improvements [21, 22], practical learning algorithms for PSRs and OOMs have been lacking. RR-HMMs and its spectral
learning algorithm are also closely related to methods in subspace identification [9, 17] in control systems for learning LDS
parameters, which use SVD to determine the relationship between hidden states and observations.
As pointed out earlier, the spectral learning algorithm presented here learns PSRs. We briefly discuss other algorithms
for learning PSRs from data. Several learning algorithms for PSRs have been proposed [23, 12, 24]. It is easier for PSR
learning algorithms to return consistent parameter estimates because the parameters are based on observable quantities.
[11] develops an SVD-based method for finding a low-dimensional variant of PSRs, called Transformed PSRs (TPSRs).
Instead of tracking the probabilities of a small number of tests, TPSRs track a small number of linear combinations of a
larger number of tests. This allows more compact representations, as well as dimensionality selection based on examining
10
the singular values of the decomposed matrix, as in subspace identification methods. Note that nonlinearity can be encoded
into the design of core tests. [25] introduced the concept of e-tests in PSRs that are indicator functions of aggregate
sets of future outcomes, e.g. all sequence of observations in the immediate future that end with a particular observation
after k timesteps. In general, tests in discrete PSRs can be indicator functions of arbitrary statistics of future events, thus
encoding nonlinearities that might be essential for modeling some dynamical systems. Recently, Exponential Family PSRs
(EFPSRs) [21] were introduced as an attempt to generalize the PLG model to allow general exponential family distributions
over the next N observations. In the EFPSR, state is represented by modeling the parameters of a time-varying exponential
family distribution over the next N timesteps. This allows graphical structure to be encoded in the distribution, by choosing
the parameters accordingly. The justification for choosing an exponential family comes from maximum entropy modeling.
Though inference and parameter learning are difficult in graphical models of non-trivial structure, approximate inference
methods can be utilized to make these problems tractable. Like PLGs, the dynamical component of EFPSRs is modeled by
extending and conditioning the distribution over time. However, the method presented [21] has some drawbacks, e.g. the
extend-and-condition method is inconsistent with respect to marginals over individual timesteps between the extended and
un-extended distributions.
5.2
Hybrid Models, Mixture Models and other recent approaches
RR-HMMs and their algorithms are also related to other hybrid models. Note that previous models of the same name
(e.g. [26]) address a completely different problem, i.e. reducing the rank of the Gaussian observation parameters. Since
shortly after the advent of LDSs, there have been attempts to combine the discrete states of HMMs with the smooth dynamics of LDSs. We perform a brief review of the literature on hybrid models; see [27] for a more thorough review. [28]
formulates a switching LDS variant where both the state and observation variable noise models are mixture of Gaussians
with the mixture switching variable evolving according to Markovian dynamics, and derives the (intractable) optimal filtering equations where the number of Gaussians needed to represent the belief increases exponentially over time. They also
propose an approximate filtering algorithm for this model based on a single Gaussian. [29] proposes learning algorithms
for an LDS with switching observation matrices. [30] reviews models where both the observations and state variable switch
according to a discrete variable with Markov transitions. Hidden Filter HMMs (HFHMMs) [31] combine discrete and realvalued state variables and outputs that depend on both. The real-valued state is deterministically dependent on previous
observations in a known manner, and only the discrete variable is hidden. This allows exact inference in this model to be
tractable. [32] formulates the Mixture Kalman Filter (MKF) model along with a filtering algorithm, similar to [28] except
that the filtering algorithm is based on sequential Monte-Carlo sampling.
The commonly used HMMs with mixture-model observations (e.g., Gaussian mixture) are a special case of RR-HMMs. A kstate HMM where each state corresponds to a Gaussian mixture of m observation models of n dimensions each is subsumed
by a k-rank RR-HMM with m distinct continuous observations of n dimensions each, since the former is constrained to
be non-negative and ≤ 1 in various places (the k-dimensional transition matrix, the k-dimensional belief vector, the matrix
which transforms this belief to observation probabilities) where the latter is not.
Switching State-Space Models (SSSMs) [27] posit the existence of several real-valued hidden state variables that evolve
linearly, with a single Markovian discrete-valued switching variable selecting the state which explains the real-valued observation at every timestep. Since exact inference and learning are intractable in this model, the authors derive a structured
variational approximation that decouples the state space and switching variable chains, effectively resulting in Kalman
smoothing on the state space variables and HMM forward-backward on the switching variable. In their experiments, the
authors find SSSMs to perform better than regular LDSs on a physiological data modeling task with multiple distinct underlying dynamical models. HMMs performed comparably well in terms of log-likelihood, indicating their ability to model
nonlinear dynamics though the resulting model was less interpretable than the best SSSM. More recently, models for nonlinear time series modeling such as Gaussian Process Dynamical Models have been proposed [33]. However, the parameter
learning algorithm is only locally optimal, and exact inference and simulation are very expensive, requiring MCMC over a
long sequence of frames all at once. This necessitates the use of heuristics for both inference and learning. Another recent
nonlinear dynamic model is [34], which differs greatly from other methods in that it treats each component of the dynamic
model learning problem separately using supervised learning algorithms, and proves consistency on the aggregate result
under certain strong assumptions.
6
Discussion
The spectral learning algorithm blurs the line between latent variable models and PSRs. PSRs were developed with a focus
on the problem of an agent planning actions in a partially observable environment. More generally, there are many scenarios
in sequential data modeling where the underlying dynamical system has inputs. The inference task for a learned model is
then to track the belief state while conditioning on observations and incorporating the inputs. The input-output HMM (IO11
HMM) [35] is a conditional probabilistic model which has these properties. A natural generalization of this work is to the
task of learning RR-HMMs with inputs, or controlled PSRs. We recently carried out this generalization to controlled PSRs;
details can be found in [10].
The question of proving containment or equivalence of RR-HMMs with respect to PSRs is of theoretical interest. The
observable representation of an RR-HMM is a Transformed PSR (TPSR) [11], so every RR-HMM is a PSR; it remains to
be seen whether every PSR corresponds to some RR-HMM (possibly with an infinite number of discrete hidden states) as
well. The idea that “difficult” PSRs should somehow correspond to RR-HMMs with very large or infinite state space is
intuitively appealing but not straightforward to prove. Another interesting direction would be to bound the performance of
the learning algorithm when the underlying model is only approximately a reduced-rank HMM, much as the HKZ algorithm
includes bounds when the underlying model is approximately an HMM [1]. This would be useful since in practice it is more
realistic to expect any underlying system to not comply with the exact model assumptions.
The positive realization problem, i.e. obtaining stochastic transition and observation matrices from the RR-HMM observable
representation, is also significant, though the observable representation allows us to carry out all possible HMM operations.
HKZ describes a method based on [36] which, however, is highly erratic in practice. In the RR-HMM case, we have the
additional challenge of firstly computing the minimal m for which a positive realization exists, and since the algorithm
learns PSRs there is no guarantee that a particular set of learned parameters conforms exactly to any RR-HMM. On the
applications side, it would be interesting to compare RR-HMMs with other dynamical models on classification tasks, as
well as on learning models of difficult video modeling and graphics problems for simulation purposes. More elaborate
choices of features may be useful in such applications, as would be the usage of high-dimensional or infinite-dimensional
features via Reducing Kernel Hilbert Spaces (RKHS).
Acknowledgements
We acknowledge helpful conversations with Sham Kakade regarding the HKZ spectral learning algorithm [1]. Julian Ramos
assisted with the gathering of robot vision data used in our experiments. SMS was supported by the NSF under grant number
0000164, the USAF under grant number FA8650-05-C-7264, the USDA under grant number 4400161514, and a project
with MobileFusion/TTC. BEB was supported by the NSF under grant number EEEC-0540865. GJG was supported by
DARPA under grant number HR0011-07-10026, the Computer Science Study Panel program, and by DARPA/ARO under
MURI grant number W911NF-08-1-0301. BEB and GJG were both supported by ONR MURI grant number N00014-091-1052.
References
[1] Daniel Hsu, Sham Kakade, and Tong Zhang. A spectral algorithm for learning hidden markov models. In COLT,
2009.
[2] L. R. Rabiner. A tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proc. IEEE,
77(2):257–285, 1989.
[3] Zoubin Ghahramani and Geoffrey E. Hinton. Parameter estimation for Linear Dynamical Systems. Technical Report
CRG-TR-96-2, U. of Toronto, Department of Comp. Sci., 1996.
[4] Michael Littman, Richard Sutton, and Satinder Singh. Predictive representations of state. In Advances in Neural
Information Processing Systems (NIPS), 2002.
[5] Herbert Jaeger. Observable operator models for discrete stochastic time series. Neural Computation, 12:1371–1398,
2000.
[6] Vijay Balasubramanian. Equivalence and Reduction of Hidden Markov Models. MSc. Thesis, MIT, 1993.
[7] M. P. Schützenberger. On the definition of a family of automata. Inf Control, 4:245–270, 1961.
[8] M. Fleiss. Matrices deHankel. J. Math. Pures Appl., 53:197–222, 1974.
[9] P. Van Overschee and B. De Moor. Subspace Identification for Linear Systems: Theory, Implementation, Applications.
Kluwer, 1996.
[10] Byron Boots, Sajid M. Siddiqi, and Geoffrey J. Gordon. Closing the Learning-Planning Loop with Predictive State
Representations. http://arxiv.org/abs/0912.2385, 2009.
[11] Matthew Rosencrantz and Geoffrey J. Gordon. Learning low dimensional predictive representations. In Proc. ICML,
2004.
12
[12] Michael James and Satinder Singh. Learning and discovery predictive state representations in dynamical systems with
reset. In Proc. ICML, 2004.
[13] L. Baum. An inequality and associated maximization technique in statistical estimation of probabilistic functions of a
Markov process. Inequalities, 3:1–8, 1972.
[14] Sajid Siddiqi, Geoffrey J. Gordon, and Andrew Moore. Fast state discovery for HMM model selection and learning.
In Proc. AISTATS, 2007.
[15] Roger Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1985.
[16] Patrick O. Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research, 5:1457–1469, 2004.
[17] Tohru Katayama. Subspace Methods for System Identification: A Realization Approach. Springer, 2005.
[18] Satinder Singh, Michael James, and Matthew Rudary. Predictive state representations: A new theory for modeling
dynamical systems. In Proc. UAI, 2004.
[19] B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, 1986.
[20] Sajid Siddiqi, Byron Boots, and Geoffrey J. Gordon. A constraint generation approach to learning stable linear dynamical systems. In Proc. NIPS, 2007.
[21] David Wingate and Satinder Singh. Exponential family predictive representations of state. In Proc. NIPS, 2007.
[22] Ming-Jie Zhao and Herbert Jaeger and Michael Thon. A Bound on Modeling Error in Observable Operator Models
and an Associated Learning Algorithm. Neural Computation.
[23] Satinder Singh, Michael L. Littman, Nicholas K. Jong, David Pardoe, and Peter Stone. Learning predictive state
representations. In Proc. ICML, 2003.
[24] Britton Wolfe, Michael James, and Satinder Singh. Learning predictive state representations in dynamical systems
without reset. In Proc. ICML, 2005.
[25] Matthew Rudary and Satinder Singh. A nonlinear predictive state representation. In Proc. NIPS, 2003.
[26] Nagendra Kumar and Andreas G. Andreou. Heteroscedastic discriminant analysis and reduced rank hmms for improved speech recognition. Journal of Speech Communication, 26:283–297, 1998.
[27] Zoubin Ghahramani and Geoffrey E. Hinton. Variational learning for switching state-space models. Neural Computation, 12(4), 2000.
[28] G. A. Ackerson and K. S. Fu. On state estimation in switching environments. IEEE Transactions on Automatic
Control, 15(1):10–17, January 1970.
[29] R. H. Shumway and D. S. Stoffer. Dynamic linear models with switching. J. Amer. Stat. Assoc., 86:763–769, 1993.
[30] Y. Bar-Shalom and X. R. Li. Estimation and Tracking. Artech House, 1993.
[31] A. M. Fraser and A. Dimitriadis. Forecasting probability densities by using hidden markov models with mixed states,
1993.
[32] R. Chen and J. Liu. Mixture kalman filters. Journal of the Royal Statistical Society B, 62:493–508, 2000.
[33] J. M. Wang, D. J. Fleet, and A. Hertzmann. Gaussian process dynamical models. In Proc. NIPS, 2005.
[34] John Langford and Ruslan Salakhutdinov and Tong Zhang. Learning Nonlinear Dynamic Models. In ICML, 2009.
[35] Yoshua Bengio and Paolo Frasconi. An Input Output HMM Architecture. In Advances in Neural Information Processing Systems, 1995.
[36] E. Mossel and S. Roch. Learning nonsingular phylogenies and hidden Markov models. Annals of Applied Probability,
2:583–614, 2006.
[37] G. W. Stewart and Ji-Guang Sun. Matrix Perturbation Theory. Academic Press, 1990.
[38] Per-Åke Wedin. Perturbation Bounds in Connection with Singular Value Decomposition. BIT Numer. Math., 12:99–
111, 1972.
[39] G. W. Stewart. Matrix Algorithms Vol 1: Basic Decompositions. SIAM, 1998.
[40] Colin McDiarmid. On the method of bounded differences. Surveys in Combinatorics, pages 148–188, 1989.
13
7
Appendix I: Proofs
The proof of Theorem 2 relies on Lemmas 18 and 24. We start off with some preliminary results and build up to proving
the main theorem and its lemmas below.
A remark on norms: The notation kXkp for matrices X ∈ Rm×n denotes the operator norm max
kXvkp
kvkp
for vector v 6= 0.
Specifically, kXk2 denotes L2 matrix norm (also known as spectral norm), which corresponds to the largest singular value
P
1/2
m Pn
2
σ1 (X). Frobenius norm is denoted by kXkF =
X
. The notation kXk1 for matrices denotes the L1
i=1
j=1 ij
Pm
matrix norm which corresponds to maximum absolute column sum maxc i=1 |Xic |. The definition of kxkp for vectors
Pn
1/p
x ∈ Rn is the standard distance measure ( i=1 xpi ) .
7.1
Preliminaries
The following conditions are assumed by the main theorems and algorithms.
Condition 3 [Modification of HKZ Condition 1] ~π > 0 element-wise, T has rank k (i.e. R and S both have rank k) and O
has rank at least k.
The following two conditions on R can always be satisfied by scaling down entries in R and scaling up S accordingly.
However we want entries in R to be as large as possible under the two conditions below, so that σk (U T OR) is large and
1/σk (U T OR) is small to make the error bound as tight as possible (Theorem 2). Hence we pay for scaling down R by
loosening the error bound we obtain for a given number of training samples.
Condition 4 kRk1 ≤ 1.
Condition 5 For some column 1 ≤ c ≤ k of R, it is the case that kR[·, c]k2 ≤
p
k/m.
The above two conditions on R ensure the bounds go through largely unchanged from HKZ aside from the improvement
due to low rank k. The first condition can be satisfied in a variety of ways without loss of generality, e.g. by choosing
the columns of R to be any k independent columns of T , and S to be the coefficients needed to reconstruct T from R.
Intuitively, the first condition implies that R does not overly magnify the magnitude of vectors it multiplies with. The
second one implies a certain degree of uniformity in at least one of the columns of R. For example, the uniform distribution
in a column of R would satisfy the constraint, whereas a column of the identity matrix would not. This does not imply that
T must have a similarly near-uniform column. We can form R from the uniform distribution along with some independent
columns of T.
The observable representation depends on a matrix U ∈ Rn×k that obeys the following condition:
Condition 6 [Modification of HKZ Condition 2] U T OR is invertible.
This is analogous to the HKZ invertibility condition on U T O, since OR is the matrix that yields observation probabilities
from a low-dimensional state vector. Hence, U defines a k-dimensional subspace that preserves the low-dimensional state
dynamics regardless of the number of states m.
Condition 7 Assume that S diag(~π )OT has full row rank (i.e. k).
This condition amounts to ensuring that that the ranges S and O, which are both at least rank k, overlap enough to preserve
the dynamics when mapping down to the low-dimensional state. As in HKZ, the left singular vectors of P2,1 give us a valid
U matrix.
Lemma 8 [Modification of HKZ Lemma 2] Assume Conditions 3 and 7. Then, rank(P2,1 ) = k. Also, if U is the matrix of
left singular vectors of P2,1 corresponding to non-zero singular values, then range(U ) = range(OR), so U ∈ Rn×k obeys
Condition 6.
14
P ROOF : From its definition, we can show P2,1 can be written as a low-rank product of RR-HMM parameters:
[P2,1 ]i,j = Pr[x2 = i, x1 = j]
m X
m
X
=
Pr[x2 = i, x1 = j, h2 = a, h1 = b] (marginalizing hidden states h)
=
=
a=1 b=1
m X
m
X
a=1 b=1
m X
m
X
Pr[x2 = i|h2 = a] Pr[h2 = a|h1 = b] Pr[x1 = j|h1 = b] Pr[h1 = b]
Oia Tab~πb [OT ]bj
a=1 b=1
⇒ P2,1 = OT diag(~π )OT
= ORS diag(~π )OT
Thus range(P2,1 ) ⊆ range(OR). This shows that rank(P2,1 ) ≤ rank(OR).
By Condition 7, S diag(~π )OT has full row rank, thus S diag(~π )OT (S diag(~π )OT )+ = Ik×k . Therefore,
OR = P2,1 (S diag(~π )OT )+
which implies range(OR) ⊆ range(P2,1 ), which in turn implies rank(OR) ≤ rank(P2,1 ).
(7.1)
(7.2)
Together this proves that rank(P2,1 ) = rank(OR), which we can show to be k as follows: Condition 3 implies that
rank (U T OR) = k, and hence rank(OR) ≥ k. Since OR ∈ Rm×k , rank(OR) ≤ k. Therefore, rank(OR) = k. Hence
rank(P2,1 ) = k.
Since range(U ) = range(P2,1 ) by definition of singular vectors, this implies range(U ) = range(OR). Therefore,
(U T OR) is invertible and hence U obeys Condition 6.
The following lemma shows that the observable representation {~b∞ , ~b1 , B1 , . . . , Bn } is linearly related to the true HMM
parameters, and can compute the probability of any sequence of observations.
Lemma 9 [Modification of HKZ Lemma 3] (Observable HMM Representation). Assume Condition 3 on the RR-HMM and
Condition 6 on the matrix U ∈ Rn×k . Then, the observable representation of the RR-HMM (Definition 1 of the paper) has
the following properties:
1. ~b1
2. ~bT
∞
=
=
(U T OR)πl
(U T O)π,
=
T
−1
,
1T
m R(U OR)
3. For all x = 1, . . . , n : Bx
4. For any time t: Pr[x1:t ]
=
=
(U T OR)Wx (U T OR)−1
~bT Bx :1~b1
∞
t
P ROOF :
1. We can write P~1 as Oπ, since
[P~1 ]i = Pr[x1 = i]
m
X
=
Pr[x1 = i|h1 = a] Pr[h1 = a]
=
a=1
m
X
Oia~πa
a=1
Combined with the fact that ~b1 = U T P~1 by definition, this proves the first claim.
2. Firstly note that P~1T = ~1T
π )OT , since
m T diag(~
P~1T = ~π T OT
= ~1T
π )OT
m diag(~
= ~1T
π )OT
m T diag(~
15
~T
(since ~1T
m T = 1m )
This allows us to write P~1 in the following form:
T
P~1T = ~1T
m T diag(π)O
T
= ~1T
m RS diag(π)O
T
−1
= ~1T
(U T OR)S diag(π)OT
m R(U OR)
T
−1 T
= ~1T
U P2,1 (by equation (7.1))
m R(U OR)
By equation (7.1), U T P2,1 = (U T OR)S diag(~π )OT . Since (U T OR) is invertible by Condition 6, and
S diag(~π )OT has full row rank by Condition 7, we know that (U T P2,1 )+ exists and
U T P2,1 (U T P2,1 )+ = Ik×k
Therefore,
T
T
+
T
T
−1
T
−1
bT
(U T P2,1 )(U T P2,1 )+ = 1T
∞ = P1 (U P2,1 ) = 1m R(U OR)
m R(U OR)
hence proving the second claim.
3. The third claim can be proven by first expressing P3,x,1 as a product of RR-HMM parameters:
[P3,x,1 ]ij = Pr[x3 = i, x2 = x, x1 = j]
m X
m X
m
X
=
Pr[x3 = i, x2 = x, x1 = j, h3 = a, h2 = b, h1 = c]
=
a=1 b=1 c=1
m X
m
m X
X
Pr[x3 = i|h3 = a] Pr[h3 = a|h2 = b] Pr[x2 = x|h2 = b]
a=1 b=1 c=1
Pr[h2 = b|h1 = c] Pr[h1 = c] Pr[x1 = j|h1 = c]
=
m X
m X
m
X
Oia [Ax ]ab Tbc~πc [OT ]cj
a=1 b=1 c=1
⇒ P3,x,1 = OAx T diag(~π )OT
This can be transformed as follows:
P3,x,1 = OAx RS diag(~π )OT
= OAx R(U T OR)−1 (U T OR)S diag(~π )OT
= OAx R(U T OR)−1 U T (OT diag(~π )OT )
= OAx R(U T OR)−1 U T P2,1
(by equation (7.1))
and then plugging in this expression into the definition of Bx , we obtain the required result:
Bx = (U T P3,x,1 )(U T P2,1 )+
(by definition)
−1
(U T P2,1 )(U T P2,1 )+
= (U T O)Ax R(U T OR)−1
(by equation (7.3))
T
T
= (U O)Ax R(U OR)
= (U OR) (S diag(Ox,· )R) (U T OR)−1
T
= (U T OR)Wx (U T OR)−1
4. Using the above three results, the fourth claim follows from equation 1 in Section 2 in the paper:
Pr[x1 , . . . , xt ]
= ~1T
πl
m RWxt . . . Wx1 ~
T
−1
= ~1T
(U T OR)Wxt (U T OR)−1 (U T OR)Wxt−1 (U T OR)−1 . . .
m R(U OR)
. . . (U T OR)Wx1 (U T OR)−1 (U T OR)~πl
= ~bT Bx . . . Bx ~b1
∞
t
1
~
= ~bT
∞ Bxt:1 b1
16
(7.3)
In addition to ~b1 above, we define normalized conditional ‘internal states’ ~bt that help us compute conditional probabilities.
These internal states are not probabilities. In contrast to HKZ where these internal states are m-dimensional vectors, in our
case the internal states are k-dimensional i.e. they correspond to the rank of the HMM. As shown above in Lemma 9,
~b1 = (U T OR)~πl = (U T O)~π
In addition for any t ≥ 1, given observations x1:t−1 with non-zero probability, the internal state is defined as:
~bt = ~bt (x1:t−1 ) =
Bxt−1:1~b1
~bT Bx :1~b1
∞
t−1
(7.4)
T
−1
~
~T
For t = 1 the formula is still consistent since ~bT
(U T OR)~πl = ~1T
πl = ~1T
π = 1.
∞ b1 = 1m R(U OR)
m R~
m~
Recall that HMM and RR-HMM parameters can be used to calculate joint probabilities as follows:
Pr[x1 , ..., xt ] = ~1T
π
m Axt Axt−1 · · · Ax1 ~
T
= ~1m RS diag(Oxt ,· )RS diag(Oxt−1 ,· )R · · · S diag(Ox1 ,· )~π
= ~1T
π
m R (S diag(Oxt ,· )R) S diag(Oxt−1 ,· )R · · · (S diag(Ox1 ,· )R) S~
= ~1T
πl
m RWxt . . . Wx1 ~
(by definition of Wx , ~πl )
(7.5)
The following Lemma shows that the conditional internal states are linearly related, and also shows how we can use them
to compute conditional probabilities.
Lemma 10 [Modification of HKZ Lemma 4] (Conditional Internal States) Assume the conditions of Lemma 9 hold, i.e.
Conditions 3 and 6 hold. Then, for any time t:
1. (Recursive update) If Pr[x1 , . . . , xt ] > 0, then
~bt+1 =
Bxt~bt
~bT Bx ~bt
∞
t
2. (Relation to hidden states)
~bt = (U T OR)lt (x1:t−1 ) = (U T O)ht (x1:t−1 )
where [~ht (x1:t−1 )]i = Pr[ht = i|x1:t−1 ] is defined as the conditional probability of the hidden state at time t given
observations x1:t−1 , and ~lt (x1:t−1 ) is its low-dimensional projection such that ~ht (x1:t−1 ) = R~lt (x1:t−1 ).
3. (Conditional observation probabilities)
~
Pr[xt |x1:t−1 ] = ~bT
∞ Bxt bt
P ROOF : The first proof is direct, the second follows by induction.
1. The t = 2 case ~b2 =
Bx1~b1
~bT Bx ~b1
∞
1
is true by definition (equation 7.4). For t ≥ 3, again by definition of bt+1 we have
~bt+1 =
Bxt:1~b1
~bT Bx ~b1
∞
t:1
~
=
Bxt Bxt−1:1 b1
~bT Bx
~b
∞
t−1:1 1 ~ T
~
~bT Bx
~b1 b∞ Bxt Bxt−1:1 b1
∞
=
t−1:1
Bxt~bt
~
~bT Bx Bxt−1:1 b1
∞
t~
bT B
∞
=
Bxt~bt
~bT Bx ~bt
∞
(by equation (7.4))
xt−1:1~
b1
(by equation (7.4))
t
17
2,3. The base case for claim 2 holds by Lemma 9, since ~h1 = ~π , ~l1 = R~π and ~b1 = (U T OR)~π . For claim 3, the base
~
~T
case holds since ~bT
πl by Lemma 9, which equals Pr[x1 ] by equation (7.5). The inductive step
∞ Bx1 b1 = 1m RWx1 ~
is:
~
~bt+1 = Bxt bt
(by claim 1 above)
T
~b Bx ~bt
∞
t
Bxt (U T OR)~lt
(by inductive hypothesis)
=
Pr[xt |x1:t−1 ]
(U T OR)Wxt~lt
(by Lemma 9)
=
Pr[xt |x1:t−1 ]
(U T O)Axt ~ht
=
(∵ RWxt~lt = RS diag(Oxt ,· )R~lt = Axt~lt )
Pr[xt |x1:t−1 ]
Now by definition of Axt ~ht ,
~bt+1 = (U T O) Pr[ht+1 = ·, xt |x1:t−1 ]
Pr[xt |x1:t−1 ]
Pr[ht+1 = ·|x1:t ] Pr[xt |x1:t−1 ]
= (U T O)
Pr[xt |x1:t−1 ]
T
~
= (U O)ht+1 (x1:t )
= (U T OR)~lt+1 (x1:t )
This proves claim 2, using which we can complete the proof for claim 3:
~bT Bx ~bt+1 = ~1T R(U T OR)−1 (U T OR)Wx (U T OR)−1~bt+1
∞
m
t+1
t
T
−1
= ~1T
(U T OR)~lt+1
m RWxt (U OR)
~
= ~1T
m RWxt lt+1
~
= ~1T
m RS diag(Oxt ,· )Rlt+1
~
= ~1T
m T diag(Oxt ,· )ht+1
(by Lemma 9)
(by claim 2 above)
(by definition of Wxt )
~
= ~1T
m Axt ht+1
Again by definition of Axt ~ht+1 ,
~bT Bx ~bt+1 =
∞
t+1
=
m
m X
X
a=1 b=1
m X
m
X
Pr[xt+1 |ht+1 = a] Pr[ht+1 = a|ht = b] Pr[ht = b|x1:t ]
Pr[xt+1 , ht+1 = a, ht = b|x1:t ]
a=1 b=1
= Pr[xt+1 |x1:t ]
Remark 11 If U is the matrix of left singular vectors of P2,1 corresponding to non-zero singular values, then U is the
observable-representation analogue of the observation probability matrix O in the sense that, given a conditional state ~bt ,
Pr[xt = i|x1:t−1 ] = [U~bt ]i in the same way as Pr[xt = i|x1:t−1 ] = [O~ht ]i for a conditional hidden state ~ht .
P ROOF : Since range(U ) = range(OR) (Lemma 2), and U U T is a projection operator to range(U ), we have U U T OR =
OR, so U~bt = U (U T OR)lt = ORlt = Oht .
7.2
Matrix Perturbation Theory
We take a diversion to matrix perturbation theory and state some standard theorems from Steward and Sun (1990) [37]
and Wedin (1972) [38] which we will use, and also prove a result from these theorems. The following lemma bounds the
L2 -norm difference between the pseudoinverse of a matrix and the pseudoinverse of its perturbation.
18
e = A + E. Then,
Lemma 12 (Theorem 3.8 of Stewart and Sun (1990) [37]) Let A ∈ Rm×n , with m ≥ n, and let A
√
2 + 2
1+ 5
e+
e kEk
.
· max A+ 2 , A
A − A+ ≤
2
2
2
2
The following lemma bounds the absolute differences between the singular values of a matrix and its perturbation.
e = A + E. If the
Lemma 13 (Theorem 4.11 of Stewart and Sun (1990) [37]). Let A ∈ Rm×n with m ≥ n, and let A
e
singular values of A and A are (σ1 ≥ . . . ≥ σn ) and (e
σ1 ≥ . . . ≥ σ
en ), respectively, then
|e
σi − σi | ≤ kEk2 i = 1, . . . , n
Before the next lemma we must define the notion of canonical angles between two subspaces:
Definition 14 (Adapted from definition 4.35 of Stewart (1998) [39]) Let X and Y be matrices whose columns comprise
orthonormal bases of two p-dimensional subspaces X and Y respectively. Let the singular values of X T Y (where X T
denotes the conjugate transpose, or Hermitian, of matrix X) be γ1 , γ2 , . . . , γp . Then the canonical angles θi between X and
Y are defined by
θi = cos−1 γi , i = 1, 2, . . . , p
The matrix of canonical angles Θ is defined as
Θ(X , Y) = diag(θ1 , θ2 , . . . , θp )
Note that ∀i γi ∈ [0, 1] in the above definition, since γ1 (assuming it’s the highest singular value) is no greater than
σ1 (X T )σ1 (Y ) ≤ 1 · 1 = 1, and hence cos−1 γi is always well-defined.
For any matrix A, define A⊥ to be the orthogonal complement of the subspace spanned by the columns of A. For example,
any subset of left singular vectors of a matrix comprise the orthogonal complement of the matrix composed of the remaining
left singular vectors. The following lemma gives us a convenient way of calculating the sines of the canonical angles between
two subspaces using orthogonal complements:
Lemma 15 (Theorem 4.37 of Stewart (1998) [39]) Let X and Y be n × p matrices, with n > p, whose columns comprise
orthonormal bases of two p-dimensional subspaces X and Y respectively. Assume X⊥ , Y⊥ ∈ Rn×n−p such that [X X⊥ ]
and [Y Y⊥ ] are orthogonal matrices. The singular values of Y⊥T X are the sines of the canonical angles between X and Y.
The following lemma bounds the L2 -norm difference between the sine of the canonical angle matrices of the range of a
matrix and its perturbation.
Lemma 16 ([38],Theorem 4.4 of Stewart and Sun (1990) [37]). Let A ∈ Rm×n with m ≥ n, with the singular value
decomposition (U1 , U2 , U3 , Σ1 , Σ2 , V1 , V2 ):
 T 
"
#
U1
Σ1 0
T
 U2  A [ V1 V2 ] =
0 Σ2
0
0
U3T
e = A + E, with analogous SVD (U
e1 , U
e2 , U
e3 , Σ
e 1, Σ
e 2, Σ
e 3 , Ve1 , Ve2 ). Let Φ be the matrix of canonical angles between
Let A
e1 ), and Θ be the matrix of canonical angles between range(V1 ) and range(Ve1 ). If there exists
range(U1 ) and range(U
e 1 ) ≥ α + δ and max σ(Σ2 ) ≤ α, then
δ > 0, α ≥ 0 such that min σ(Σ
max {ksin Φk2 , ksin Θk2 } ≤
kEk2
δ
The above two lemmas can be adapted to prove that Corollary 22 of HKZ holds for the low-rank case as well, assuming
that the perturbation is bounded by a number less than σk . The following lemma shows that (1) the k th singular value of
a matrix and its perturbation are close to each other, and (2) that the subspace spanned by the first k singular vectors of a
matrix is nearly orthogonal to the subspace spanned by the (k + 1)th , . . . , mth singular vectors of its perturbation, with the
matrix product of their bases being bounded.
Corollary 17 [Modification of HKZ Corollary 22] Let A ∈ Rm×n , with m ≥ n, have rank k < n, and let U ∈ Rm×k be
e = A+E.
the matrix of k left singular vectors corresponding to the non-zero singular values σ1 ≥ . . . ≥ σk ≥ 0 of A. Let A
m×k
e
e
Let U ∈ R
be the matrix of k left singular vectors corresponding to the largest k singular values σ
e1 ≥ . . . ≥ σ
ek of A,
m×(m−k)
e
and let U⊥ ∈ R
be the remaining left singular vectors. Assume kEk2 ≤ σk for some < 1. Then:
19
1. σ
ek ≥ (1 − )σk .
eT σk .
2. U
U
⊥ ≤ kEk2 /e
2
P ROOF :
1. From Lemma 13,
|e
σk − σk | ≤ kEk2
|e
σk − σk | ≤ σk
σ
ek − σk ≥ −σk
σ
ek ≥ (1 − )σk
which proves the first claim.
2. Recall that by Lemma 15, if Φ is a matrix of all canonical angles between range(P2,1 ) and range(Pb2,1 ), then sin Φ
e T U along its diagonal.
contains all the singular values of U
⊥
Also recall that the L2 norm of a matrix is its top singular value. Then,
ksin Φk2 = σ1 (sin Φ) (by definition)
= max diag(sin Φ) (since sin Φ is a diagonal matrix)
e T U ) (by Lemma 15)
= σ1 ( U
⊥
eT = U⊥ U (by definition)
2
Invoking Lemma 16 with the parameter values δ = σ
ek and α = 0 yields ksin Φk2 ≤ kEk2 /e
σk . Combining this
eT with ksin Φk = (U U ) proves claim 2.
2
⊥
2
7.3
Supporting Lemmas
In this section we develop the main supporting lemmas that help us prove Theorem 2
7.3.1
Estimation Errors
We define 1 ,2,1 and 3,x,1 as sampling errors for P~1 ,P2,1 and P3,x,1 respectively:
1 = Pb1 − P~1 F
b
2,1 = P2,1 − P2,1 F
3,x,1 = Pb3,x,1 − P3,x,1 for x = 1, . . . , n
F
(7.6a)
(7.6b)
(7.6c)
Lemma 18 [Modification of HKZ Lemma 8] If the algorithm independently samples N observation triples from the HMM,
then with probability at least 1 − η:
r
r
1
3
1
1 ≤
ln +
N
η
N
r
r
1
3
1
2,1 ≤
ln +
N
η
N
r
r
1
3
1
max 3,x,1 ≤
ln +
x
N
η
N
s
! r
r
r
X
k
3
k
1
3
1
3,x,1 ≤ min
ln +
+ 2(k) +
ln +
k
N
η
N
N
η
N
x
20
Before proving this lemma, we need some definitions and a preliminary result. First, we restate McDiarmid’s Inequality [40]:
Theorem 19 Let Z1 , . . . , Zm be independent random variables all taking values in the set Z. Let ci be some positive real
numbers. Further, let f : Z m 7→ R be a function of Z1 , . . . , Zm that satisfies ∀i, ∀z1 , . . . , zm , zi0 ∈ Z,
|f (z1 , . . . , zi , . . . , zm ) − f (z1 , . . . , zi0 , . . . , zm )| ≤ ci .
Then for all > 0,
Pr[f − E[f ] ≥ ] ≤ exp
−22
Pm 2
i=1 ci
.
Assume z is a discrete random variable that takes on values in 1, . . . , d. The goal is to estimate the vector ~q = [Pr(z =
j)]dj=1 from N i.i.d. samples zi (i = 1, . . . , N ). Let ej denote the j th column of the d×d identity matrix. For i = 1, . . . , N ,
suppose ~qi is a column of the d × d identity matrix such that ~qi (j) = ezi . In other words, the zith component of ~qi is 1 and
PN
the rest are 0. Then the empirical estimate of ~q in terms of ~qi is qb = i=1 ~qi /N .
Each part of Lemma 18 corresponds to bounding, for some ~q, the quantity
2
kb
q − ~qk2
.
We first state a result based on McDiarmid’s inequality (Theorem 19):
Proposition 20 [Modification of HKZ Proposition 19] For all > 0 and qb, ~q and N as defined above:
√
2
Pr kb
q − ~qk2 ≥ 1/ N + ≤ e−N PN
PN
P ROOF : Recall qb = i=1 ~qi /N , and define pb = i=1 p~i /N where p~i = ~qi except for i = k, and pk is an arbitrary column
of the appropriate-sized identity matrix. Then we have
kb
q − ~qk2 − kb
p − ~qk2 ≤ kb
q − pbk2 (by triangle inequality)
X
X
= (
~qi )/N − (
p~i )/N i
i
= (1/N ) k~qk − p~k k2
p
≤ (1/N ) 12 + 12
√
= 2/N
2
(by definition of pb, qb and L2 -norm)
This shows that kb
q − ~qk2 is a function of random variables ~q1 , . . . , ~qN such changing √
the k th random variable qk for any
1 ≤ k ≤ N (resulting in kb
p − ~qk2 ) changes the value of the function by at most ck = 2/N . Note that ~q is not a random
variable but rather the variable we are trying to estimate. In this case, McDiarmid’s inequality (Theorem 19) bounds the
deviation kb
q − ~qk2 from its expectation E kb
q − ~qk2 as:
−22
Pr(kb
q − ~qk2 ≥ E kb
q − ~qk2 + ) ≤ exp PN
2
i=1 ci
= exp
−22
N · 2/N 2
= e−N 21
2
(7.7)
We can bound the expected value using the following inequality:

2 1/2
N
N
X
X
~qi − N ~q 
E
~qi − N ~q = E 
i=1
i=1
2
2
 2 1/2
N
X
≤ E ~qi − N ~q 
i=1
=
N
X
(by concavity of square root, and Jensens inequality)
2
!1/2
2
E k~qi − ~qk2
i=1
=
N
X
!1/2
E(~qi − ~q)T (~qi − ~q)
i=1
Multiplying out and using linearity of expectation and properties of ~qi (namely, that ~qiT ~qi = 1, E(~qi ) = ~q and ~q is constant),
we get:
N
!1/2
N
X
X
2
T
E(1 − 2~qi ~q + k~qk2 )
(since ~qiT ~qi = 1)
~qi − N ~q ≤
E
i=1
i=1
2
!1/2
N
N
N
X
X
X
2
=
E(1) − 2
E(~qiT ~q) +
E k~qk2
i=1
i=1
2
k~qk2
= N − 2N
+N
q
2
= N (1 − k~qk2 )
i=1
2
k~qk2
1/2
This implies an upper bound on the expected value:
E kb
q−
2
~qk2
2
N
X
~qi − N ~q
= (1/N )E 2
i=1
2
2
2
≤ (1/N ) · N (1 − k~qk2 )
√ q
2
⇒ E kb
q − ~qk2 ≤ (1/ N ) (1 − k~qk2 )
√
≤ (1/ N )
Using this upper bound in McDiarmids inequality (equation (7.7)), we get a looser version of the bound that proves the
proposition:
√
2
Pr(kb
q − ~qk2 ≥ 1/ N + ) ≤ e−N We are now ready to prove Lemma 18.
P ROOF :[Lemma 18] We will treat Pb1 ,Pb2,1 and Pb3,x,1 as vectors, and use McDiarmid’s inequality to bound the error in
estimating a distribution over a simplex based on indicator vector samples, using Proposition 20. We know that
p
2
Pr kb
q − ~qk2 ≥ 1/N + ≤ e−N .
2
Now let η = e−N . This implies
ln η = −N 2
ln(1/η) = N 2
p
= ln(1/η)/N
22
Hence,
p
p
Pr kb
q − ~qk2 ≥ 1/N + ln(1/η)/N ≤ η
Therefore, with probability at least 1 − η,
p
√
kb
q − ~qk2 ≤ 1/ N + ln(1/η)/N
(7.8)
Now, in place of ~q in equation (7.8), we substitute the stochastic vector P~1 to prove the first claim, the vectorized version of
the stochastic matrix P2,1 to prove the second claim, and the vectorized version of the stochastic tensor P3,2,1 ∈ Rn×n×n
obtained by stacking P3,x,1 matrices over all x, to prove the third claim. The matrices Pb3,x,1 are stacked accordingly to
obtain the estimated tensor Pb3,2,1 . We get the following:
p
√
1 ≤ 1/ N + ln(1/η)/N (hence proving the first claim)
p
√
2,1 ≤ 1/ N + ln(1/η)/N (hence proving the second claim)
s
X
max 3,x,1 ≤
3,x,1 2
x
x
s
2
X
=
P3,x,1 − Pb3,x,1 2
x
=
sX X X
x
i
([P3,x,1 ]i,j − [Pb3,x,1 ]i,j )2
j
r
2
= P3,2,1 − Pb3,2,1 2
b
= P3,2,1 − P3,2,1 2
p
p
≤ 1/N + ln(1/η)/N
(hence proving the third claim)
Note the following useful inequality from the above proof:
s
X
p
p
3,x,1 2 ≤ 1/N + ln(1/η)/N
(7.9)
x
P
It remains to prove the fourth claim, regarding x 3,x,1 . First we get a bound that depends on n as follows:
X
X
3,x,1 =
|3,x,1 | (∵ ∀x, 3,x,1 ≥ 0)
x
x
≤
√
s
n
X
(∵ ∀~x ∈ Ra , k~xk1 ≤
3,x,1 2
√
a k~xk2 )
x
≤
p
r
n/N +
n
1
ln
N
η
We aren’t going to use the above bound. Instead, if n is large and N small, this bound can be improved by removing direct
dependence on n. Let (k) be the sum of smallest n − k probabilities of the second observation x2 . Let Sk be the set of
these n − k such observations x, for any k. Therefore,
X
X X
(k) =
Pr[x2 = x] =
[P3,x,1 ]ij
x∈Sk i,j
x∈Sk
Now, first note that we can bound
X
P
x∈S
/ k 3,x,1
3,x,1 ≤
x∈S
/ k
X
as follows:
|3,x,1 |
x∈S
/ k
≤
√ sX
k
3,x,1 2
x∈S
/ k
23
(∵ ∀~x ∈ Ra , k~xk1 ≤
√
a k~xk2 )
By combining with equation 7.9, we get
X
3,x,1 ≤
p
p
k/N + k ln(1/η)/N
(7.10)
x∈S
/ k
P
To bound x∈Sk 3,x,1 , we first apply equation (7.8) again. Consider the vector ~q of length kn2 + 1 whose first kn2 entries
comprise the elements of P3,x,1 for all x ∈
/ Sk , and whose last entry is the cumulative sum of elements of P3,x,1 for all
x ∈ Sk . Define qb accordingly with Pb3,x,1 instead of P3,x,1 . Now equation (7.8) directly gives us with probability at least
1 − η:

2  21
X
X
X
X
X
X
p
p


2
b
b
([P3,x,1 ]i,j − [P3,x,1 ]i,j ) + ([P3,x,1 ]ij −
[P3,x,1 ]ij )  ≤ 1/N + ln(1/η)/N

x∈Sk i,j
x∈Sk i,j
x∈S
/ k i,j
2
2 X X
p
2
X p
b
b
([P3,x,1 ]ij − [P3,x,1 ]ij ) ≤
1/N + ln(1/η)/N
P3,x,1 − P3,x,1 + F
x∈Sk i,j
x∈S
/ k
Since the first term above is positive, we get
p
X X
p
b3,x,1 ]ij − [P3,x,1 ]ij ) ≤ 1/N + ln(1/η)/N
([
P
x∈Sk i,j
Now, by definition of Sk :,
X
X
3,x,1 =
x∈Sk
b
P3,x,1 − P3,x,1 F
x∈Sk
X X ≤
[Pb3,x,1 ]ij − [P3,x,1 ]ij (∵ ∀~x, k~xk2 ≤ k~xk1 )
x∈Sk i,j
=
X X
max 0, [Pb3,x,1 ]ij − [P3,x,1 ]ij
x∈Sk i,j
−
X X
min 0, [Pb3,x,1 ]ij − [P3,x,1 ]ij
(∵ ∀~x, |~x| = [max(0, ~x) − min(0, ~x)])
x∈Sk i,j
≤
X X
X X
max 0, [Pb3,x,1 ]ij − [P3,x,1 ]ij +
[P3,x,1 ]ij
x∈Sk i,j
+
x∈Sk i,j
X X
min 0, [Pb3,x,1 ]ij − [P3,x,1 ]ij +
x∈Sk i,j
=
X X
X X
[P3,x,1 ]ij
x∈Sk i,j
max 0, [Pb3,x,1 ]ij − [P3,x,1 ]ij + (k)
x∈Sk i,j
+
X X
min 0, [Pb3,x,1 ]ij − [P3,x,1 ]ij + (k) (by definition of (k))
x∈Sk i,j
X X
≤ [Pb3,x,1 ]ij − [P3,x,1 ]ij + 2(k)
x∈Sk i,j
P
Plugging in equation (7.11), we get a bound on x∈Sk 3,x,1 :
X
p
p
3,x,1 ≤ 1/N + ln(1/η)/N + 2(k)
x∈Sk
Combining with equation (7.10) and noting that k is arbitrary, we get the desired bound:
X
p
p
p
p
3,x,1 ≤ min[ k ln(1/η)/N + k/N + ln(1/η)/N + 1/N + 2(k)]
x
k
24
(7.11)
Note that, to get the term ln(3/η) instead of ln(1/η) as in the fourth claim, we simply use η/3 instead of η. This bound on
P
x 3,x,1 will be small if the number of frequently occurring observations is small, even if n itself is large.
b , and
The next lemma uses the perturbation bound in Corollary 17 to bound the effect of sampling error on the estimate U
T
b OR).
on the conditioning of (U
Lemma 21 [Modification of HKZ Lemma 9] Suppose 2,1 ≤ ε · σk (P2,1 ) for some ε < 1/2. Let ε0 = 22,1 /((1 −
b ∈ Rm×k as the matrices of the first k left singular vectors of P2,1 , Pb2,1 respectively. Let
ε)σk (P2,1 ))2 . Define U, U
b ). Then:
θ1 , . . . , θk be the canonical angles between span(U ) and span(U
1. ε0 < 1
b T Pb2,1 ) ≥ (1 − ε)σk (P2,1 )
2. σk (U
b T P2,1 ) ≥ √1 − ε0 σk (P2,1 )
3. σk (U
b T OR) ≥ √1 − ε0 σk (OR)
4. σk (U
b⊥ to be the remaining n − k left singular vectors of Pb2,1
P ROOF : First some additional definitions and notation. Define U
corresponding to the lower n − k singular values, and correspondingly U⊥ for P2,1 . Suppose U ΣV T = P2,1 is the thin
SVD of P2,1 . Finally, we use the notation ~νi {A} ∈ Rq to denote the ith right singular vector of a matrix A ∈ Rp×q . Recall
that σi (A) = kA~νi {A}k2 by definition.
First claim: ε0 < 1 follows from the assumptions:
ε0 =
2,1 2
((1 − ε)σk (P2,1 ))2
2
≤
ε2 σk (P2,1 )
2
(1 − ε)2 σk (P2,1 )
ε2
(1 − ε)2
< 1 (since ε < 1/2)
=
b T Pb2,1 ) =
Second claim: By Corollary 17, σk (Pb2,1 ) ≥ (1 − ε)σk (P2,1 ). The second claim follows from noting that σk (U
b
σk (P2,1 ).
b T U . For any vector x ∈ Rk :
Third and fourth claims: First consider the k th singular value of U
bT bT U U x
U U y 2
2
≥ min
y
kxk2
kyk2
b T U ) (by definition of smallest singular value)
= σk (U
= cos(θk ) (by Definition 14)
q
= 1 − sin2 (θk )
q
b T U )2 (by Lemma 15)
= 1 − σk ( U
⊥
q
b T U )2
≥ 1 − σ1 (U
⊥
r
b T 2
(by definition of L2 matrix norm)
= 1 − U
⊥U 2
Therefore,
r
bT b T 2
U
U U x ≥ kxk2 1 − U
⊥ 2
2
25
(7.12)
Note that
b T 2
2
U⊥ U ≤ 2,1 2 /σk (P2,1 )
2
(by Corollary 17)
2,1 2
≤
(since 0 ≤ ε < 1/2)
2
(1 − ε)2 σk (P2,1 )
= ε0 (by definition)
Hence, by combining the above with equation (7.12), since 0 ≤ ε0 < 1:
√
bT U U x ≥ kxk2 1 − ε0 (for all x ∈ Rk )
2
(7.13)
The remaining claims follow by taking different choices of x in equation (7.13), and by using the intuition that the smallest
singular value of a matrix is the smallest possible L2 norm of a unit-length vector after the matrix has left-multiplied
that vector, and the particular vector for which this holds is the corresponding right singular vector. For claim 3, let
b T P2,1 }. Then by equation (7.13):
x = ΣV T~νk {U
√
bT
b T P2,1 }
b T P2,1 }
1 − ε0
U U ΣV T~νk {U
≥ ΣV T~νk {U
2
2
Since P2,1
b T P2,1 }
= U ΣV T , and ΣV T~νk {ΣV T }2 ≤ ΣV T~νk {U
by definition of ~νk {ΣV T }, we have:
2
√
bT
b T P2,1 }
U P2,1~νk {U
≥ ΣV T~νk {ΣV T }2 1 − ε0
2
√
T
b
b T P2,1 ), σk (ΣV T ))
σk (U P2,1 ) ≥ σk (ΣV T ) 1 − ε0 (by definition of σk (U
√
b T P2,1 ) ≥ σk (P2,1 ) 1 − ε0 (∵ σk (ΣV T ) = σk (P2,1 ))
σk ( U
which proves claim 3.
For claim 4, first recall that OR can be exactly expressed as P2,1 (S diag(~π )OT )+ (equation (7.2)). For brevity, let A =
b T OR} in equation (7.13):
(S diag(~π )OT )+ , so that OR = P2,1 A. Then, let x = ΣV T A~νk {U
√
bT
b T OR}
b T OR}
≥ ΣV T A~νk {U
1 − ε0
U U ΣV T A~νk {U
2
Since P2,1
2
b T OR}
= U ΣV T , and ΣV T A~νk {ΣV T A}2 ≤ ΣV T A~νk {U
by definition of ~νk {ΣV T A}, we get:
2
√
bT
b T OR}
U P2,1 A~νk {U
≥ ΣV T A~νk {ΣV T A}2 1 − ε0
2
√
bT
T
b
U OR~νk {U OR} ≥ σk (ΣV T A) 1 − ε0 (by equation (7.2))
2
b T OR), σk (ΣV T A), we see that
By definition of σk (U
√
b T OR) ≥ σk (ΣV T A) 1 − ε0
σk (U
√
b T OR) ≥ σk (OR) 1 − ε0 (∵ σk (ΣV T A) = σk (P2,1 A) = σk (OR))
σk (U
hence proving claim 4.
b , which constitutes a true observable representation for the
Define the following observable representation using U = U
HMM as long as (U T OR) is invertible:
eb∞ = (P2,1 T U
b )+ P~1 = (U
b T OR)−T RT~1m
ex = (U
b T P3,x,1 )(U
b T P2,1 )+ = (U
b T OR)Wx (U
b T OR)−1
B
eb1 = U
b T P~1
26
for x = 1, . . . , n
Define the following error measures of estimated parameters with respect to the true observable representation. The error
vector in δ1 is projected to Rm before applying the vector norm, for convenience in later theorems.
bT T b
O) (b∞ − eb∞ )
δ∞ = (U
∞
bT
bT
bx (U
b T OR) − Wx bx − B
ex (U
b T OR)
OR)−1 B
OR)−1 B
∆x = (U
= (U
1
1
X
∆=
∆x
x
bT
bT
OR)−1bb1 − ~π OR)−1 (bb1 − eb1 ) = R(U
δ1 = R(U
1
1
bx , bb1 are close to the true parameters eb∞ , B
ex , eb1 if the sampling
The next Lemma proves that the estimated parameters bb∞ , B
errors 1 , 2,1 , 3,x,1 are small:
Lemma 22 [Modification of HKZ Lemma 10] Assume 2,1 < σk (P2,1 )/3. Then:
δ∞ ≤ 4 ·
8
∆x ≤ √
3
8
∆≤ √
3
2
δ1 ≤ √
3
!
1
2 + 3σ (P
k
2,1 )
σk (P2,1 )
!
√
2,1
Σx 3,x,1
k
·
· Pr[x2 = x] ·
2 + 3σ (P
σk (OR)
k
2,1 )
σk (P2,1 )
!
√
k
2,1
Σx 3,x,1
·
·
2 + 3σ (P
σk (OR)
k
2,1 )
σk (P2,1 )
√
k
· 1
·
σk (OR)
2,1
b T OR) to be invertible by Lemma 21, claim 4.
P ROOF : Note that the assumption on 2,1 guarantees (U
δ∞ bound:
We first see that δ∞ can be bounded by bb∞ − eb∞ :
2
δ∞
= (OT U )(bb∞ − eb∞ )
∞
T b
e
≤ O ∞ U (b∞ − b∞ )
∞
b
e
≤ U (b∞ − b∞ )
∞
b
≤ U (b∞ − eb∞ )
2
b
e
≤ b∞ − b∞ 2
In turn, this leads to the following expression:
b
T b +b
b )+ P~1 U ) P1 − (P2,1 T U
b∞ − eb∞ = (Pb2,1
2
2
T b +b
b )+ Pb1 + (P2,1 T U
b )+ Pb1 − (P2,1 T U
b )+ P~1 = (Pb2,1
U ) P1 − (P2,1 T U
2
bT b +
Tb + b
Tb + b
~
= (P2,1 U ) − (P2,1 U ) P1 + (P2,1 U ) (P1 − P1 )
2
bT b +
Tb + b Tb + b
~
≤ ((P2,1 U ) − (P2,1 U ) )P1 + (P2,1 U ) (P1 − P1 )
2
2
bT b +
Tb + b T b + b
≤ ((P2,1 U ) − (P2,1 U ) ) P1 + (P2,1 U ) (P1 − P~1 )
2
27
1
2
2
The last step above obtains from the consistency of the L2 matrix norm with L1 vector norm, and from the definition of L2
kAxk
b has orthonormal columns, and hence multiplying a
matrix norm (spectral norm) as kAk2 = max kxk 2 . Now, recall that U
2
b cannot increase its spectral norm. Hence,
matrix with U
T
T
bT b
b
b
− P2,1 T )U
− P2,1 T = 2,1 .
≤ Pb2,1
P2,1 U − P2,1 T U
= (Pb2,1
2
2
2
T b
b using 2,1 as an upper
So, we can use Lemma 12 to bound the L2 -distance between pseudoinverses of Pb2,1
U and P2,1 T U
bound on the difference between the matrices themselves. Also recall that singular values of the pseudoinverse of a matrix are the reciprocals of
values.
Substituting this in the above expression, along with the facts that
the
matrix singular
b
b T b
~
b
b
σk (P2,1 U ) = σk (P2,1 ), P1 = 1 and (P1 − P1 ) = 1 , gives us:
2
1
√
1+ 5
b
·
b∞ − eb∞ ≤
2
2
2,1
1
2 +
b)
σk (P2,1 T U
b)
min σk (Pb2,1 ), σk (P2,1 T U
Now, to simplify the last expression further, consider Lemma 21 in the above context. Here, 2,1 ≤ σk (P2,1 )/3 and hence
b T P2,1 ) ≥ √1 − ε0 σk (P2,1 ). Hence
b T Pb2,1 ) ≥ (2/3)σk (P2,1 ) and σk (U
ε = 1/3. Therefore σk (Pb2,1 ) = σk (U
2
√
b ) = σk (P2,1 )2 · min(2/3, 1 − ε0 )2
min σk (Pb2,1 ), σk (P2,1 T U
The latter term is larger since
ε0 =
2,1 2
((1 − ε)σk (P2,1 ))2
2
≤
σk (P2,1 ) /9
2
4σk (P2,1 ) /9
= 1/4
√
√
⇒ 1 − ε0 ≥ 3/2 > 2/3
2
b ) ≥ σk (P2,1 )2 (2/3)2 . Plugging this into the expression above along with the fact
Therefore min σk (Pb2,1 ), σk (P2,1 T U
√
b T P2,1 ) ≥ ( 3/2)σk (P2,1 ), we prove the required result for δ∞ :
that σk (U
δ∞
∆x ,∆ bounds:
√
1+ 5
21
92,1
≤
+√
·
2
2
3σk (P2,1 )
4σk (P2,1 )
!
2,1
1
≤4·
2 + σ (P
k
2,1 )
σk (P2,1 )
. We first bound each term ∆x by
√ b
e bT
k B
x − Bx /σk ((U OR)):
2
bT
bx − B
ex (U
b T OR)
∆x = (U
OR)−1 B
1
bT
−1 b
T
e
b
≤ (U OR) (Bx − Bx )U kORk1 (by norm consistency)
1
√ bT
bx − B
ex )U
b T
≤ k (U
OR)−1 (B
kORk1 (by L1 vs. L2 norm inequality)
2
√ T
b T
−1 b
b
e
≤ k (U OR) Bx − Bx U kOk1 kRk1 (by norm consistency)
2 2 2 √ bT
b T
−1 b
e
≤ k (U OR) Bx − Bx U , kOk1 , kRk1 ≤ 1
2
2
2
√ b
T
ex /σk (U
b OR) (∵ σmax (U
b T OR)−1 = 1/σmin (U
b T OR))
= k Bx − B
2
28
b
e The term B
x − Bx in the numerator can be bounded by
2
bT
b
+
bT
bT b
b T b +
e
B
−
B
x
x = (U P3,x,1 )(U P2,1 ) − (U P3,x,1 )(U P2,1 ) 2
2
bT
bT
T
+
Tb
+ b T P2,1 )+ b
b
P3,x,1 − Pb3,x,1 (U
≤ (U P3,x,1 ) (U P2,1 ) − (U P2,1 ) + U
2
2
√
1+ 5
3,x,1
2,1
≤ kP3,x,1 k2 ·
·
2 +
b T P2,1 )
2
σk ( U
b T P2,1 )
min σk (Pb2,1 ), σk (U
√
1+ 5
2,1
3,x,1
≤ Pr[x2 = x] ·
·
2 +
b T P2,1 )
2
σk (U
b T P2,1 )
min σk (Pb2,1 ), σk (U
where the second inequality is from Lemma 12 and the last one uses the fact that
sX
X
[P3,x,1 ]i,j = Pr[x2 = x].
kP3,x,1 k2 ≤ kP3,x,1 kF =
[P3,x,1 ]2i,j ≤
i,j
i,j
Applying Lemma 21 as in the δ∞ bound above, gives us the required result on ∆x . Summing both sides over x results in
the required bound on ∆.
δ1 bound: For δ1 , we invoke Condition 4 to use the fact that kRk1 ≤ 1. Specifically,
bT
b T (Pb1 − P~1 )
δ1 = R(U
OR)−1 U
1
bT
−1 b T b
~
(norm consistency)
≤ kRk1 (U OR) U (P1 − P1 )
1
√
√
bT
b T (Pb1 − P~1 )
≤ k kRk1 (U
OR)−1 U
(kxk1 ≤ n kxk2 for any x ∈ Rn )
2
√
bT
−1 b T b
~
≤ k kRk1 (U OR) U (P1 − P1 )
(norm consistency)
2
2
√
bT
b T has orthogonal columns)
OR)−1 · 1 (defn. of 1 , U
≤ k kRk1 (U
2
√
k1
(kRk1 ≤ 1, defn. of L2 -norm)
=
b T OR)
σk (U
The desired bound
√ on δ1 is obtained by using Lemma 21. With ε, ε0 as described in the above proof for δ∞ , we have that
b T OR) ≥ ( 3/2)σk (U T OR). The required bound follows by plugging this inequality into the above upper bound for
σk (U
δ1 .
7.4
Proof of Theorem 2
The following Lemmas 23 and 24 together with Lemmas 18,21,22 above, constitute the proof of Theorem 2 on joint
probability accuracy. We state the results based on appropriate modifications of HKZ, and provide complete proofs. We
also describe how the proofs generalize to the case of handling continuous observations using Kernel Density Estimation
(KDE). First, define the following as in HKZ


X

(i) = min
Pr[x2 = j] : S ⊆ {1 . . . n}, |S| = n − i


j∈S
and let
n0 (ε) = min{i : (i) ≤ ε}
The term n0 (ε), which occurs in the theorem statement, can be interpreted as the minimum number of discrete observations
that accounts for 1 − of total marginal observation probability mass. Since this can be much lower than (and independent
of) n in many applications, the analysis of HKZ is able to use n0 instead of n in the sample complexity bound. This is
useful in domains with large n, and our relaxation of HKZ preserves this advantageous property.
The following lemma quantifies how estimation errors accumulate while computing the joint probability of a length t
bx and bb.
sequence, due to errors in B
29
b T OR is invertible. For any time t:
Lemma 23 [Modification of HKZ Lemma 11] Assume U
X
bx bb1 − B
ex eb1 ≤ (1 + ∆)t δ1 + (1 + ∆)t − 1
R(U T OR)−1 B
t:1
t:1
1
x1:t
P ROOF : Proof by induction. The base case for t = 0, i.e. that R(U T OR)−1 (bb1 − eb1 ) ≤ δ1 is true by definition of
1
bx −1:1bb1 and ebt = ebt (x1:t−1 ) = B
ex :1eb1 . For some
δ1 . For the rest, define unnormalized states bbt = bbt (x1:t−1 ) = B
t
t−1
particular t > 1, assume the inductive hypothesis as follows
X
bT
R(U OR)−1 bbt − ebt ≤ (1 + ∆)t δ1 + (1 + ∆)t − 1
1
x1:t
The sum over x1:t in the LHS can be decomposed as:
X
bT
R(U OR)−1 bbt − ebt 1
x1:t
=
X X bx − B
ex )ebt + (B
bx − B
ex )(bbt − ebt ) + B
ex (bbt − ebt ) R(U T OR)−1 (B
t
t
t
t
t
1
x x1:t−1
Using triangle inequality, the above sum is bounded by
X X bT
bT
bx − B
ex (U
b T O)
OR)−1ebt R(U OR)−1 B
R(U
t
t
1
xt x1:t−1
1
X X bT
bT
−1 b
ebt bx − B
ex (U
b T O)
+
U
OR)
b
−
R(U OR)−1 B
R(
t
t
t
1
xt x1:t−1
1
X X bT
et (U
b T OR)(U
b T OR)−1 bbt − ebt +
R(U OR)−1 B
1
xt x1:t−1
bT
Each of the above double sums is bounded separately. For the first, we note that R(U
OR)−1ebt = Pr[x1:t−1 ], which
1
sums to 1 over x1:t−1 . The remainder of
the double sum is bounded by ∆, by definition. For the second double sum, the
bT
inner sum over R(U
OR)−1 (bbt − ebt ) is bounded using the inductive hypothesis. The outer sum scales this bound by
1
∆, by definition. Hence the second double sum is bounded by ∆((1 + ∆)t−1 δ1 + (1 + ∆)t−1 − 1). Finally, we deal with
et (U
b T OR) by Wx , and note that R · Wx = Ax R. Since
b T OR)−1 B
the third double sum as follows. We first replace (U
t
t
t
T
~
Axt is entry-wise nonnegative by definition, kAxt ~v k1 ≤ 1m Axt |~v |, where |~v | denotes element-wise absolute value. Also
P
b T OR)−1 (bbt − ebt ) in the third double
v | = k~v k1 . Using this result with ~v = R(U
v | = ~1T
note that ~1T
v | = ~1T
m |~
m
m T |~
xt Axt |~
t−1
sum above, the inductive hypothesis bounds the double sum by (1 + ∆) δ1 + (1 + ∆)t−1 − 1. Combining these three
bounds gives us the required result:
X
bT
R(U OR)−1 bbt − ebt 1
x1:t
≤ ∆ + ∆((1 + ∆)t−1 δ1 + (1 + ∆)t−1 − 1) + (1 + ∆)t−1 δ1 + (1 + ∆)t−1 − 1
= ∆ + (1 + ∆)((1 + ∆)t−1 δ1 + (1 + ∆)t−1 − 1)
= ∆ + (1 + ∆)t δ1 + (1 + ∆)t − 1 − ∆
= (1 + ∆)t δ1 + (1 + ∆)t − 1
thus completing the induction.
The following lemma bounds the effect of errors in the normalizer bb∞ .
Lemma 24 [Modification of HKZ Lemma 12] Assume 2,1 ≤ σk (P2,1 )/3. Then for any t,
X c 1:t ] ≤ (1 + δ∞ )(1 + δ1 )(1 + ∆)t − 1
Pr[x1:t ] − Pr[x
x1:t
30
b T OR) > 0 and so (U
b T OR) is
P ROOF : First note that the upper bound on 2,1 along with Lemma 21, ensure that σk (U
invertible. The LHS above can be decomposed into three sums that are dealt with separately:
X
X bT b b ~ T
c 1:t ] =
b∞ Bxt:1 b1 − b∞ Bxt:1~b1 Pr[x1:t ] − Pr[x
x1:t
x1:t
X bb1 − ebT B
eb1 b
e
=
B
bbT
x
x
∞
∞
t:1
t:1
x1:t
X b T OR)(U
b T OR)−1 B
ex eb1 ≤
(bb∞ − eb∞ )T (U
t:1
x1:t
+
X b T OR)(U
b T OR)−1 (B
bx bb1 − B
ex eb1 )
(bb∞ − eb∞ )T (U
t:1
t:1
x
1:t
X −1 b
bT
bT
ex eb1 )
(Bxt:1 bb1 − B
+
ebT
∞ (U OR)(U OR)
t:1
x1:t
The first sum can be bounded as follows, using Hölders inequality and bounds from Lemma 22:
X
X bT T b
bT
b T OR)(U
b T OR)−1 B
ex eb1 ≤
ex eb1 OR)−1 B
(bb∞ − eb∞ )T (U
(U O) (b∞ − eb∞ ) R(U
t:1
t:1
x1:t
∞
x1:t
≤
X
1
δ∞ kAxt:1 ~π k1
x1:t
=
X
δ∞ Pr[x1:t ]
x1:t
= δ∞
The second sum can be bounded also using Hölders, as well as the bound in Lemma 23:
X b T OR)(U
b T OR)−1 (B
bx bb1 − B
ex eb1 )
(bb∞ − eb∞ )T (U
t:1
t:1
x1:t
bT T b
≤ (U
O) (b∞ − eb∞ )
∞
bT
bx bb1 − B
ex eb1 )
R(U OR)−1 (B
t:1
t:1
1
≤ δ∞ ((1 + ∆)t δ1 + (1 + ∆)t − 1)
The third sum again uses Lemma 23:
X
X ~ T b T
T
T
−1 b
−1 b
b
e
b
e
b
b
e
e
(
U
OR)(
U
OR)
(
B
b
−
B
1
R(
U
OR)
(
B
b
)
=
b
−
B
b
)
ebT
xt:1 1
xt:1 1
xt:1 1
xt:1 1 ∞
x1:t
x1:t
bT
bx bb1 − B
ex eb1 )
≤ R(U
OR)−1 (B
t:1
t:1
1
≤ (1 + ∆)t δ1 + (1 + ∆)t − 1
Adding these three sums gives us:
X c 1:t ] ≤ δ∞ + δ∞ ((1 + ∆)t δ1 + (1 + ∆)t − 1) + (1 + ∆)t δ1 + (1 + ∆)t − 1
Pr[x1:t ] − Pr[x
x1:t
≤ δ∞ + (1 + δ∞ )((1 + ∆)t δ1 + (1 + ∆)t − 1)
which is the required bound.
P ROOF :(Theorem 2). Assume N and ε as in the theorem statement:
√
ε = σk (OR)σk (P2,1 )/(4t k)
t2
k
k · n0 (ε)
N ≥C· 2 ·
+
· log(1/η)
σk (OR)2 σk (P2,1 )4
σk (OR)2 σk (P2,1 )2
31
First note that
X c 1:t ] ≤ 2
Pr[x1:t ] − Pr[x
x1:t
since it is the L1 difference between two stochastic vectors. Therefore, the theorem is vacuous for ≥ 2. Hence we can
assume
<1
in the proof and let the constant C absorb the factor 4 difference due to the 1/2 term in the expression for N .
The proof has three steps. We first list these steps then prove them below.
First step: for a suitable constant C, the following sampling error bounds follow from Lemma 18:
√
√
1 ≤ min .05 · (3/8) · σk (P2,1 ) · , .05 · ( 3/2) · σk (OR) · (1/ k) · √
√
2
2,1 ≤ min .05 · (1/8) · σk (P2,1 ) · (/5), .01 · ( 3/8) · σk (OR) · σk (P2,1 )2 · (1/(t k)) · X
√
√
3,x,1 ≤ 0.39 · (3 3/8) · σk (OR) · σk (P2,1 ) · (1/(t k)) · (7.14a)
(7.14b)
(7.14c)
x
Second step: Lemma 22 together with equations (7.14) imply:
δ∞ ≤ .05
δ1 ≤ .05
∆ ≤ 0.4/t
(7.15a)
(7.15b)
(7.15c)
Third step: By Lemma 24, equations (7.15) and the inequality
(1 + (a/t))t ≤ 1 + 2a
for a ≤ 1/2
we get the theorem statement.
Proof of first step: Note that for any value of matrix P2,1 , we can upper-bound σk (P2,1 ) by 1:
σk (P2,1 ) ≤ σ1 (P2,1 )
= max kP2,1 xk2
kxk2 =1

!2 1/2
n
n
X
X
= max 
[P2,1 ]ij xi 
kxk2 =1
≤ max
kxk2 =1
≤
=
j=1
i=1
n X
n
X
j=1 i=1
n X
n
X
j=1 i=1
n X
n
X
[P2,1 ]ij xi |[P2,1 ]ij |
[P2,1 ]ij
(by norm inequality)
(|xi | ≤ 1 since kxk2 = 1)
(by non-negativity of P2,1 )
j=1 i=1
= 1 (by definition)
32
(7.16)
Similarly, for any column-stochastic observation probability matrix O we can bound σk (OR) by
√
m:
√
k. First see that σ1 (O) ≤
σ1 (O) = max kOxk2
kxk2 =1

1/2
m X
n
X
= max 
(Oij xi )2 
kxk2 =1
j=1 i=1

1/2
m X
n
X
2
≤ max 
Oij
kxk2 =1
(kxk2 = 1 ⇒ |xi | ≤ 1)
j=1 i=1
1/2
m X
n
X
≤ (
Oij )2 

(by triangle inequality)
j=1 i=1

≤
=
√
m
X
1/2
12 
(by definition of O)
j=1
m
Now the bound on σk (OR) follows from Condition 5 i.e. σk (OR) ≤
p
k/m:
σk (OR) = min kORxk2
kxk2 =1
≤ kOk2 · min kRxk2 (by norm consistency)
kxk2 =1
v
um k
uX X
√
(Rij xj )2 (∵ kAk2 = σ1 (A) for any matrix A)
≤ m min t
kxk2 =1
i=1 j=1
Assume the cth column of R obeys Condition 5 for some 1 ≤ c ≤ k. Also assume x = ec , the cth column of the k × k
identity matrix, which obeys the constraint kxk2 = 1. Then every component of the inner sum is zero except when j = c,
and the min expression can only get larger:
v
um
X
√ u
2
σk (OR) ≤ mt
Ric
√
i=1
m kR[·, c]k2
√ p
≤ m k/m
√
= k
=
hence proving that σk (OR) ≤
√
k.
√
Now we begin the proof with the 1 case. Choose a C that satisfies all previous bounds and also obeys ( C/4)·0.05·(3/8) ≥
1.
p
p
1 ≤ 1/N ( ln(3/η) + 1) (by Lemma 18)
(7.17)
p
p
p
√
(7.18)
≤ 1/N (2 ln(3/η)) (since ln(3/η) ≥ ln 3 > 1)
Now, plugging in the assumed value of N :
2
2(σk (P2,1 ) σk (OR))
1 ≤ q
2
t Ck(1 + n0 (ε)σk (P2,1 ) )
33
s
ln(3/η)
ln(1/η)
(7.19)
Any substitutions that increasep
the right hand side of the above inequality preserve
the inequality. We nowp
drop the additive
√
1 in the denominator, replace ln(3/η)/ ln(1/η) by 2 since it is at most ln 3, and drop the factors t, n0 (ε) from the
denominator.
2
4σk (OR)σk (P2,1 ) √
(7.20)
Ckσk (P2,1 )
h
√ i
1
≤ √ · 4 · σk (OR)/ k · [σk (P2,1 )] · C
h
√
√ i
1
≤ √ min 4 · σk (P2,1 ) · , 4 · σk (OR) · 1/ k · (∵ both σk (OR)/ k and [σk (P2,1 )] are ≤ 1)
C
√
√
1
= 0 min 0.05 · 3/8 · σk (P2,1 ) · , 0.05 · 3/2 · σk (OR) · 1/ k · C
√
(for C 0 = 4C · 0.05 · 3/8)
√
√
= min 0.05 · 3/8 · σk (P2,1 ) · , 0.05 · 3/2 · σk (OR) · 1/ k · (∵ C 0 ≥ 1)
1 ≤
Hence proving the required bound for 1 .
√
√
Next we prove the 2,1 case. Choose a C that satisfies all previous bounds also obeys ( C/4) · 0.01 · ( 3/8) ≥ 1. Note
that, since the bound on 2,1 in Lemma 18 is the same as for 1 , we can start with the analogue of equation (7.19):
2
2(σk (P2,1 ) σk (OR))
≤ q
2
t Ck(1 + n0 (ε)σk (P2,1 ) )
2,1
2
s
ln(3/η)
ln(1/η)
We
√ now drop the additive n0 (ε)σk (P2,1 ) in the denominator, again replace
ln 3, and drop the multiplicative factor t from the denominator.
p
ln(3/η)/ ln(1/η) by 2 since it is at most
√
1
2
2,1 ≤ √ · 4 · σk (P2,1 ) · σk (OR) · (1/ k) · C
h
i h
√ i
1
2
= √ · 4 · σk (P2,1 ) · σk (OR)/ k · C
h
√
√ i
1
2
2
(∵ σk (OR)/ k ≤ 1)
≤ √ min 4 · σk (P2,1 ) · , 4 · σk (OR) · σk (P2,1 ) · 1/ k · C
√
√
1
2
2
≤ 0 min 0.05 · 1/8 · σk (P2,1 ) · , 0.01 · 3/8 · σk (OR) · σk (P2,1 ) · 1/ k · C
√
√
(for C 0 = ( C/4) · 0.01 · ( 3/8))
√
√
2
2
≤ min 0.05 · 1/8 · σk (P2,1 ) · , 0.01 · 3/8 · σk (OR) · σk (P2,1 ) · 1/ k · (since C 0 ≥ 1)
hence proving the bound on 2,1 .
P
Finally for x 3,x,1 , assume C such that
the bound from Lemma 18:
X
3,x,1 ≤ min
j
x
p
√
√
2·0.39·(3 3/8) C
√
16+ C
j/N
p
≥ 1 in addition to previous requirements on C. we first restate
p
p
ln 3/η + 1 + 2(j) + 1/N
ln 3/η + 1
p
p
p
n0 (ε)/N
ln 3/η + 1 + 2(n0 (ε)) + 1/N
ln 3/η + 1
p
p
≤ 1/N
ln 3/η + 1 (n0 (ε) + 1) + 2ε (since (n0 (ε)) ≤ ε)
≤
p
34
The
p first two terms are exactly as before, so we perform the same steps as in equations (7.17)-(7.20) except we do not drop
t n0 (ε), to get:
2
X
x
4σk (OR)σk (P2,1 ) (n0 (ε) + 1) + 2ε
3,x,1 ≤ p
Ckn0 (ε)σk (P2,1 )
√
4σk (OR)σk (P2,1 )
p
· (2 · n0 (ε)) + 2σk (OR)σk (P2,1 )/4t k
t Ckn0 (ε)
(since 1 + n0 (ε) ≤ 2 · n0 (ε), and plugging in ε)
√
√
≤ σk (OR) · σk (P2,1 ) · t k · · 8/ C + 1/2
√
√
√
√
1
3/8) C
√
≤ 0 0.39 · (3 3/8) · σk (OR) · σk (P2,1 ) · t k · (for C 0 = 2·0.39·(3
)
16+ C
C
√
√
≤ 0.39 · (3 3/8) · σk (OR) · σk (P2,1 ) · t k · (since C 0 > 1 by assumption)
≤
Hence proving the required bound for
P
x 3,x,1 .
Proof of second step: Substituting from equation (7.14) into δ1 in Lemma 22:
√
k
2
δ1 ≤ √
· 1
3 σk (OR)
!
√
√
2
3
1
k
3
≤√
min .05 · σk (P2,1 ), .05 ·
σk (OR) √ 8
2
3 σk (OR)
k
!
√
√
3
k
σk (P2,1 ), 1
= .05 · min
4 σk (OR)
≤ .05
Substituting from equation (7.14) into δ∞ in Lemma 22:
δ∞
!
1
≤4
2 + 3σ (P
k
2,1 )
σk (P2,1 )
√
√
4
2
2
≤
2 min .05 · (1/8) · σk (P2,1 ) · (/5), .01 · ( 3/8) · σk (OR) · σk (P2,1 ) · (1/(t k)) · σk (P2,1 )
√
√
4
min .05 · (3/8) · σk (P2,1 ) · , .05 · ( 3/2) · σk (OR) · (1/ k) · +
3σk (P2,1 )
√
√
≤ min .05, .04 · ( 3/8) · σk (OR) · (1/(t k)) · √
√
σk (OR)
+ min .05 · (1/2) · , .05 · (2/ 3) ·
· (1/ k) · σk (P2,1 )
≤ .05(.01 + .5)
≤ .05
2,1
35
Substituting from equation (7.14) into ∆ in Lemma 22:
!
√
8
k
2,1
Σx 3,x,1
∆≤ √ ·
·
2 + 3σ (P
3 σk (OR)
k
2,1 )
σk (P2,1 )
√
√
8 k
1
2
2
√
≤√
·
min
.05
·
(1/8)
·
σ
(P
)
·
(/5),
.01
·
(
3/8)
·
σ
(OR)
·
σ
(P
)
·
k
2,1
k
k
2,1
2
3σk (OR)
t k
σk (P2,1 )
!
√
√
1
+
0.39 · (3 3/8) · σk (OR) · σk (P2,1 ) · (1/(t k)) · 3σk (P2,1 )
!
√
k
= min .05 · (/5) √
, .01 ·
+ 0.39 ·
t
t
3σk (OR)
≤ .01 · + 0.39 ·
t
t
≤ 0.4/t
Proof of third step: By Lemma 24,
X c 1:t ] ≤ (1 + δ∞ )(1 + δ1 )(1 + ∆)t − 1
Pr[x1:t ] − Pr[x
x1:t
≤ (1 + .05)(1 + .05)(1 + 0.4/t)t − 1 (by equations (7.15))
≤ (1 + .05)(1 + .05)(1 + 0.8) − 1 (by equation (7.16), since 0.4 < 1/2)
= 1 + .05 + .05 + .052 + 0.8 + .042 + .042 + (.05)2 · .083 − 1
= .00023 + .08252 + 0.9
≤ (.0002 + .0825 + 0.9) (since < 1 by assumption)
= 0.9827
<
This completes the proof of Theorem 2.
7.5
Proof of Theorem 2 for Continuous Observations
For continuous observations, we use Kernel Density Estimation (KDE) [19] to model the observation probability density
function (PDF). We use a fraction of the training data points as kernel centers, placing one multivariate Gaussian kernel
at each point.1 The KDE estimator of the observation PDF is a convex combination of these kernels; since each kernel
integrates to 1, this estimator also integrates to 1. KDE theory [19] tells us that as the number of kernel centers and the
number of samples go to infinity and the kernel bandwidth goes to zero (at appropriate rates), the KDE estimator converges
to the observation PDF in L1 norm. The kernel density estimator is completely determined by the normalized vector of
kernel weights; therefore, if we can estimate this vector accurately, our estimate will converge to the observation PDF as
well.
Hence our goal is to predict the correct expected value of this normalized kernel vector given all past observations (or more
precisely, given the appropriate sequence of past observations, or the appropriate indicative events/features). In the context
of Theorem 2, joint probability estimates for t-length observation sequences are effectively the expectation of entries in a
t-dimensional tensor formed by the outer product of t indicator vectors. When we move to KDE, we instead estimate the
expected outer product of t stochastic vectors, namely, the normalized kernel weights at each time step. As long as the sum
of errors in estimating entries of this table goes to zero for any fixed t as the number of samples increases, our estimated
observation PDFs will have bounded error.
The only differences in the proof are as follows. In Lemma 18, we observe ~qi to be stochastic vectors instead of indicator
vectors; their expectation is still the true value of the quantity we are trying to predict. p~i are also stochastic vectors in that
proof. In the proof of Proposition 20, pk is an arbitrary stochastic vector. Also, ~qiT ~qi ≤ k~qi k1 = 1 now instead of being
always equal to 1, and the same holds for p~T
~i . Also kb
pi − p~i k2 ≤ kb
pi − p~i k1 = 1 (by triangle inequality). Besides these
ip
things, the above proof goes through as it is.
1
We use a general elliptical covariance matrix, chosen by SVD: that is, we use a spherical covariance after projecting onto the singular
vectors and scaling by the square roots of the singular values.
36
Note that in the continuous observation case, there are continuously many observable operators Wx that can be computed.
We compute one base operator for each kernel center, and use convex combinations of these base operators to compute
observable operators as needed.
8
Appendix II: An Example of Learning with Ambiguous Observations
When stacking observations, the modified, larger P2,1 ∈ Rn×n still has rank at most k since it can be written in the form
P2,1 = GT H for some matrices G, H T ∈ Rn×m . For example, if n = 2 for an HMM with ambiguous observations,
and we believe stacking 2 observations per timestep will yield a sufficiently informative observation, the new observation
2
2
space will consist of all n = n2 = 4 possible tuples of single observations and P2,1 ∈ Rn ×n , with each observation i
corresponding to a tuple < i1 , i2 > of the original observations. Specifically,
P2,1 (j, i) = Pr(x4 = j2 , x3 = j1 , x2 = i2 , x1 = i1 )
X
=
Pr(x4 = j2 , x3 = j1 , x2 = i2 , x1 = i1 , h4 = d, h3 = c, h2 = b, h1 = a)
a,b,c,d
=
X
Oj2 d Tdc Oj1 c Tcb Oi2 b Tba Oi1 a πa
a,b,c,d
=
X
T
Oj,c Tcb [ diag(π)O ]b,i
where Oj,c =
X
Oj2 d Tdc Oj1 c
d
b,c
⇒ P2,1 = OT diag(π)O
T
Similarly, we can show that P3,x,1 = GT H T for some matrices G, H T ∈ Rn×m . The exact formulae will differ for
different choices of past and future observable statistics.
9
Appendix III: Synthetic Example RR-HMM Parameters
Example 1
"
T =
0.3894 0.2371
0.2371 0.4985
0.3735 0.2644
0.3735
0.2644
0.3621
#
"
O=
0.6000 0.2000 0.2000
0.2000 0.6000 0.2000
0.2000 0.2000 0.6000
Example 2
"
T =
0.6736
0.0330
0.2935
0.0051
0.8203
0.1746
0.1639
0.2577
0.5784
#
O=
1 0
0 1
.5
.5
0
1
1
0
Example 3

0.7829 0.1036 0.0399 0.0736
 0.1036 0.4237 0.4262 0.0465 
T =
0.0399 0.4262 0.4380 0.0959 
0.0736 0.0465 0.0959 0.7840

37
O=
1
0
0
1
#
(9.1)