Completion (on tensors)
N ATA L I R U C H A N S K Y
Motivating Example: Netflix
What data does Netflix have?
•  Users rate movies they have seen
Data represented in matrix form:
•  Harry rates The Hobbit 5/5 but Gravity 2/5
3 4 1 1 5
Harry
2 5 1 1 2
3 3 4 3 1
5 4 5 2 1
Movies
Users
Motivating Example: Netflix
What does Netflix want to do?
•  Recommend you movie you would like – make you happy
with the service
How can it do this?
•  Knowing which movies Harry would rate 5/5
•  Predict ratings it cannot see
3 4 1 ? 5
Harry
? 5 ? 1 2
3 3 4 3 1
5 ? ? ? ?
Netflix Challenge
Users
Movies
Want to predict all ratings, but we know only 1% of
the entries!
Matrix Completion
On Input:
Partially-observed matrix M
The Goal is to:
Find a low-error completion of the remaining entries
Notation
From a ground truth fully-known an n × m matrix T, we
say only some of the values are observed.
Call the set of observed locations:
Ω = {(i, j) that are observed in T }
Define the partially observed matrix as:
#T (i, j) if (i, j) ∈ Ω
M (i, j) = $
%0 otherwise
Approaches to Matrix Completion
Need to make some assumption.
•  Currently the most popular is that T is low rank.
Find an estimate T by solving the optimization
problem:
minimize
subject to
rank(T )
M ij = Tij (i, j) ∈ Ω
This problem is NP-hard.
Approaches to Matrix Completion
Need to make some additional assumption.
1.  Low rank
2.  Known rank
A proposed strategy called OptSpace takes an
SVD-based approach:
minimize
subject to
T
n
*
M ij = Tij (i, j) ∈ Ω
T = ∑σ k (T )
*
k=1
Characteristics
Such optimization approaches
•  Output an estimate of the whole matrix T on any
input – any size Ω.
•  Give no indication of of confidence
•  Rely on randomly sampled Ω, i.e. the known entries
are randomly dispersed
Completion Questions
If we want very small error between T and T …
1.  How many entries does M need to have?
2.  When there are not enough entries, how can I
add them?
Lower Bound
Assuming that Ω is random, there does exist a bound
that say:
If there are at least r n log(n) entries then w.h.p the
completion will be accurate
This is the only way to answer the questions.
Simply sample randomly up to threshold, and hope
for the best…
Structural Methods
Recent methods take a more algebraic approach by
analyzing the structure of locations of the observed
entries.
Yes, there is a
unique
completion
Ω, n, m, r
Structural Matrix
Completion
No, there are
many possible
matrices
consistent with
the observations
Structural Methods Example
Consider the 2 x 2 matrix:
! 1 2 $
#
&
" 2 ? %
If I tell you that rank=1, can you fill the missing element?
Yes! The only possible value is 4. i.e.
What about if I tell you rank=2?
! 1 2 $
#
&
" 2 4 %
Structural Method
•  Explicitly state which entries can be recovered
•  Provide means to answer the questions other
methods could not
1.  How many entries does Ω need to have?
2.  When there are not enough entries, how can I add them?
Summary
Matrix Completion: Recover missing entries
•  Statistical approaches
•  Random Sampling
•  Optimization – always output a full estimate
•  Structural approaches
•  Indicate which entries can be recovered exactly
•  Only recover those entries
Tensor Completion. Let’s talk about tensors…
Tensor
Multi-dimensional matrix
1-dim
A(i)
2-dim
3-dim
A(i,j)
A(i,j,k)
Tensors: Example
Netflix Matrix
Users
Movies
Netflix Tensor
Tensors: Example
Netflix Matrix
Netflix Tensor
Movies
Users
Users
Genres
Tensor Applications
•  Traffic analysis (urban, internet)
•  Recommender systems
•  Biology
•  Vision/Images
•  Physics (immense application) e.g.
Image Source: http://arxiv.org/pdf/1306.2164v2.pdf
Tensors are hard
Tensors are hard
Example 1: Rank
•  Matrix Rank:
•  # independent row vectors
•  How many vectors are needed to represent the whole
matrix?
•  For m x n M, rank(M)≤max(m,n)
•  Tensor Rank
•  Definition 1:
Smallest r such that T is the sum of r rank-1 tensors B
•  Definition 2:
A tuple (r1, …, rn) where each ri is the rank of the unfolding
Tensor Unfloding
Example 2: SVD
•  Matrix SVD:
•  Decomposition into
•  Tensor SVD??
Tensor Completion
Given a partially observed tensor
Reconstruct the missing values
n-rank in this workminimize
and consider
the minimization
f (n-rank(X))
s. t.problem:
A (X) = b,
X∈T
Tensor completion
and low-n-rank
tensor recovery
convex
4
minimize
f! (n-rank(X))
s. t. via
A (X)
= b,optimization
"
X∈T via convex optimization
ompletion and low-n-rankwhere
tensor
recovery
f (n-rank(X))
= !f rank X(1) , rank X(2) , . . . , rank X(N") . 4
In order to keep
In
general
such a we
bestwill
approximation
but
introducing
conditions
things
simple,
use
the X
sum
of the
ranks
theXby
different
in the
where
f (n-rank(X))
=
f rank
rank
Xnot
,exist,
. . . , of
rank
. Inunfoldings
order
to keep
(1) ,might
(2)
(N
) #
N
on
the appearing
rank-1
terms,
uniqueness
can
be enforced
[7].
place
of
the function
fuse
, i.e.,
(n-rank(X))
= ||n-rank(X)||
rank (X(i)in). the
This
things
simple,
wenot
willexist,
thefbut
sumby
of the
ranks
of theconditions
different
unfoldings
1 =
l such a best approximation
might
introducing
#N i=1
tensor
rankisfis,also
difficult
to handle,
there aisweighting,
no 1straightforward
is This
one
choice,
but
possible
to incorporate
e.g.,
f (n-rank(X))
place
of the
function
i.e.,
f (n-rank(X))
=as||n-rank(X)||
= i=1
rank
(Xalgorithm
(i) ). This =
#N
pearing rank-1 terms, uniqueness
can
be
enforced
[7].
to
a specific
tensor; in fact,
the problem
[1, 8].
is determine
one
choice,rank
butCPis of
also
possiblegiven
to incorporate
a weighting,
e.g.,isfNP-hard
(n-rank(X))
=
i=1 γi rank X(i) . Thus, our minimization problem of interest becomes:
#
N
tensor rank is difficultThe
to
handle,
as other
there
noeasy
straightforward
algorithm
n-rank
onXthe
hand
is
to compute.
Therefore,
we only focus on the
ourisminimization
problem
of interest
becomes:
(i) . Thus,
i=1 γi ranksetting
Problem
3.1 (Low-n-rank
tensor
recovery).
in this
work and
consider
the minimization
problem:
mine rankCP of a specificn-rank
given
tensor;
in fact,
the
is NP-hard
[1, 8].
N problem
Problem
setting 3.1
(Low-n-rank
tensor
$
!
" recovery).Recall:
minimize
f$
(n-rank(X))
s. t.s. t.
A (X)
= b,on
minimize
rank
A (X)
= b the
(1)
N
(i)
nk on the other hand is easy to compute.
Therefore,
focus
! Xwe
" only
X∈T
X∈T
i=1
minimize !
rank X(i)
s. t. A (X) = b"
(1)
 ) to keep
this work and considerwhere
the fminimization
minimize
rank(
T
(n-rank(X))X∈T
= problem:
f rank
X
,
rank
X
,
.
.
.
,
rank
X
.
In
order
(1)
(2)
(N
)
i=1 constraints on the tensor X is governed by the linear
The exact form of the linear
things simple, we will use the sum of the ranks of the different
unfoldings
in the
 like
operator
A which
can
many
possible
forms
in general.
We
to
The exact
form
of
thetake
linear
constraints
on
thesubject
tensor
Xtois#
governed
by
the
linear
minimize f (n-rank(X))
s.
t.
A
(X)
=
b,
NMwould
=
T
(i,
j)point
∈Ω
ij
ij
place of the function f , i.e., f (n-rank(X)) = ||n-rank(X)||1 = i=1 rank (X(i) ). This
X∈T
out oneA
special
case.inIngeneral.
tensor completion,
subset
of the
operator
whichcase,
can the
taketensor
manycompletion
possible forms
We would alike
to point
one choice, but is also possible to incorporate
a weighting, e.g., f (n-rank(X)) =
! is#
"
entries
of the case,
tensor
Xtensor
is given,
and under
theIn
low-n-rank
assumption,
the unknown
the
completion
case.
tensor completion,
a subset
of the
N one special
(n-rank(X)) = f rankout
X
,
rank
X
,
.
.
.
,
rank
X
.
In
order
to
keep
γ
rank
X
.
Thus,
our
minimization
problem
of
interest
becomes:
i
(i)
(1)
(2)
(N
)
i=1
entriesofare
be deduced.
Denoting
the setthe
of revealed
entries
by Ω, thethe
corresponding
entries
thetotensor
X is given,
and under
low-n-rank
assumption,
unknown
mple, we will use the Problem
sum
of
the
ranks
of
the
different
unfoldings
in
the
operator
A be
retrieves
values of the
these
The
vector
contain the
3.1 the
(Low-n-rank
tensor
recovery).
entries
aresetting
to
deduced.
Denoting
set
of revealed
entries
by bΩ,will
thethen
corresponding
#locations.
N
N
given entries
the the
tensor
X.
Equivalently,
werank
can
compactly
write
the contain
constraint
operator
A retrieves
values
these
The
vector
then
theas
$of
"i=1
he function f , i.e., f (n-rank(X))
=of ||n-rank(X)||
=! locations.
(X
This
1
(i) ).b will
minimize
rank
X
s.
t.
A
(X)
=
b
(1)
XΩ entries
= TΩ , where
XΩ denotes
the restriction
of the
tensor onwrite
the entries
given byasΩ,
(i) we can
given
of the X∈T
tensor
X. Equivalently,
compactly
the constraint
oice, but is also possible
to
incorporate
a
weighting,
e.g.,
f
(n-rank(X))
=
i=1
values of
entries of
of the
X. tensor on the entries given by Ω,
Xand
TΩΩ ,contains
where Xthe
thethose
restriction
Ω =T
Ω denotes
ank X(i) . Thus, our minimization
problem
ofof interest
becomes:
and
TΩ contains
theof3.2
values
those
entries
of
The
exact
form
the(Tensor
linear
constraints
on X.
the tensor X is governed by the linear
Problem
setting
completion).
operator
which can
many
possible forms in general. We would like to point
Ncompletion).
$
ProblemAsetting
3.2take
(Tensor
!
"
m setting 3.1 (Low-n-rank
tensor
recovery).
out one special case,
the tensorNcompletion
case.
In
tensor
a subset of the(2)
minimize
rank
X
s. t.
XΩcompletion,
= TΩ
(i)
$
!
"
X∈T
N entries of the tensor
X is given, i=1
and
low-n-rank
the unknown
rankunder
X(i) the s.
t. XΩ =assumption,
TΩ
(2)
$
!
" minimize
X∈T
entries
are
to
be
deduced.
Denoting
the
set
of
revealed
entries
by
Ω,
the
corresponding
minimize
rankTheXlow-n-rank
s. t.
(X) = problem
b
(1)(2)) is a difficult
tensorAi=1
recovery
(1) (also its variant
(i)
operator A retrieves the values of these locations. The vector b will then contain the
X∈T
non-convex
problem.
Therefore
we will
relax (1)
it to(also
the following
The low-n-rank
tensor
recovery
problem
its variantconvex
(2)) isproblem:
a difficult
i=1given
entries of the tensor X. Equivalently, we can compactly write the constraint as
non-convex
Therefore
we will relax
it to
the following convex problem:
Problem problem.
setting
3.3
(Low-n-rank
tensor
pursuit).
XΩthe
denotes
the restriction
of the
tensor
on the
entries given by Ω,
Ω = TΩ , whereon
exact form of the linearXconstraints
tensor
X
is
governed
by
the
linear
N
$ entries
Problem
setting
(Low-n-rank
tensor
pursuit).
and
TΩ contains
the3.3
values
of those
of X.
minimize
||X
||
s.
t. A(X)
b. point
(3)
A which can take many possible forms
in
general.
We
would
like= to
N
(i) ∗
$
Problem setting 3.2X∈T
(Tensor
completion).
i=1
minimize
||X
s. t. A(X)
= b. of the
(3)
(i) ||∗
pecial case, the tensor completion case.
In tensor
completion,
a subset
N
#nX∈T
$
!
"
i=1
Here,under
||Z||∗ =the
σi (Z) denotes
theXnuclear
the
sum
rank
s.norm,
t. X
TΩ of the singular values
(2) of
i=1
the tensor X is given, and
low-n-rank
assumption,
the
Ω =unknown
(i)
#minimize
X∈T
n
a matrix
known
thatthe
thenuclear
nuclearnorm,
norm the
is the
greatest
convex minorant
Here,
||Z|| Z.
= It is well
σ (Z)
denotes
sum
of the singular
values of of
i=1
Existing Tensor Completion Approaches
•  Translate directly
Tensor Completion Question
•  Can the ideas from structural matrix completion be
applied to tensors?
•  What kinds of bounds exist for the amount of
information needed?
•  Are there special cases that make tensors easier?
•  How does rigidity translate to tensors?