slides

advertisement
Noisy Sparse Subspace Clustering
with dimension reduction
YINING WANG , YU - X I A N G WA N G, A A RTI SI N GH
MAC HI NE L EA R NING DE PARTMENT
C A R N EGI E ME L LO N U N I V ERS ITY
1
Subspace Clustering
2
Subspace Clustering Applications
Motion Trajectories tracking1
1 (Elhamifar
and Vidal, 2013), (Tomasi and Kanade, 1992)
3
Subspace Clustering Applications
Face Clustering1
Network hop counts, movie ratings, social graphs, …
1 (Elhamifar
and Vidal, 2013), (Basri and Jacobs, 2003)
4
Sparse Subspace Clustering
β—¦ (Elhamifar and Vidal, 2013), (Wang and Xu, 2013).
β—¦ Data: 𝑋 = π‘₯1 , π‘₯2 β‹― , π‘₯𝑁 ⊆ 𝑅𝑑
β—¦ Key idea: similarity graph based on l1 self-regression
π‘₯1
π‘₯2
π‘₯3
π‘₯𝑁
π‘₯1 π‘₯2 π‘₯3
π‘₯𝑁
5
Sparse Subspace Clustering
β—¦ (Elhamifar and Vidal, 2013), (Wang and Xu, 2013).
β—¦ Data: 𝑋 = π‘₯1 , π‘₯2 , β‹― , π‘₯𝑁 ⊆ 𝑅𝑑
β—¦ Key idea: similarity graph based on l1 self-regression
𝑐𝑖 = argmin𝑐𝑖 𝑐𝑖
s.t. π‘₯𝑖 = 𝑗≠𝑖 𝑐𝑖𝑗 π‘₯𝑗
𝑐𝑖 = argmin𝑐𝑖 π‘₯𝑖 −
1
𝑗≠𝑖 𝑐𝑖𝑗 π‘₯𝑗
Noiseless data
+πœ† 𝑐𝑖
1
Noisy data
6
SSC with dimension reduction
β—¦ Real-world data are usually high-dimensional
β—¦ Hopkins-155: 𝑑 = 112~240
β—¦ Extended Yale Face-B: 𝑑 ≥ 1000
β—¦ Computational concerns
β—¦ Data availability: values of some features might be missing
β—¦ Privacy concerns: releasing the raw data might cause privacy
breaches.
7
SSC with dimension reduction
β—¦ Dimensionality reduction:
𝑋 = Ψ𝑋,
Ψ ∈ 𝑅 𝑝×𝑑 ,
𝑝β‰ͺ𝑑
β—¦ How small can p be?
β—¦ A trivial result: 𝑝 = Ω(πΏπ‘Ÿ) is OK.
β—¦ L: the number of subspaces (clusters)
β—¦ r: the intrinsic dimension of each subspace
β—¦ Can we do better?
8
Pr ∀𝒙 ∈ 𝑺, Ψπ‘₯
2
2
∈ 1±πœ– π‘₯
2
2
≥ 1−𝛿
Main Result
1. 𝑝 = Ω πΏπ‘Ÿ ⟹ 𝑝 = Ω(π‘Ÿ log 𝑁),if Ψ is a subspace embedding
β—¦
β—¦
β—¦
β—¦
β—¦
2.
Random Gaussian projection
Fast Johnson-Lindenstrauss Transform (FJLT)
Uniform row subsampling under incoherence conditions
Sketching
……
Lasso SSC should be used even if data are noiseless.
9
Proof sketch
β—¦ Review of deterministic success conditions for SSC
(Soltanolkotabi and Candes, 12)(Elhamifar and Vidal, 13)
β—¦ Subspace incoherence
β—¦ Inradius
β—¦ Analyze perturbation under dimension reduction
β—¦ Main results for noiseless and noisy cases.
10
Review of SSC success condition
β—¦ Subspace incoherence
β—¦ Characterizing inter-subspace separation
πœ‡β„“ ≔ max(β„“) max normalize 𝑣 π‘₯𝑖
𝑖
π‘₯∈𝑋\X
β„“
,π‘₯
where 𝑣(π‘₯) solves
πœ†
𝑣 22
2
𝑇
𝑋 𝑣 ∞≤1
max 𝑣, π‘₯ −
𝑣
𝑠. 𝑑.
Lasso SSC
formulation
Dual problem of Lasso SSC
11
Review of SSC success condition
β—¦ Inradius
β—¦ Characterzing inner-subspace data point distribution
Large inradius 𝜌
Small inradius 𝜌
12
Review of SSC success condition
(Soltanolkotabi & Candes, 2012)
Noiseless SSC succeeds (similarity graph has
no false connection) if
𝝁<𝝆
With dimensionality reduction:
πœ‡→πœ‡,
𝜌→𝜌
Bound 𝝁 − 𝝁 , 𝝆 − 𝝆
13
Perturbation of subspace
incoherence πœ‡
𝜈 = argmax
𝜈: 𝑋 𝑇 𝜈 ∞ ≤1
𝜈 = argmax
𝜈: 𝑋 𝑇 𝜈 ∞ ≤1
πœ†
𝜈, π‘₯ − 𝜈
2
πœ†
𝜈, π‘₯ − 𝜈
2
2
2
2
2
We know that 𝜈, π‘₯ ≈ 𝜈, π‘₯ …
So 𝜈 ≈ 𝜈 because of strong convexity
14
Perturbation of inradius 𝜌
Main idea: linear operator transforms a ball to an ellipsoid
15
Main result
SSC with dimensionality reduction succeeds
(similarity graph has no false connection) if
𝝁 + πŸ‘πŸ 𝝐/𝝀 + πŸ‘π < 𝝆
Regularization
Errorgap
of approximate
Lasso
Noisy
case: (πœ‚the
is the
adversarial
noise
Takeaways:
geometric
Δ = parameter
𝜌level)
− πœ‡isometry
is aofresource
ifrequired
𝑝 =dimension
Ω(π‘Ÿeven
log 𝑁)
that can be traded-off
for
data
LassoO(1)
SSC
forreduction
noiseless problem
2
5πœ‚
8(πœ– + 3πœ‚)
πœ‡ + 16
+
+ 3πœ– < 𝜌
πœŒβ„“
πœ†
16
Simulation results (Hopkins 155)
17
Conclusion
β—¦ SSC provably succeeds with dimensionality reduction
β—¦ Dimension after reduction 𝑝 can be as small as Ω(π‘Ÿ log 𝑁)
β—¦ Lasso SSC is required for provable results.
Questions?
18
References
β—¦ M. Soltanolkotabi and E. Candes. A Geometric Analysis of Subspace
Clustering with Outliers. Annals of Statistics, 2012.
β—¦ E. Elhamifar and R. Vidal. Sparse Subspace Clustering: Algorithm, Theory and
Applications. IEEE TPAMI, 2013
β—¦ C. Tomasi and T. Kanade. Shape and Motion from Image Streams under
Orthography. IJCV, 1992.
β—¦ R. Basri and D. Jacobs. Lambertian Reflection and Linear Subspaces. IEEE
TPAMI, 2003.
β—¦ Y.-X., Wang and H., Xu. Noisy Sparse Subspace Clustering. ICML, 2013.
19
Download