Noisy Sparse Subspace Clustering with dimension reduction YINING WANG , YU - X I A N G WA N G, A A RTI SI N GH MAC HI NE L EA R NING DE PARTMENT C A R N EGI E ME L LO N U N I V ERS ITY 1 Subspace Clustering 2 Subspace Clustering Applications Motion Trajectories tracking1 1 (Elhamifar and Vidal, 2013), (Tomasi and Kanade, 1992) 3 Subspace Clustering Applications Face Clustering1 Network hop counts, movie ratings, social graphs, … 1 (Elhamifar and Vidal, 2013), (Basri and Jacobs, 2003) 4 Sparse Subspace Clustering β¦ (Elhamifar and Vidal, 2013), (Wang and Xu, 2013). β¦ Data: π = π₯1 , π₯2 β― , π₯π ⊆ π π β¦ Key idea: similarity graph based on l1 self-regression π₯1 π₯2 π₯3 π₯π π₯1 π₯2 π₯3 π₯π 5 Sparse Subspace Clustering β¦ (Elhamifar and Vidal, 2013), (Wang and Xu, 2013). β¦ Data: π = π₯1 , π₯2 , β― , π₯π ⊆ π π β¦ Key idea: similarity graph based on l1 self-regression ππ = argminππ ππ s.t. π₯π = π≠π πππ π₯π ππ = argminππ π₯π − 1 π≠π πππ π₯π Noiseless data +π ππ 1 Noisy data 6 SSC with dimension reduction β¦ Real-world data are usually high-dimensional β¦ Hopkins-155: π = 112~240 β¦ Extended Yale Face-B: π ≥ 1000 β¦ Computational concerns β¦ Data availability: values of some features might be missing β¦ Privacy concerns: releasing the raw data might cause privacy breaches. 7 SSC with dimension reduction β¦ Dimensionality reduction: π = Ψπ, Ψ ∈ π π×π , πβͺπ β¦ How small can p be? β¦ A trivial result: π = Ω(πΏπ) is OK. β¦ L: the number of subspaces (clusters) β¦ r: the intrinsic dimension of each subspace β¦ Can we do better? 8 Pr ∀π ∈ πΊ, Ψπ₯ 2 2 ∈ 1±π π₯ 2 2 ≥ 1−πΏ Main Result 1. π = Ω πΏπ βΉ π = Ω(π log π),if Ψ is a subspace embedding β¦ β¦ β¦ β¦ β¦ 2. Random Gaussian projection Fast Johnson-Lindenstrauss Transform (FJLT) Uniform row subsampling under incoherence conditions Sketching …… Lasso SSC should be used even if data are noiseless. 9 Proof sketch β¦ Review of deterministic success conditions for SSC (Soltanolkotabi and Candes, 12)(Elhamifar and Vidal, 13) β¦ Subspace incoherence β¦ Inradius β¦ Analyze perturbation under dimension reduction β¦ Main results for noiseless and noisy cases. 10 Review of SSC success condition β¦ Subspace incoherence β¦ Characterizing inter-subspace separation πβ β max(β) max normalize π£ π₯π π π₯∈π\X β ,π₯ where π£(π₯) solves π π£ 22 2 π π π£ ∞≤1 max π£, π₯ − π£ π . π‘. Lasso SSC formulation Dual problem of Lasso SSC 11 Review of SSC success condition β¦ Inradius β¦ Characterzing inner-subspace data point distribution Large inradius π Small inradius π 12 Review of SSC success condition (Soltanolkotabi & Candes, 2012) Noiseless SSC succeeds (similarity graph has no false connection) if π<π With dimensionality reduction: π→π, π→π Bound π − π , π − π 13 Perturbation of subspace incoherence π π = argmax π: π π π ∞ ≤1 π = argmax π: π π π ∞ ≤1 π π, π₯ − π 2 π π, π₯ − π 2 2 2 2 2 We know that π, π₯ ≈ π, π₯ … So π ≈ π because of strong convexity 14 Perturbation of inradius π Main idea: linear operator transforms a ball to an ellipsoid 15 Main result SSC with dimensionality reduction succeeds (similarity graph has no false connection) if π + ππ π/π + ππ < π Regularization Errorgap of approximate Lasso Noisy case: (πthe is the adversarial noise Takeaways: geometric Δ = parameter πlevel) − πisometry is aofresource ifrequired π =dimension Ω(πeven log π) that can be traded-off for data LassoO(1) SSC forreduction noiseless problem 2 5π 8(π + 3π) π + 16 + + 3π < π πβ π 16 Simulation results (Hopkins 155) 17 Conclusion β¦ SSC provably succeeds with dimensionality reduction β¦ Dimension after reduction π can be as small as Ω(π log π) β¦ Lasso SSC is required for provable results. Questions? 18 References β¦ M. Soltanolkotabi and E. Candes. A Geometric Analysis of Subspace Clustering with Outliers. Annals of Statistics, 2012. β¦ E. Elhamifar and R. Vidal. Sparse Subspace Clustering: Algorithm, Theory and Applications. IEEE TPAMI, 2013 β¦ C. Tomasi and T. Kanade. Shape and Motion from Image Streams under Orthography. IJCV, 1992. β¦ R. Basri and D. Jacobs. Lambertian Reflection and Linear Subspaces. IEEE TPAMI, 2003. β¦ Y.-X., Wang and H., Xu. Noisy Sparse Subspace Clustering. ICML, 2013. 19