From Stability to Differential Privacy
Abhradeep Guha Thakurta
Yahoo! Labs, Sunnyvale
Thesis: Stable algorithms yield
differentially private algorithms
Differential privacy: A short tutorial
Privacy in Machine Learning Systems
Individuals
𝑑1
𝑑2
𝑑1
𝑑𝑛−1
𝑑𝑛
Privacy in Machine Learning Systems
Individuals
𝑑1
𝑑2
𝑑1
𝑑𝑛−1
𝑑𝑛
Trusted learning Algorithm
ℳ
Privacy in Machine Learning Systems
Individuals
𝑑1
𝑑2
𝑑1
𝑑𝑛−1
𝑑𝑛
Trusted learning Algorithm
ℳ
Summary
statistics
1. Classifiers
2. Clusters
3. Regression
coefficients
Users
Privacy in Machine Learning Systems
Individuals
𝑑1
𝑑2
𝑑1
𝑑𝑛−1
𝑑𝑛
Attacker
Trusted learning Algorithm
ℳ
Summary
statistics
1. Classifiers
2. Clusters
3. Regression
coefficients
Users
Privacy in Machine Learning Systems
Two conflicting goals:
𝑑1
1.
Utility: Release accurate information
𝑑1
2.
Privacy: Protect privacy of individual entries
𝑑𝑛
𝑑2
𝑑𝑛−1
Balancing the tradeoff is a difficult problem:
1.
Netflix prize database attack [NS08]
2.
Facebook advertisement system attack [Korolova11]
3.
Amazon recommendation system attack [CKNFS11]
Data privacy is an active area of research:
• Computer science, economics, statistics, biology, social sciences …
Learning
Algorithm
ℳ
Users
Differential Privacy [DMNS06, DKMMN06]
Intuition:
• Adversary learns essentially the same thing irrespective of your
presence or absence in the data set
𝑑1
Data set: 𝐷
M
Random coins
M(𝐷)
𝑑1
Data set: 𝐷’
M
M(𝐷’)
Random coins
• 𝐷 and 𝐷′ are called neighboring data sets
• Require: Neighboring data sets induce close distribution on outputs
Differential Privacy [DMNS06, DKMMN06]
Definition:
A randomized algorithm M is (𝜖, 𝛿)-differentially private if
• for all data sets 𝐷 and 𝐷 ′ that differ in one element
• for all sets of answers 𝑆
Pr M 𝐷 ∈ 𝑆 ≤ 𝑒 Pr[M(𝐷′) ∈ 𝑆] + 𝛿
𝜖
Semantics of Differential Privacy
• Differential privacy is a condition on the algorithm
• Guarantee is meaningful in the presence of any auxiliary information
• Typically, think of privacy parameters: 𝜖 ≈ 0.1 and 𝛿 = 1/𝑛log 𝑛 ,
where 𝑛 = # of data samples
• Composition: 𝜖 ’s and 𝛿 ‘s add up over multiple executions
Laplace Mechanism [DMNS06]
Data set 𝐷 = {𝑑1 , ⋯ , 𝑑𝑛 } and 𝑓: 𝑈 ∗ → ℝ be a function on 𝐷
Sensitivity: S(𝑓)=
max
𝐷,𝐷′ ,𝑑𝐻 𝐷,𝐷′ =1
|𝑓 𝐷 − 𝑓 𝐷 ′ |
1. 𝐸: Random variable sampled from Lap(𝑆(𝑓)/𝜖)
2. Output 𝑓 𝐷 + 𝐸
Theorem (Privacy): Algorithm is 𝜖-differentially private
This Talk
1. Differential privacy via stability arguments: A meta-algorithm
2. Sample and aggregate framework and private model selection
3. Non-private sparse linear regression in high-dimensions
4. Private sparse linear regression with (nearly) optimal rate
Perturbation stability
(a.k.a. zero local sensitivity)
Perturbation Stability
Data set 𝐷
𝑑1
𝑑2
Function 𝑓
⋮
𝑑𝑛
Output
Perturbation Stability
Data set 𝐷′
𝑑1
𝑑2 ′
Function 𝑓
⋮
𝑑𝑛
Output
Stability of 𝑓 at 𝐷: The output does not change on changing any one entry
Equivalently, local sensitivity of 𝑓 at 𝐷 is zero
Distance to Instability Property
• Definition: A function 𝑓: 𝑈 ∗ → ℜ is 𝑘 −stable at a data set 𝐷 if
• For any data set 𝐷′ ∈ 𝑈 ∗ , with 𝐷Δ𝐷′ ≤ 𝑘,
𝑓 𝐷 = 𝑓(𝐷 ′ )
• Distance to instability:
max(𝑓 𝐷 𝑖𝑠 𝑘 − 𝑠𝑡𝑎𝑏𝑙𝑒)
𝑘
• Objective: Output 𝑓(𝐷) while preserving
differential privacy
All data sets
Unstable data sets
Distance > 𝑘
𝐷
Stable data sets
Propose-Test-Release (PTR) framework
[DL09, KRSY11, Smith T.’13]
A Meta-algorithm: Propose-Test-Release (PTR)
Basic tool: Laplace
mechanism
1. 𝑑𝑖𝑠𝑡 ← max(𝑓 𝐷 𝑖𝑠 𝑘 − 𝑠𝑡𝑎𝑏𝑙𝑒)
𝑘
2. 𝑑𝑖𝑠𝑡 ← 𝑑𝑖𝑠𝑡 + 𝐿𝑎𝑝
3. If 𝑑𝑖𝑠𝑡 >
𝑙𝑜𝑔(1/𝛿)
,
𝜖
1
𝜖
then return 𝑓(𝐷), else return ⊥
Theorem: The algorithm is 𝜖, 𝛿 −differentially private
2𝑙𝑜𝑔 1/𝛿
𝜖
Theorem: If 𝑓 is
-stable at 𝐷, then w.p. ≥ 1 − 𝛿 the algorithm
outputs 𝑓(𝐷)
This Talk
1. Differential privacy via stability arguments: A meta-algorithm
2. Sample and aggregate framework and private model selection
3. Non-private sparse linear regression in high-dimensions
4. Private sparse linear regression with (nearly) optimal rate
Sample and aggregate framework
[NRS07, Smith11, Smith T.’13]
Sample and Aggregate Framework
Data set 𝐷
Subsample
𝐷1
𝐷𝑚
Aggregator
Algorithm
Output
Sample and Aggregate Framework
Theorem: If the aggregator is 𝜖/𝜂, 𝛿 −differentially private, then
the overall framework is 𝜖, 𝛿 −differentially private
Assumption: Each entry appears in
𝜂 data blocks
Proof: Each data entry affects only
one data block
A differentially private aggregator using PTR
framework [Smith T.’13]
An 𝜖, 𝛿 −differentially Private Aggregator
Assumption: 𝑟 discrete possible outputs 𝑆1 , ⋯ , 𝑆𝑟
𝐷1
𝐷𝑚
Vote
Count
Vote
𝑆1
𝑆2
𝑆∗
𝑆𝑟
PTR+Report-Noisy-Max Aggregator
Function 𝑓: Candidate output with the maximum votes
1. 𝑑𝑖𝑠𝑡 ← max(𝑓 𝐷 𝑖𝑠 𝑘 − 𝑠𝑡𝑎𝑏𝑙𝑒)
𝑘
2. 𝑑𝑖𝑠𝑡 ← 𝑑𝑖𝑠𝑡 + 𝐿𝑎𝑝
3. If 𝑑𝑖𝑠𝑡 >
𝑙𝑜𝑔(1/𝛿)
,
𝜖
1
𝜖
then return 𝑓(𝐷), else return ⊥
Observation: 2 × (𝑑𝑖𝑠𝑡 + 1) is the gap between the counts of
highest and the second highest scoring model
Observation: The algorithm is always computationally efficient
Analysis of the aggregator under subsampling
stability [Smith T.’13]
Subsampling Stability
Data set 𝐷
Random subsample
with replacement
𝐷1
Stability:
𝐷𝑚
Function
𝑓(𝐷𝑖 )
=
Function
𝑓(𝐷)
w.p. >
3
4
A Private Aggregator using Subsampling Stability
Voting histogram (in expectation)
• 𝐷𝑖 : Sample each entry from 𝐷
w.p. 𝑞 = 𝑂(𝜖/log(1/𝛿))
• Each entry of 𝐷 appears in
≈ 𝑚𝑞 data blocks
1
m
2
3
𝑚
4
1
𝑚
2
1
𝑚
4
𝑆1
𝑆2
𝑆∗
𝑆𝑟
PTR+Report-Noisy-Max Aggregator
• 𝐷𝑖 : Sample each entry from 𝐷 w.p. 𝑞 = 𝑂(𝜖/log(1/𝛿))
• Each entry of 𝐷 appears in 2𝑚𝑞 data blocks w.p. ≥ 1 − 𝛿
1. 𝑑𝑖𝑠𝑡 ← (𝑆∗ − 𝑆2 )/4𝑚𝑞 − 1
2. 𝑑𝑖𝑠𝑡 ← 𝑑𝑖𝑠𝑡 + 𝐿𝑎𝑝
3. If 𝑑𝑖𝑠𝑡 >
𝑙𝑜𝑔(1/𝛿)
,
𝜖
1
𝜖
then return 𝑓(𝐷), else return ⊥
𝑆1
𝑆2
𝑆∗ 𝑆𝑟
A Private Aggregator using Subsampling Stability
Theorem: Above algorithm
Notice: Utility guarantee does not depend on the
of candidate private
models
is 𝜖, 2𝛿number
−differentially
Theorem: If 𝑚 = log 𝑛/𝛿 /𝑞2 ,then with probability at least 1 − 2𝛿,
the true answer 𝑓 𝐷 is output
This Talk
1. Differential privacy via stability arguments: A meta-algorithm
2. Sample and aggregate framework and private model selection
3. Non-private sparse linear regression in high-dimensions
4. Private sparse linear regression with (nearly) optimal rate
Sparse linear regression in high-dimensions
and the LASSO
Sparse Linear Regression in High-dimensions (𝑝 ≫ 𝑛)
• Data set: 𝐷 = { 𝑥1 , 𝑦1 , ⋯ , 𝑥𝑛 , 𝑦𝑛 } where 𝑥𝑖 ∈ ℝ𝑝 and 𝑦𝑖 ∈ ℝ
• Assumption: Data generated by noisy linear system
=
Feature vector
𝑥𝑖
Parameter vector
𝑦𝑖
∗
𝜃𝑝×1
+
𝑤𝑖
Field noise
Data normalization:
• ∀𝑖 ∈ 𝑛 , ||𝑥𝑖 ||∞ ≤ 1
• ∀𝑖, 𝑤𝑖 is sub-Gaussian
Sparse Linear Regression in High-dimensions (𝑝 ≫ 𝑛)
• Data set: 𝐷 = { 𝑥1 , 𝑦1 , ⋯ , 𝑥𝑛 , 𝑦𝑛 } where 𝑥𝑖 ∈ ℝ𝑝 and 𝑦𝑖 ∈ ℝ
𝑦𝑛×1
𝑋𝑛×𝑝
∗
𝜃𝑝×1
+
Field noise
=
Design matrix
Parameter vector
Response vector
• Assumption: Data generated by noisy linear system
𝑤𝑛×1
𝑦𝑛×1
=
Design matrix
+
Field noise
Response vector
Sparse Linear Regression in High-dimensions (𝑝 ≫ 𝑛)
𝑤𝑛×1
𝑋𝑛×𝑝
∗
𝜃𝑝×1
• Sparsity: 𝜃 ∗ has 𝑠 < 𝑛 non-zero entries
• Bounded norm: ∀𝑖
∗
|𝜃𝑖 |
∈ (1 − Φ, Φ) for arbitrary small const. Φ
Model selection problem: Find the non-zero coordinates of 𝜃 ∗
𝑦𝑛×1
=
Design matrix
+
Field noise
Response vector
Sparse Linear Regression in High-dimensions (𝑝 ≫ 𝑛)
𝑤𝑛×1
𝑋𝑛×𝑝
∗
𝜃𝑝×1
Model selection: Non-zero coordinates (or the support) of 𝜃 ∗
Solution: LASSO estimator [Tibshirani94,EFJT03,Wainwright06,CT07,ZY07,…]
1
𝜃 ∈ arg min𝑝
||𝑦 − 𝑋𝜃||22 + Λ||𝜃||1
𝜃∈ℝ 2𝑛
Consistency of the LASSO Estimator
=
+
Consistency conditions* [Wainwright06,ZY07]:
• Γ: Support of the underlying parameter vector 𝜃 ∗
𝑋Γ
Incoherence
||| 𝑋Γ𝑇𝑐 𝑋Γ
𝑋Γ𝑐
Restricted Strong Convexity
1
−1
𝑇
𝑋Γ 𝑋Γ |||∞ <
4
𝜆𝑚𝑖𝑛 𝑋Γ𝑇 𝑋Γ = Ω(𝑛)
Consistency of the LASSO Estimator
=
+
Consistency conditions* [Wainwright06,ZY07]:
• Γ: Support of the underlying parameter vector 𝜃 ∗
Theorem*: Under proper choice of Λ and 𝑛 = Ω(𝑠log 𝑝), support
of the LASSO estimator 𝜃 equals support of 𝜃 ∗
Incoherence
||| 𝑋Γ𝑇𝑐 𝑋Γ
Restricted Strong Convexity
1
−1
𝑇
𝑋Γ 𝑋Γ |||∞ <
4
𝜆𝑚𝑖𝑛 𝑋Γ𝑇 𝑋Γ = Ω(𝑛)
Stochastic Consistency of the LASSO
=
+
Consistency conditions* [Wainwright06,ZY07]:
• Γ: Support of the underlying parameter vector 𝜃 ∗
Incoherence
||| 𝑋Γ𝑇𝑐 𝑋Γ 𝑋Γ𝑇 𝑋Γ
Restricted Strong Convexity
−1
1
|||∞ <
4
𝜆𝑚𝑖𝑛 𝑋Γ𝑇 𝑋Γ = Ω(𝑛)
Theorem [Wainwright06,ZY07]: If each data entry in 𝑋 ∼ 𝒩(0,1/4),
then the assumptions above are satisfied w.h.p.
We show [Smith,T.’13]
Consistency conditions
Perturbation stability
Proxy conditions
This Talk
1. Differential privacy via stability arguments: A meta-algorithm
2. Sample and aggregate framework and private model selection
3. Non-private sparse linear regression in high-dimensions
4. Private sparse linear regression with (nearly) optimal rate
Interlude: A simple subsampling based private
LASSO algorithm [Smith,T.’13]
Notion of Neighboring Data sets
Design matrix
Response vector
𝑥𝑖
𝑦𝑖
Data set 𝐷 = 𝑛
𝑝
Notion of Neighboring Data sets
Design matrix
𝑥𝑖 ′
Data set 𝐷′ =
Response vector
𝑦𝑖 ′
𝑛
𝑝
𝐷 and 𝐷′ are neighboring data sets
Recap: Subsampling Stability
Data set 𝐷
Random subsample
with replacement
𝐷1
Stability:
𝐷𝑚
Function
𝑓(𝐷𝑖 )
=
Function
𝑓(𝐷)
w.p. >
3
4
Recap: PTR+Report-Noisy-Max Aggregator
=
+
𝑝
Assumption: All 𝑟 =
candidate models 𝑆1 , ⋯ , 𝑆𝑟
𝑠
𝐷1
𝐷𝑚
𝑓
Vote
Count
Vote
𝑓
𝑓
𝑆1
𝑆2
𝑆∗
𝑆𝑘
Recap: PTR+Report-Noisy-Max Aggregator
• 𝐷𝑖 : Sample each entry from 𝐷 w.p. 𝑞 = 𝑂(𝜖/log(1/𝛿))
• Each entry of 𝐷 appears in 2𝑚𝑞 data blocks w.p. ≥ 1 − 𝛿
• Fix 𝑚 = log(𝑛/𝛿)/𝑞2
1. 𝑑𝑖𝑠𝑡 ← (𝑆∗ − 𝑆2 )/4𝑚𝑞 − 1
2. 𝑑𝑖𝑠𝑡 ← 𝑑𝑖𝑠𝑡 + 𝐿𝑎𝑝
3. If 𝑑𝑖𝑠𝑡 >
𝑙𝑜𝑔(1/𝛿)
,
𝜖
1
𝜖
then return 𝑓(𝐷), else return ⊥
𝑆1
𝑆2
𝑆∗ 𝑆𝑟
Subsampling Stability of the LASSO
𝑦𝑛×1
𝑋𝑛×𝑝
∗
𝜃𝑝×1
+
Field noise
=
Design matrix
Parameter vector
Response vector
Stochastic assumptions: Each data entry in 𝑋 ∼ 𝒩(0,1/4)
Noise 𝑤 ∼ 𝒩(0, 𝜎 2 𝕀𝑛 )
𝑤𝑛×1
Subsampling Stability of the LASSO
=
+
Notice the gap of log(1/𝛿) /𝜖
Stochastic assumptions: Each data entry in 𝑋 ∼ 𝒩(0,1/4)
Noise 𝑤 ∼ 𝒩(0, 𝜎 2 𝕀Scale
𝑛 ) of 𝛿 = 1/𝑛𝜔 1
Theorem [Wainwright06,ZY07]: Under proper choice of Λ and 𝑛 =
Ω(𝑠 log 𝑝) , support of the LASSO estimator 𝜃 equals support of 𝜃 ∗
Theorem: Under proper choice of Λ , 𝑛 = Ω
log(1/𝛿)𝑠 log 𝑝
𝜖
and
𝑓 = 𝐿𝐴𝑆𝑆𝑂, the output of the aggregator equals support of 𝜃 ∗
Perturbation stability based private LASSO and
optimal sample complexity [Smith,T.’13]
Recap: Distance to Instability Property
• Definition: A function 𝑓: 𝑈 ∗ → ℜ is 𝑘 −stable at a data set 𝐷 if
• For any data set 𝐷′ ∈ 𝑈 ∗ , with 𝐷Δ𝐷′ ≤ 𝑘,
𝑓 𝐷 = 𝑓(𝐷 ′ )
• Distance to instability:
max(𝑓 𝐷 𝑖𝑠 𝑘 − 𝑠𝑡𝑎𝑏𝑙𝑒)
𝑘
• Objective: Output 𝑓(𝐷) while preserving
differential privacy
All data sets
Unstable data sets
Distance > 𝑘
𝐷
Stable data sets
Recap: Propose-Test-Release
Framework
(PTR)
TBD: Some global sensitivity one query
1. 𝑑𝑖𝑠𝑡 ← max(𝑓 𝐷 𝑖𝑠 𝑘 − 𝑠𝑡𝑎𝑏𝑙𝑒)
𝑘
2. 𝑑𝑖𝑠𝑡 ← 𝑑𝑖𝑠𝑡 + 𝐿𝑎𝑝
3. If 𝑑𝑖𝑠𝑡 >
𝑙𝑜𝑔(1/𝛿)
,
𝜖
1
𝜖
then return 𝑓(𝐷), else return ⊥
Theorem: The algorithm is 𝜖, 𝛿 −differentially private
2𝑙𝑜𝑔 1/𝛿
𝜖
Theorem: If 𝑓 is
-stable at 𝐷, then w.p. ≥ 1 − 𝛿 the algorithm
outputs 𝑓(𝐷)
Instantiation of PTR for the LASSO
=
LASSO: 𝜃 ∈
1
arg min𝑝 ||𝑦
𝜃∈ℝ 2𝑛
− 𝑋𝜃||22 + Λ||𝜃||1
• Set function 𝑓 =support of 𝜃
• Issue: For 𝑓, distance to instability might not be efficiently
computable
+
From [Smith,T.’13]
Consistency conditions
Perturbation stability
Proxy conditions
This talk
Consistency conditions
Perturbation stability
Proxy conditions
(Efficiently testable with privacy)
Perturbation Stability of the LASSO
=
LASSO: 𝜃 ∈
1
arg min𝑝 ||𝑦
𝜃∈ℝ 2𝑛
+
− 𝑋𝜃||22 + Λ||𝜃||1
𝐽(𝜃)
Theorem: Consistency conditions on LASSO are sufficient for
perturbation stability
Proof Sketch: 1. Analyze Karush-Kuhn-Tucker (KKT) optimality
conditions at 𝜃
2. Show that support(𝜃) is stable via using
‘’dual certificate’’ on stable instances
Perturbation Stability of the LASSO
=
Proof Sketch:
1 𝑇
Gradient of LASSO 𝜕𝐽𝐷 (𝜃)= − 𝑋 𝑦 − 𝑋𝜃 + Λ𝜕||𝜃||1
𝑛
Lasso objective on 𝐷
𝜃
0 ∈ 𝜕𝐽𝐷 (𝜃)
Lasso objective on 𝐷′
𝜃′
0 ∈ 𝜕𝐽𝐷′ (𝜃′)
+
Perturbation Stability of the LASSO
=
Proof Sketch:
1 𝑇
Gradient of LASSO 𝜕𝐽𝐷 (𝜃)= − 𝑋 𝑦 − 𝑋𝜃 + Λ𝜕||𝜃||1
𝑛
Argue using the optimality conditions of 𝜕𝐽𝐷 (𝜃) and 𝜕𝐽𝐷′ (𝜃′)
1. No zero coordinates of 𝜃 become non-zero in 𝜃′
(use mutual incoherence condition)
2. No non-zero coordinates of 𝜃 become zero in 𝜃′
(use restricted strong convexity condition)
+
Perturbation Stability Test for the LASSO
=
+
0 0
Γ: Support of 𝜃
Γ c : Complement of the support of 𝜃
Test for the following (real test is more complex):
• Restricted Strong Convexity (RSC): Minimum eigenvalue of 𝑋ΓT XΓ is Ω(𝑛)
• Strong stability: Negative of the (absolute) coordinates of the gradient
of the least-squared loss in Γ 𝑐 are ≪ Λ
Geometry of the Stability of LASSO
=
+
Intuition: Strong convexity ensures supp(𝜃)⊆ supp(𝜃′)
1. Strong convexity ensures
||𝜃Γ − 𝜃′Γ ||∞ is small
Lasso objective along Γ
2. If ∀𝑖, |𝜃Γ (𝑖)| is large, then
∀𝑖, 𝜃′Γ 𝑖 > 0
3. Consistency conditions imply
∀𝑖, |𝜃Γ (𝑖)| is large
Dimension 2 in Γ 𝑐
Dimension 1 in Γ
𝜃
Geometry of the Stability of LASSO
=
+
Intuition: Strong stability ensures no zero coordinate in 𝜃 becomes
non-zero in 𝜃′
Lasso objective along Γ c
Slope: Λ
Slope: -Λ
Dimension 1 in Γ
Dimension 2 in Γ 𝑐
𝜃
• For the minimizer 𝜃 to move along Γ 𝑐 , the perturbation to the
gradient of least-squared loss has to be large
Geometry of the Stability of LASSO
=
+
Gradient of the least-squared loss:
Γ
−𝑋 𝑇 𝑦 − 𝑋𝜃 =
Lasso objective along Γ c
𝑎𝑖
Γc
Slope: Λ
Slope: -Λ
𝑎𝑝
Dimension 1 in Γ
Dimension 2 in Γ 𝑐
𝜃
• Strong stability: |𝑎𝑖 | ≪ Λ for all 𝑖 ∈ Γ 𝑐 ⇒
𝑖 ∈ Γ 𝑐 has a sub-gradient of zero for LASSO(𝐷′)
Making the Stability Test Private (Simplified)
=
+
Test for Restricted Strong Convexity: 𝑔1 𝐷
𝑔2
Test for strong stability: 𝑔2 𝐷
Issue: If 𝑔1 𝐷 > 𝑡1 and 𝑔2 𝐷 > 𝑡2 , then
sensitivities are Δ1 and Δ2
𝑔1
Our solution: Proxy distance
𝑑 = max
𝑔𝑖 𝐷 −𝑡𝑖
min
Δ𝑖
𝑖
• 𝑑 has global sensitivity of one
𝑔1 and 𝑔2 are both large
and insensitive
+ 1,0
Private Model Selection with Optimal Sample
Complexity
=
Nearly optimal sample complexity
+
1. Compute 𝑑= function of 𝑔1 (𝐷) and 𝑔2 𝐷
2. 𝑑𝑖𝑠𝑡 ← 𝑑 + 𝐿𝑎𝑝
3. If 𝑑𝑖𝑠𝑡 >
𝒍𝒐𝒈(𝟏/𝜹)
,
𝝐
1
𝜖
then return 𝑠𝑢𝑝𝑝(𝜃), else return ⊥
Theorem: The algorithm is 𝜖, 𝛿 −differentially private
Theorem: Under consistency conditions , log 𝑝 > 𝛼 2 𝑠 3 and 𝑛 = Ω(𝑠 log 𝑝),
w.h.p. the support of 𝜃 ∗ is output. Here 𝛼 = log(1/𝛿)/𝜖.
Thesis: Stable algorithms yield
differentially private algorithms
Two notions of stability:
1. Perturbation stability
2. Subsampling stability
This Talk
1. Differential privacy via stability arguments: A meta-algorithm
2. Sample and aggregate framework and private model selection
3. Non-private sparse linear regression in high-dimensions
4. Private sparse linear regression with (nearly) optimal rate
Concluding Remarks
1. Sample and aggregate framework with PTR+report-noisy-max
aggregator is a generic tool for designing learning algorithms
• Example: learning with non-convex models [Bilenko,Dwork,Rothblum,T.]
2. Propose-test-release framework is an interesting tool if one can
compute distance to instability efficiently
3. Open problem: Private high-dimensional learning without
assumptions like incoherence and restricted
strong convexity