List decoding and pseudorandom constructions:
lossless expanders and extractors
from Parvaresh-Vardy codes
Venkatesan Guruswami
Carnegie Mellon University
--- CMI Pseudorandomness Workshop, Aug 23, 2011 ---
Connections in Pseudorandomness
Randomness
Extractors
[Tre99,RRV99,
ISW99,SU01,Uma02]
[Tre99,TZ01,
TZS01,SU01]
[GW94,WZ95,
TUZ01,RVW00,
CRVW02]
Pseudorandom
Generators
[STV99,SU01,Uma02]
Error-Correcting
Algebraic list Codes
Expander
decoding
codes
[GLR08,GLW08]
Expander
Graphs
[SS96,Spi96,
GI02,GI03,
GR06,GUV07]
Euclidean Sections,
Compressed sensing
Connections in Pseudorandomness
Randomness
Extractors
[Tre99,RRV99,
ISW99,SU01,U02]
[Tre99,TZ01,
TZS01,SU01]
This talk
[GW94,WZ95,
TUZ01,RVW00,
CRVW02]
This
talk
Expander
Graphs
List-Decodable
[PV05,GR06]
Error-Correcting
Codes
[GI02,GI03]
Pseudorandom
Generators
[STV99,SU01,U02]
List Decodable codes
• Code C  D with N codewords, alphabet size || = Q
• (e,L)-list-decodable: Every Hamming ball of radius e has
at most L codewords of C
– Combinatorial packing condition
– Balls of radius e around codewords cover each point  L times.
– List error correction of e errors with worst-case list size L
List Decoding Centric View of
Pseudorandom Objects
List decoding, in different notation
• Encoding function E : [N]  [Q]D
• View as map (bipartite graph)  : [N] x [D]  [D] x [Q]
– (x, y) = (y , E(x)y)
• List decoding property:
N
DxQ
For all r  [Q]D , if
T = { (y , ry) : y  [D] } then
|LIST(T)|  L where we define
D
LIST(T) =
x
{ x : (x, y)  T for at least
D - e values of y }
Bipartite expanders
N
S, |S| K
M
D
 : [N] x [D]  [M]
“(K,A) expander”
|(S)|  A¢|S|
(vertex expansion
A = expansion factor)
• For all K’ ≤ K, and T  [M] with |T| < AK’, LIST(T) < K’
where
LIST(T) = { x  [N] : for all y  [D], (x, y)  T }
Extractors
unknown source of length n with
k bits of “min-entropy”
N = 2n
“seed”
D = 2d
EXT
d random bits
m almost-uniform bits
M = 2m
Would like m  k
•  : [N] x [D]  [M] is a (k,)-extractor if for all T  [M],
|LIST(T)| < 2k where
LIST(T) = { x  [N] : Pry [ (x,y)  T ] ≥ |T|/M + }
Condensers
(weaker object en route extractors)
k-source of length n
seed
d random bits
COND
~ k’-source of length m
•
Output not close to uniform but is close to source with good min-entropy
– Ideally k’  k (don’t lose entropy), m  k (good entropy “rate”)
•
Can also be captured by list decoding type definition
– LIST(T) small for all small subsets T  [M] , where
LIST(T) = { x : Pry [ (x,y)  T ] ≥  }
The common framework
Definitions of various useful objects
 : [N] x [D]  [M] captured as:
“For all subsets T  [M] that obey certain property, a suitably defined
list decoding of T, LIST(T), has small size”
– List decodable codes: T arising out of received words
– Expanders, condensers: T of small size
• Also case for “list recoverable codes”
– Extractors: arbitrary T
The framework gives not just unified abstractions, but also a proof
method that leads to the best constructions and analysis.
Parameters of interest
• Map  : [N] x [D]  [M]
• What we care about varies for different objects
• Extractors: small seed length D (= poly(log N)); large
output length M
• Codes: want small alphabet size M, small D (= O(log N))
– Small |LIST(T)|, plus efficient algorithm to recover LIST(T)
• Tight analysis of size of LIST(T) :
– exact value not too crucial for codes;
– for lossless expanders it is crucial (factor 2 worse bound implies
factor 2 worse expansion)
The abstraction in action
• Unbalanced expanders
• Expander Construction from Parvaresh-Vardy codes
• View as condensers and application to extractors
• Conclusions
Unbalanced Expander Graphs
N
M
“(K,A) expander”
S, |S| K
D
|(S)|  A¢|S|
(vertex expansion)
Goals:
• Minimize D
• Maximize A (lossless expansion: A close to D)
• Minimize M (not much larger than O(KD))
Expanders have many uses …
•
•
•
•
•
•
•
•
•
Fault-tolerant networks (e.g., [Pin73,Chu78,GG81])
Sorting in parallel [AKS83]
Derandomization [AKS87,IZ89,INW94,IW97,Rei05,…]
PCP theorem [Din06]
Randomness Extractors [CW89,GW94,TUZ01,RVW00,GUV07]
Error-correcting codes [SS96,Spi96,LMSS01,GI01-04]
Distributed routing in networks [PU89,ALM96,BFU99].
Data structures [BMRV00].
Hard tautologies in proof complexity [BW99,ABRW00,AR01].
• Pseudorandom matrices, Almost Euclidean sections of L1N
[GLR’08,GLW’08]
• ….
Need explicit constructions (deterministic, time poly(log N)).
(Bipartite) Expander Graphs
N
M
“(K,A) expander”
S, |S| K
Goals:
• Minimize D
• Maximize A
• Minimize M
D
|(S)|  A¢|S|
Optimal (Non-constructive):
• D = O(log (N/M) / )
• A = (1-)¢D
• M = O(KD/)
Explicit Constructions
degree D
Optimal
O(log (N/M))
expansion A
|right-side| M
(1-)¢D
O(KD)
Ramanujan graphs O(1)
¼ D/2
N
Zig-zag [CRVW02] O(1)
(1-)¢D
.01 N
Ta-Shma, Umans, polylog(N)
(1-)¢D
Zuckerman[TUZ01] exp(poly(log log N)) (1-)¢D
exp(poly(log KD)
G., Umans,
Vadhan
poly(KD)
polylog(N)
 arbitrary positive constant.
(1-)¢D
poly(KD)
Utility of Expansion (1-)¢D
• At least (1-2) D |S| elements of (S) are
unique neighbors: touch exactly one edge from S
N
M
S, |S| K
Useful in Expander
codes [SS’96]
x
|(S)| (1-) D |S|
D
• Set membership in bit-probe model [BMRV’00]
• Fault tolerance: Even if an adversary removes say ¾ edges
from each vertex, lossless expansion maintained (with =4)
The Result
N
M
“(K,A) expander”
S, |S| K
D
|(S)|  A¢|S|
Theorem [GUV]:  N, K, >0, 9 explicit (K,A) expander with
• degree D = poly(log N, 1/)
• expansion A = (1-)¢D
• #right vertices M = D2¢ K1.01
Parvaresh-Vardy codes
• Variant of Reed-Solomon codes
• Parameters of construction: n, Fq , m, h, an irreducible
polynomial E(Y) of degree n over Fq
• Encoding: Given message f  Fqn or polynomial f(Y) 
Fq[Y] of degree (n-1),
– PV(f)y = (f0(y) , f1(y) , … , fm-1(y)) for y  Fq
where
fi(Y) = (f(Y))h^i mod E(Y)
• Define (f, y) = (y , PV(f)y)
– Consider bipartite expander with neighborhood given by 
Expander theorem
Left vertices = polynomials of degree · n-1 over Fq (N = qn)
Degree D = q
Right vertices = Fqm+1 (M = qm+1)
(f,y) = y’th neighbor of f =
2
m-1
(y, f(y), (fh mod E)(y), (fh mod E)(y), …, (fh mod E)(y))
where E(Y) = irreducible* poly of degree n over Fq
h = a parameter
Thm [GUV’07]: This is a (K,A) expander for K = hm, A = q-hnm.
* can be found deterministically in poly(n, log q, char(F )) time
q
Close relation to list decoding
• Proof of expansion based on list decoding of ParvareshVardy codes
– Need a tight analysis of list size
– For “list recovery” version
y1
 K
Possible values
for each position
S1
y2
S2
yq
Sq
Recall list decoding view
N
M
“(=K,A) expander”
S, |S|=K
D
|(S)|  A¢ K
• For Tµ [M], define LIST(T) = {x2 [N] : (x)µT}
• Lemma: G is a (=K,A) expander if and only if
for all Tµ [M] of size AK-1, we have |LIST(T)| · K-1
Expansion analysis
(f,y) = (y, f(y), (fh mod E)(y), …, (fh
m-1
mod E)(y))
f = poly of degree · n-1, y  Fq, E = irreducible of degree n
= hm, we have
Theorem: For A = q - nmh and any K ·
Tµ Fqm+1 of size AK-1) |LIST(T)|· K-1
Proof outline, following [S97,GS99,PV05]:
1. Find a nonzero low-degree multivariate polynomial Q
vanishing on T.
2. Show that every f 2LIST(T) is a root of a related
univariate polynomial Q*.
3. Show that Q* is nonzero and deg(Q*) · K-1
Proof of Expansion: Step 1
Thm: For A=q-nmh, K= hm,
|T|·AK-1) |LIST(T)|· K-1.
Step 1: Find a low-degree poly Q vanishing on T µ Fqm+1
•
Take Q(Y,Z1,…,Zm) to be of degree · A-1 in Y,
degree · h-1 in each Zi.
•
# coefficients = A K > |T| = # homogeneous constraints,
so a nonzero solution exists
•
Wlog E(Y) doesn’t divide Q(Y,Z1,…,Zm).
Proof of Expansion: Step 2
(f,y) = (y, f(y), (fh mod E)(y), …, (fh
m-1
mod E)(y))
Step 1: 9 Q(Y,Z1,…,Zm) vanishing on T, deg · A-1 in Y, h-1
in Zi, E-Q
Degree ≤ A-1+nmh < q ≤ # roots
Step 2: Every f 2LIST(T) is a “root” of a related Q*
Polynomial f 2 LIST(T)
)
)
)
)
m-1
8 y2 Fq Q(y, f(y), (fh mod E)(y), …, (fh mod E)(y)) = 0
m-1
Q(Y, f(Y), (fh mod E)(Y), …, (fh mod E)(Y))  0
m-1
Q(Y, f(Y), f(Y)h, …, f(Y)h )  0 (mod E(Y))
Q*(f) = 0 in extension field U=Fq[Y]/(E(Y)), where Q*
m-1
U[Z] is given by
Q*(Z) = Q(Y,Z,Zh,…,Zh ) mod
E(Y)
Proof of Expansion: Step 3
Step 2: 8 f2LIST(T) Q*(f) = 0 where
m-1
Q*(Z) = Q(Y,Z,Zh,…,Zh ) mod E(Y)
Step 3: Show that Q* is nonzero and
•
deg(Q*) · K-1
Q*(Z) nonzero because
– Q(Y,Z1,….,Zm) mod E(Y) is nonzero
– Q is of deg · h-1 in Zi so distinct monomals get
i
h
mapped to distinct powers of Z when we set Zi = Z
•
deg(Q*) · h-1+(h-1)¢ h++(h-1)¢ hm-1 = hm-1 = K-1
Proof of Expansion: Wrap-Up
(f,y) = (y, f(y), (fh mod E)(y), …, (fh
LIST(T) = { x2 [N] : (x)µT }
Theorem: For A = q - nmh, K= hm,
m-1
mod E)(y))
|T|·AK-1) |LIST(T)|· K-1.
There is a nonzero polynomial Q* over U=Fq[Y]/(E(Y)) with
deg(Q*) · K - 1 such that every f  LIST(T) satisfies Q*(f) = 0.
Hence
|LIST(T)| · deg(Q*) · K - 1.
¥
Parameter Choices
LHS = Fqn , degree D = q, RHS = Fqm+1
We have a (K,A) expander with K = hm, A = q - nmh
To make A  (1-)¢ D, pick q  nmh/.
To make M ¼ KD, need qm+1 ¼ q hm, so take q ¼ h1+
Set h ¼ (nm/)1/, q ¼ h1+ . Then:
• A = q - nmh  (1- ) q = (1-)¢ D
• M = qm+1 ¼ q¢ h(1+)m ¼ D¢ K1+
• D = (nm/)1+1/ ¼ ((log N)(log K)/)1+1/
Our Expander Result
N
M
“(K,A) expander”
S, |S| K
D
|(S)|  A¢|S|
Thm: For every N, K, , >0, 9 explicit (K,A) expander with
• degree D = O((log N)¢ (log K)/)1+1/
• expansion A = (1-)¢D
• #right vertices M = (D¢K)1+
Outline
 Unbalanced expanders
 Expander Construction from Parvaresh-Vardy codes
• View as condensers and application to Extractors
• Conclusions
Extractors [NZ’93]
Uniform sample from unknown
subset X  {0,1}n of size 2k
“seed”
EXT
d random bits
m almost-uniform bits
• Goal: Output -close to uniform on {0,1}m (for large m and small d)
• Optimal (nonconstructive):
d = log n + 2 log(1/) + O(1)
m = (k+d) - 2 log(1/) - O(1)
Extractors: Original Motivation
• Randomization is pervasive in CS
– Algorithm design, cryptography, distributed computing, …
• Typically assume perfect random source.
– Unbiased, independent random bits
– Unrealistic?
• Can we use a “weak” random source?
– Source of biased & correlated bits.
– More realistic model of physical sources.
• (Randomness) Extractors: convert a weak random
source into an almost-perfect random source.
• Dozens of constructions over 15+ years
Extractors: many “extraneous” uses…
• Derandomization of (poly-time/log-space) algorithms
[Sip88,NZ93,INW94, GZ97,RR99, MV99,STV99,GW02]
• Distributed & Network Algorithms
[WZ95,Zuc97,RZ98,Ind02].
• Hardness of Approximation [Zuc93,Uma99,MU01,Zuc06]
• Data Structures [Ta02]
• Cryptography [BBR85,HILL89,CDHKS00,Lu02,DRS04,NV04]
• List decodable codes [TZ01,Gur04]
• Metric Embeddings [Ind06]
• Compressed sensing [Ind07]
[GUV] Result on Extractors
Optimal up to
constant factors
Thm: For every n, k, >0, 9 explicit (k,) extractor with seed
length d=O(log n + log (1/)) and output length m=.99k.
• Previously achieved by [LRVW03]
– Only worked for  ¸ 1/no(1)
– Complicated recursive construction
Expanders & Lossless Condensers
{0,1}n
2k
x
n-bit source with entropy k
2d
y
d-bit
seed
{0,1}m
(x,y)
¸ (1-) 2d¢ 2k
COND
m ¼ 1.01k bit
source with entropy
(k+d)
Lemma [TUZ01]:  : {0,1}n £{0,1}d ! {0,1}m is a
lossless ((n,k) ! (m,k+d)) condenser if
graph is a (2k,(1-)¢2d) expander.
Proof: Expansion ) can make 1-1 by moving  fraction of edges
Extractor
• Using PV code, we have compressed the n bit source to
1.01k bits while retaining all the entropy (using O(log n)
bit seed)
m-1
– Cond(f,y) = (y, f(y), (fh mod E)(y), …, (fh mod E)(y))
• Now extract 0.99k bits from the 1.01k bit source with
entropy k
– Easier, specialized task (due to high entropy percentage)
– Good constructions already known
• For constant error , can use a simple random walk based extractor
– Compose with our condenser to get final extractor
Extractor for high min-entropy
Extractor for min-entropy rate 99% that extracts 99% of the
input min-entropy with constant error:
2c-degree expander on 2(1-)n nodes
Ext(x,y) = y’th vertex on expander walk
specified by x
(n bit source: specify walk of length n/c)
Extraction follows from Chernoff bound for
expander walks [Gil98]
Variation on the Condenser
Cond(f,y) = (y, f(y), (fh mod E)(y), …, (fh
m-1
mod E)(y))
f(Y)q = f(Yq)  f(Y) mod E(Y)
• Use E(Y) = Yq-1 - , for generator  of Fq* [G.-Rudra’06]
i
) (fq mod E)(y) = f (i y)
Cond(f,y) = (y, f(y), f (γy), f(γ2y)…, f(γm-1y))
• Condenser from Folded Reed-Solomon code [GR06]
– Loses small constant fraction of min-entropy
• Okay for the extractor application
– Univariate analogue of Shaltiel-Umans extractor
Conclusions
• List decoding view + an algebraic code construction )
best known constructions of
– Highly unbalanced expanders
– Lossless condensers
– Randomness extractors
• Future directions?
– Constant degree lossless expanders (alternative to zig-zag)
• Non-bipartite expanders?
– Direct construction of a simple, algebraic extractor
– Extractors with better (or even optimal) entropy loss?
• Suffices to achieve this for entropy rate 0.999
– Other pseudorandom objects: multi-source extractors?