Generative adversarial
networks
Ian
Goodfellow
Jean
Pouget-Abadie
Bing
Xu
David
Warde-Farley
Aaron
Courville
Mehdi
Mirza
Sherjil
Ozair
Yoshua
Bengio
1
Discriminative deep learning
• Recipe for success
x
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
2
Discriminative deep learning
• Recipe for success:
Google’s winning entry
into the ImageNet 1K
competition (with extra data).
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
Figure 3: GoogLeNet network with all the bells and whistles
7
3
Discriminative deep learning
• Recipe for success:
- Gradient backpropagation.
- Dropout
- Activation functions:
• rectified linear
• maxout
Google’s winning entry
into the ImageNet 1K
competition (with extra data).
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
Figure 3: GoogLeNet network with all the bells and whistles
7
4
Generative modeling
• Have training examples x ~ pdata(x )
• Want a model that can draw samples: x ~
pmodel(x )
• Where pmodel ≈ pdata
x ~ pdata(x )
x ~ pmodel(x )
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
5
Why generative models?
• Conditional generative models
- Speech synthesis: Text
Speech
- Machine Translation: French
•
•
English
French: Si mon tonton tond ton tonton, ton tonton sera tondu.
English: If my uncle shaves your uncle, your uncle will be shaved
- Image
Image segmentation
• Environment simulator
- Reinforcement learning
- Planning
• Leverage unlabeled data
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
6
Maximum likelihood: the dominant approach
• ML objective function
m
⇣
⌘
X
1
⇤
(i)
✓ = max
log p x ; ✓
✓ m
i=1
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
7
Undirected graphical models
• State-of-the-art general purpose undirected
graphical model: Deep Boltzmann machines
• Several “hidden layers” h
1
p(h, x) = p̃(h, x)
Z
p̃(h, x) = exp( E(h, x))
Z=
p̃(h, x)
h(3)
h(2)
h(1)
x
h,x
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
8
Undirected
graphical
p̃(h, x) = exp
( E (h, x)) models: disadvantage
X
Z=
p̃(h, x)
• ML Learning
requires
that we draw samples:
h,x
h(3)
"
d
d
log p(x) =
log
d✓i
d✓i
X
p̃(h, x)
log Z(✓)
h
d
log Z(✓) =
d✓i
#
h(2)
h(1)
x
d
d✓i Z(✓)
Z(✓)
• Common way to do this is via MCMC (Gibbs sampling).
h) = p(x | h(1) )p(h(1) | h(2) ) . . . p(h(L
1)
| h(L) )p(h(L) )
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
9
Boltzmann Machines: disadvantage
• Model is badly parameterized for learning high
quality samples.
• Why?
- Learning leads to large values of the model parameters.
‣ Large valued parameters = peaky distribution.
- Large valued parameters means slow mixing of sampler.
- Slow mixing means that the gradient updates are
correlated leads to divergence of learning.
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
10
Boltzmann Machines: disadvantage
• Model is badly parameterized for learning high
quality samples.
• Why poor mixing?
Coordinated
flipping of lowlevel features
MNIST dataset
1st layer features (RBM)
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
11
d
log Z(✓) =
d✓i
d
d✓i Z(✓)
Z(✓)
Directed graphical models
p(x, h) = p(x | h(1) )p(h(1) | h(2) ) . . . p(h(L
d
1 d
log p(x) =
p(x)
d i
p(x) d i
p(x) =
h
p(x | h)p(h)
1)
| h(L) )p(h(L) )
h(3)
h(2)
h(1)
x
• Two problems:
1. Summation over exponentially many states in h
2. Posterior inference, i.e. calculating p(h | x), is intractable.
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
12
Directed graphical models: New approaches
• The Variational Autoencoder model:
-
Kingma and Welling, Auto-Encoding Variational Bayes, International
Conference on Learning Representations (ICLR) 2014.
-
Rezende, Mohamed and Wierstra, Stochastic back-propagation and
variational inference in deep latent Gaussian models. ArXiv.
-
Use a reparametrization that allows them to train very efficiently
with gradient backpropagation.
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
13
Generative stochastic networks
• General strategy: Do not write a formula for p(x),
just learn to sample incrementally.
...
• Main issue: Subject to some of the same constraints
on mixing as undirected graphical models.
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
14
Generative adversarial networks
• Don’t write a formula for p(x), just learn to sample
directly.
• No summation over all states.
• How? By playing a game.
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
15
Two-player zero-sum game
• Your winnings + your opponent’s winnings = 0
• Minimax theorem: a rational strategy exists for all
such finite games
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
16
Two-player zero-sum game
• Strategy: specification of which moves you make in which
circumstances.
• Equilibrium: each player’s strategy is the best possible for
their opponent’s strategy.
Your opponent
Rock Paper Scissors
You
- Choose you action at random
Rock
- Mixed strategy equilibrium
0
-1
1
Scissors Paper
• Example: Rock-paper-scissors:
1
0
-1
-1
1
0
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
17
Generative modeling with game theory?
• Can we design a game with a mixed-strategy
equilibrium that forces one player to learn to generate
from the data distribution?
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
18
Adversarial nets framework
• A game between two players:
1. Discriminator D
2. Generator G
• D tries to discriminate between:
- A sample from the data distribution.
- And a sample from the generator G.
• G tries to “trick” D by generating samples that are
hard for D to distinguish from data.
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
19
Adversarial nets framework
x
D tries to
output 1
D tries to
output 0
Differentiable
function D
Differentiable
function D
x sampled
from data
x sampled
from model
x
Differentiable
function G
Input noise
Z
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
z
20
Zero-sum game
er words,
D and G play
the following
two-player minimax game with value function V
• Minimax
objective
function:
min max V (D, G) = Ex⇠pdata (x) [log D(x)] + Ez⇠pz (z) [log(1
G
D
D(G(z)))].
next section, we present a theoretical analysis of adversarial nets, essentially show
ining criterion allows one to recover the data generating distribution as G and D
h capacity,
in the non-parametric
Seeuse:
Figure 1 for a less formal, more ped
• In i.e.,
practice,
to estimatelimit.
G we
ation of the approach. In practice, we must implement the game using an iterative, n
ach. Optimizing D to completion in the inner loop of training is computationally pro
maxinEoverfitting.
D(G(z))]
z∼pz (z) [log
finite datasets would result
Instead,
we alternate between k steps of op
G
one step of optimizing G. This results in D being maintained near its optimal sol
Why?
Stronger
forisGanalogous
when D
is very
good.
s G changes
slowly
enough. gradient
This strategy
to the
way that
SML/PCD
g maintains samples from a Markov chain from one learning step to the next in order
g in a Markov chain as part of the inner loop of learning. The procedure is formally p
orithm
1.Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
2014 NIPS
21
h
Discriminator strategy
• Optimal strategy for any pmodel(x) is always
pdata (x)
D(x) =
pdata (x) + pmodel (x)
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
22
burning in a Markov chain as part of the inner loop of learning. The procedure is formally presented
in Algorithm 1.
Learning process
In practice, equation 1 may not provide sufficient gradient for G to learn well. Early in learning,
when G is poor, D can reject samples with high confidence because they are clearly different from
the training data. In this case, log(1 D(G(z))) saturates. Rather than training G to minimize
log(1 D(G(z))) we can train G to maximize log D(G(z)). This objective function results in the
same
point of the Data
dynamics
of G and D but provides much stronger gradients early in learning.
pDfixed
(data)
distribution
Model distribution
...
D After updating
G
Poorly fit(a)model After updating
(b)
(c)
Mixed (d)
strategy
equilibrium
Figure 1: Generative adversarial nets are trained by simultaneously updating the discriminative distribution
(D, blue, dashed line) so that it discriminates between samples from the data generating distribution (black,
dotted line) px from those of the generative distribution pg (G) (green, solid line). The lower horizontal line is
the domain from which z is sampled, in this case uniformly. The horizontal line above is part of the domain
of x. The upward arrows show how the mapping x = G(z) imposes the non-uniform distribution pg on
transformed samples. G contracts in regions of high density and expands in regions of low density of pg . (a)
Consider an adversarial pair near convergence: pg is similar to pdata and D is a partially accurate classifier.
(b) In
theWorkshop
inner loop
the algorithm
D is trained
discriminate
samples from data, converging to D⇤ (x) =23
2014
NIPS
onof
Perturbations,
Optimization,
andto
Statistics
--- Ian Goodfellow
pdata (x)
burning in a Markov chain as part of the inner loop of learning. The procedure is formally presented
in Algorithm 1.
Learning process
In practice, equation 1 may not provide sufficient gradient for G to learn well. Early in learning,
when G is poor, D can reject samples with high confidence because they are clearly different from
the training data. In this case, log(1 D(G(z))) saturates. Rather than training G to minimize
log(1 D(G(z))) we can train G to maximize log D(G(z)). This objective function results in the
same
point of the Data
dynamics
of G and D but provides much stronger gradients early in learning.
pDfixed
(data)
distribution
Model distribution
...
D After updating
G
Poorly fit(a)model After updating
(b)
(c)
Mixed (d)
strategy
equilibrium
Figure 1: Generative adversarial nets are trained by simultaneously updating the discriminative distribution
(D, blue, dashed line) so that it discriminates between samples from the data generating distribution (black,
dotted line) px from those of the generative distribution pg (G) (green, solid line). The lower horizontal line is
the domain from which z is sampled, in this case uniformly. The horizontal line above is part of the domain
of x. The upward arrows show how the mapping x = G(z) imposes the non-uniform distribution pg on
transformed samples. G contracts in regions of high density and expands in regions of low density of pg . (a)
Consider an adversarial pair near convergence: pg is similar to pdata and D is a partially accurate classifier.
(b) In
theWorkshop
inner loop
the algorithm
D is trained
discriminate
samples from data, converging to D⇤ (x) =24
2014
NIPS
onof
Perturbations,
Optimization,
andto
Statistics
--- Ian Goodfellow
pdata (x)
burning in a Markov chain as part of the inner loop of learning. The procedure is formally presented
in Algorithm 1.
Learning process
In practice, equation 1 may not provide sufficient gradient for G to learn well. Early in learning,
when G is poor, D can reject samples with high confidence because they are clearly different from
the training data. In this case, log(1 D(G(z))) saturates. Rather than training G to minimize
log(1 D(G(z))) we can train G to maximize log D(G(z)). This objective function results in the
same
point of the Data
dynamics
of G and D but provides much stronger gradients early in learning.
pDfixed
(data)
distribution
Model distribution
...
D After updating
G
Poorly fit(a)model After updating
(b)
(c)
Mixed (d)
strategy
equilibrium
Figure 1: Generative adversarial nets are trained by simultaneously updating the discriminative distribution
(D, blue, dashed line) so that it discriminates between samples from the data generating distribution (black,
dotted line) px from those of the generative distribution pg (G) (green, solid line). The lower horizontal line is
the domain from which z is sampled, in this case uniformly. The horizontal line above is part of the domain
of x. The upward arrows show how the mapping x = G(z) imposes the non-uniform distribution pg on
transformed samples. G contracts in regions of high density and expands in regions of low density of pg . (a)
Consider an adversarial pair near convergence: pg is similar to pdata and D is a partially accurate classifier.
(b) In
theWorkshop
inner loop
the algorithm
D is trained
discriminate
samples from data, converging to D⇤ (x) =25
2014
NIPS
onof
Perturbations,
Optimization,
andto
Statistics
--- Ian Goodfellow
pdata (x)
burning in a Markov chain as part of the inner loop of learning. The procedure is formally presented
in Algorithm 1.
Learning process
In practice, equation 1 may not provide sufficient gradient for G to learn well. Early in learning,
when G is poor, D can reject samples with high confidence because they are clearly different from
the training data. In this case, log(1 D(G(z))) saturates. Rather than training G to minimize
log(1 D(G(z))) we can train G to maximize log D(G(z)). This objective function results in the
same
point of the Data
dynamics
of G and D but provides much stronger gradients early in learning.
pDfixed
(data)
distribution
Model distribution
...
D After updating
G
Poorly fit(a)model After updating
(b)
(c)
Mixed (d)
strategy
equilibrium
Figure 1: Generative adversarial nets are trained by simultaneously updating the discriminative distribution
(D, blue, dashed line) so that it discriminates between samples from the data generating distribution (black,
dotted line) px from those of the generative distribution pg (G) (green, solid line). The lower horizontal line is
the domain from which z is sampled, in this case uniformly. The horizontal line above is part of the domain
of x. The upward arrows show how the mapping x = G(z) imposes the non-uniform distribution pg on
transformed samples. G contracts in regions of high density and expands in regions of low density of pg . (a)
Consider an adversarial pair near convergence: pg is similar to pdata and D is a partially accurate classifier.
(b) In
theWorkshop
inner loop
the algorithm
D is trained
discriminate
samples from data, converging to D⇤ (x) =26
2014
NIPS
onof
Perturbations,
Optimization,
andto
Statistics
--- Ian Goodfellow
pdata (x)
Theoretical properties
er words, D and G play the following two-player minimax game with value function V
min max V (D, G) = Ex⇠pdata (x) [log D(x)] + Ez⇠pz (z) [log(1
G
D
D(G(z)))].
next•section,
we presentproperties
a theoretical analysis
of adversarial
Theoretical
(assuming
infinite nets,
data,essentially
infinite show
ining criterion allows one to recover the data generating distribution as G and D
model
direct limit.
updating
of generator’s
h capacity,
i.e., in capacity,
the non-parametric
See Figure
1 for a less formal, more ped
ation of distribution):
the approach. In practice, we must implement the game using an iterative, n
ach. Optimizing D to completion in the inner loop of training is computationally pro
finite datasets
wouldglobal
result inoptimum.
overfitting. Instead, we alternate between k steps of op
- Unique
one step of optimizing G. This results in D being maintained near its optimal sol
s G changes
slowly enough.
This strategy
is analogous
to the way that SML/PCD
- Optimum
corresponds
to data
distribution.
g maintains samples from a Markov chain from one learning step to the next in order
g in a Markov
chain as part of
inner loopguaranteed.
of learning. The procedure is formally p
- Convergence
tothe
optimum
orithm 1.
ctice, equation 1 may not provide sufficient gradient for G to learn well. Early in
G is poor, D can reject samples with high confidence because they are clearly differ
ining data. In this case, log(1 D(G(z))) saturates. Rather than training G to m
2014
NIPS Workshop
Perturbations,
and Statistics
--- Ian Goodfellow
27
D(G(z)))
weoncan
train GOptimization,
to maximize
log D(G(z)).
This objective function resu
Quantitative likelihood results
• Parzen window-based log-likelihood estimates.
- Density estimate with Gaussian kernels centered on
the samples drawn from the model.
Model
DBN [3]
Stacked CAE [3]
Deep GSN [6]
Adversarial nets
MNIST
138 ± 2
121 ± 1.6
214 ± 1.1
225 ± 2
TFD
1909 ± 66
2110 ± 50
1890 ± 29
2057 ± 26
indow-based log-likelihood estimates. The reported numbers on MNIST
les on test set, with the standard error of the mean computed across exam
dard
error
across
folds of
the dataset,
a different chosen using28th
2014 NIPS
Workshop
on Perturbations,
Optimization,
and Statisticswith
--- Ian Goodfellow
Visualization of model samples
MNIST
TFD
CIFAR-10 (fully connected)
CIFAR-10 (convolutional)
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
29
Learned 2-D manifold of MNIST
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
30
Visualizing trajectories
1. Draw sample (A)
B
2. Draw sample (B)
3. Simulate samples
along the path
between A and B
4. Repeat steps 1-3 as
desired.
A
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
31
Visualization of model trajectories
MNIST digit dataset
Toronto Face Dataset (TFD)
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
32
Visualization of model trajectories
CIFAR-10
(convolutional)
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
33
Extensions
• Conditional model:
- Learn p(x | y)
- Discriminator is trained on (x,y) pairs
- Generator net gets y and z as input
- Useful for: Translation, speech synth, image
segmentation.
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
34
Extensions
• Inference net:
- Learn a network to model p(z | x)
- Infinite training set!
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
35
Extensions
• Take advantage of high amounts of unlabeled data
using the generator.
• Train G on a large, unlabeled dataset
• Train G’ to learn p(z|x) on an infinite training set
• Add a layer on top of G’, train on a small labeled
training set
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
36
Extensions
• Take advantage of unlabeled data using the
discriminator
• Train G and D on a large amount of unlabeled data
- Replace the last layer of D
- Continue training D on a small amount of labeled
data
2014 NIPS Workshop on Perturbations, Optimization, and Statistics --- Ian Goodfellow
37
Thank You.
Questions?
38