Published as a conference paper at ICLR 2022
P SEUDO N UMERICAL M ETHODS
M ODELS ON M ANIFOLDS
FOR
D IFFUSION
Luping Liu, Yi Ren, Zhijie Lin & Zhou Zhao∗
Zhejiang University
{luping.liu,rayeren,linzhijie,zhaozhou}@zju.edu.cn
arXiv:2202.09778v2 [cs.CV] 31 Oct 2022
A BSTRACT
Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality
samples such as image and audio samples. However, DDPMs require hundreds
to thousands of iterations to produce final samples. Several prior works have
successfully accelerated DDPMs through adjusting the variance schedule (e.g.,
Improved Denoising Diffusion Probabilistic Models) or the denoising equation
(e.g., Denoising Diffusion Implicit Models (DDIMs)). However, these acceleration methods cannot maintain the quality of samples and even introduce new
noise at a high speedup rate, which limit their practicability. To accelerate the
inference process while keeping the sample quality, we provide a fresh perspective that DDPMs should be treated as solving differential equations on manifolds.
Under such a perspective, we propose pseudo numerical methods for diffusion
models (PNDMs). Specifically, we figure out how to solve differential equations
on manifolds and show that DDIMs are simple cases of pseudo numerical methods. We change several classical numerical methods to corresponding pseudo
numerical methods and find that the pseudo linear multi-step method is the best
in most situations. According to our experiments, by directly using pre-trained
models on Cifar10, CelebA and LSUN, PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup),
significantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have
good generalization on different variance schedules.1
1
I NTRODUCTION
Denoising Diffusion Probabilistic Models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020)
is a class of generative models which model the data distribution through an iterative denoising
process reversing a multi-step noising process. DDPMs have been applied successfully to a variety
of applications, including image generation (Ho et al., 2020; Song et al., 2020b), text generation
(Hoogeboom et al., 2021; Austin et al., 2021), 3D point cloud generation (Luo & Hu, 2021), textto-speech (Kong et al., 2021; Chen et al., 2020) and image super-resolution (Saharia et al., 2021).
Unlike Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which require careful
hyperparameter tuning according to different model structures and datasets, DDPMs can use similar
model structures and be trained by a simple denoising objective which makes the models fit the
noise in the data. To generate samples, the iterative denoising process starts from white noise and
progressively denoises it into the target domain according to the noise predicted by the model at
every step. However, a critical drawback of DDPMs is that DDPMs require hundreds to thousands
of iterations to produce high-quality samples and need to pass through a network at least once at
every step, which makes the generation of a large number of samples extremely slow and infeasible.
In contrast, GANs only need one pass through a network.
There have been many recent works focusing on improving the speed of the denoising process.
Some works search for better variance schedules, including Nichol & Dhariwal (2021) and Watson
et al. (2021). Some works focus on changing the inference equation, including Song et al. (2020a)
∗
1
Corresponding author
Our implementation is available at https://github.com/luping-liu/PNDM.
1
Published as a conference paper at ICLR 2022
and Song et al. (2020b). Denoising Diffusion Implicit Models (DDIMs) (Song et al., 2020a) relying
on a non-Markovian process accelerate the denoising process by taking multiple steps every iteration. Probability Flows (PFs) (Song et al., 2020b) build a connection between the denoising process
and solving ordinary differential equations and use numerical methods of differential equations to
accelerate the denoising process. Additionally, we introduce more related works in Appendix A.1.
However, this direct connection between DDPMs and numerical methods (e.g., forward Euler method, linear multi-step method and RungeKutta method (Timothy, 2017)) has
weaknesses in both speed and effect
(see Section 3.1). Some numerical
methods are straightforward, like the
forward Euler method, but they can
only trade quality for speed. Some
numerical methods can accelerate the
reverse process without loss of qual- Figure 1: 5, 10, 20, 50 and 100-steps generated results using
ity, like the Runge-Kutta method, but DDIMs, classical numerical methods and PNDMs.
they need to propagate forward more times along a neural network at every step. Furthermore, we
also notice that numerical methods can introduce noticeable noise at a high speedup rate, which
makes high-order numerical methods (e.g., Runge-Kutta method) even less effective than DDIMs.
This phenomenon is also mentioned in Salimans & Ho (2022).
To figure out the reason for the performance degradation in classical numerical methods, we conduct
some analyses and find that classical numerical methods may sample data far away from the main
distribution area of the data, and the inference equations of DDPMs do not satisfy a necessary
condition of numerical methods at the last several steps (see Section 3.2).
To tackle these problems, we design new numerical methods called pseudo numerical methods for
diffusion models (PNDMs) to generate samples along a specific manifold in Rn , which is the highdensity region of the data. We first compute the corresponding differential equations of diffusion
models directly and self-consistently, which builds a theoretical connection between DDPMs and
numerical methods. Considering that classical numerical methods cannot guarantee to generate
samples on certain manifolds, we provide brand-new numerical methods called pseudo numerical
methods based on our theoretical analyses. We also find that DDIMs are simple cases of pseudo
numerical methods, which means that we also provide a new way to understand DDIMs better.
Furthermore, we find that the pseudo linear multi-step method is the fastest method for diffusion
models under similar generated quality.
Besides, we provide a detailed theoretical analysis of our new theory and give visualization results to
support our theory intuitively. According to our experiments, our methods have several advantages:
• Our methods combine the benefits of DDIMs and high-order numerical methods successfully. We
theoretically prove that our new methods PNDMs are second-order convergent while DDIMs are
first-order convergent, which makes PNDMs 20x faster without loss of quality on Cifar10 and
CelebA.
• Our methods can reduce the best FID of pre-trained models with even shorter sampling time. With
only 250 steps, our new denoising process can reduce the best FID by around 0.4 points Cifar10
and CelebA. We achieve a new SOTA FID score of 2.71 on CelebA.
• Our methods work well with different variance schedules, which means that our methods have a
good generalization and can be used together with those works introducing better variance schedules to accelerate the denoising process further.
2
BACKGROUND
In this section, we introduce some backgrounds. Firstly, we present the classical understanding of
DDPMs. Then we provide another understanding based on Song et al. (2020b), which inspires us
to use numerical methods to accelerate the denoising process of diffusion models. After that, we
introduce some background on numerical methods used later in this paper.
2
Published as a conference paper at ICLR 2022
2.1
D ENOISING D IFFUSION P ROBABILISTIC M ODELS
DDPMs model the data distribution from Gaussian distribution to image distribution through an
iterative denoising process. Let x0 be an image, then the diffusion process is a Markov process and
the reverse process has a similar form to the diffusion process, which satisfies:
p
xt+1 ∼N ( 1 − βt xt , βt I), t = 0, 1, · · · , N − 1.
(1)
xt−1 ∼N (µθ (xt , t), βθ (xt , t)I), t = N, N − 1, · · · , 1.
Here, βt controls the speed of adding noise to the data, calling them the variance schedule. N is the
total number of steps of the denoising process. µθ and βθ are two neural networks, and θ are their
parameters.
Ho et al. (2020) get some statistics estimations of µθ and βθ . According to the properties of the
conditional Gaussian distribution, we have:
√
q(xt |x0 ) = N ( ᾱt x0 , (1 − ᾱt )I),
(2)
q(xt−1 |xt , x0 ) = N (µ̄t (xt , x0 ), β̄t I).
√
√
Qt
αt (1−ᾱt−1 )
ᾱ
βt
ᾱt−1
Here, αt = 1 − βt , ᾱt = i=1 αi , µ̄t = 1−t−1
xt and β̄t = 1−
ᾱt x0 +
1−ᾱt
1−ᾱt βt . Then
this paper sets βθ = β̄t and designs a objective function to help neural networks to represent µθ .
Objective Function The objective function is defined by:
Lt−1 = Eq ||µ̄t (xt , x0 ) − µθ (xt , t)||2
βt
1
2
− µθ (xt (x0 , ), t)||
= Ex0 , || √
xt (x0 , ) − √
(3)
αt
1 − ᾱt
2
√
√
βt
= Ex0 ,
|| − θ ( ᾱt x0 + 1 − ᾱt , t)||2 .
αt (1 − ᾱt )
√
√
Here, xt (x0 , ) = ᾱt x0 + 1 − ᾱt , ∼ N (0, 1), θ is an estimate of the noise . The relationship
βt
). Because ∼ N (0, 1), we assume that the mean
between µθ and θ is µθ = √1α (xt − √1−
ᾱt θ
t
and variance of θ are 0 and 1.
2.2
S TOCHASTIC D IFFERENTIAL E QUATION
According to Song et al. (2020b), there is another understanding of DDPMs.
p The diffusion process
can
be
treated
as
solving
a
certain
stochastic
differential
equation
dx
=
(
1 − β(t) − 1)x(t)dt +
p
β(t)dw. According to Anderson (1982), the denoising process also satisfies a similar stochastic
differential equation:
p
p
dx = ( 1 − β(t) − 1)x(t) − β(t)θ (x(t), t) dt + β(t)dw̄.
(4)
This is Variance Preserving stochastic differential equations (VP-SDEs). Here, we change the doN
main of t from [1, N ] to [0, 1]. When N tends to infinity, {βi }N
i=1 , {xi }i=1 become continuous
functions β(t) and x(t) on [0, 1]. Song et al. (2020b) also show that this equation has an ordinary
differential equation (ODE) version with the same marginal probability density as Equation (4):
p
1
dx = ( 1 − β(t) − 1)x(t) − β(t)θ (x(t), t) dt.
(5)
2
This different denoising equation with no random item and the same diffusion equation together is
Probability Flows (PFs). These two denoising equations show us a new possibility that we can use
numerical methods to accelerate the reverse process. As far as we know, DDIMs first try to remove
this random item, so PFs can also be treated as an acceleration of DDIMs, while VP-SDEs are an
acceleration of DDPMs.
2.3
N UMERICAL M ETHOD
Many classical numerical methods can be used to solve ODEs, including the forward Euler method,
Runge-Kutta method and linear multi-step method (Timothy, 2017).
3
Published as a conference paper at ICLR 2022
Forward Euler Method For a certain differential equation satisfying dx
dt = f (x, t). The trivial
numerical method is forward Euler method satisfying xt+δ = xt + δf (xt , t).
Runge-Kutta Method Runge-Kutta method uses more information at every step, so it can achieve
higher accuracy 2 . Runge-Kutta method satisfies:

, k2 = f (xt + 2δ k1 , t + 2δ )
 k1 = f (xt , t)
δ
δ
(6)
k3 = f (xt + 2 k2 , t + 2 ) , k4 = f (xt + δk3 , t + δ)

δ
xt+δ = xt + 6 (k1 + 2k2 + 2k3 + k4 ).
Linear Multi-Step Method Linear multi-step method is another numerical method and satisfies:
xt+δ = xt +
3
δ
(55ft − 59ft−δ + 37ft−2δ − 9ft−3δ ), ft = f (xt , t).
24
(7)
P SEUDO N UMERICAL M ETHOD FOR DDPM
In this section, we first compute the corresponding differential equations of diffusion models to build
a direct connection between DDPMs and numerical methods. As a byproduct, we can directly use
pre-trained models from DDPMs. After establishing this connection, we provide detailed analyses
on the weakness of classical numerical methods. To solve the problems in classical numerical methods, we dive into the structure of numerical methods by dividing their equations into a gradient part
and a transfer part and define pseudo numerical methods by introducing nonlinear transfer parts. We
find that DDIMs can be regarded as simple pseudo numerical methods. Then, We explore the pros
and cons of different numerical methods and choose the linear multi-step method to make numerical methods faster. Finally, we summarize our findings and analyses and safely propose our novel
pseudo numerical methods for diffusion models (PNDMs), which combine our proposed transfer
part and the gradient part of the linear multi-step method. Furthermore, we analyze the convergence
order of pseudo numerical methods to demonstrate the effectiveness of our methods theoretically.
3.1
F ORMULA T RANSFORMATION
According to Song et al. (2020a), the reverse process of DDPMs and DDIMs satisfies:
√
q
√
xt − 1 − ᾱt θ (xt , t)
√
xt−1 = ᾱt−1
+ 1 − ᾱt−1 − σt2 θ (xt , t) + σt t .
ᾱt
(8)
Here, σt controls the ratio of random noise. If σt equals one, Equation (8) represents the reverse
process of DDPMs; if σt equals zero, this equation represents the reverse process of DDIMs. And
only when σt equals zero, this equation removes the random item and becomes a discrete form
of a certain ODE. Theoretically, the numerical methods that can be used on differential equations
with random items are limited. And Song et al. (2020b) have done enough research in this case.
Empirically, Song et al. (2020a) have shown that DDIMs have a better acceleration effect when the
number of total steps is relatively small. Therefore, our work concentrate on the case σt equals zero.
To find the corresponding ODE of Equation (8), we replace discrete t − 1 with a continuous version
t − δ according to (Song et al., 2020a) and change this equation into a differential form, namely,
subtract xt from both sides of this equation:
!
xt
θ (xt , t)
p
√
xt−δ − xt = (ᾱt−δ − ᾱt ) √ √
−√ p
.
ᾱt ( ᾱt−δ + ᾱt )
ᾱt ( (1 − ᾱt−δ )ᾱt + (1 − ᾱt )ᾱt−δ )
(9)
Because δ is a continuous variable from 0 tot, we can now computethe derivative of the generation
θ (x(t),t)
t−δ
√
data xt and get that lim xt −x
= −ᾱ0 (t) 2x(t)
. Here, ᾱ(t) is the continuous
δ
ᾱ(t) −
2ᾱ(t)
δ→0
1−ᾱ(t)
version of {ᾱi }N
i=1 like the definition of x(t). Therefore, the corresponding ODE when δ tends to
2
To achieve higher accuracy, more information is not enough. The reason why these methods achieve higher
accuracy can be found in Appendix A.2
4
Published as a conference paper at ICLR 2022
zero of Equation (9) is:
dx
= −ᾱ0 (t)
dt
3.2
θ (x(t), t)
x(t)
p
−
2ᾱ(t) 2ᾱ(t) 1 − ᾱ(t)
!
.
(10)
C LASSICAL N UMERICAL M ETHOD
After getting the target ODE, the easiest way to solve it is through classical numerical methods.
However, We notice that classical numerical methods can introduce noticeable noise at a high
speedup rate, making high-order numerical methods (e.g., Runge-Kutta method) even less effective than DDIMs. This phenomenon is also mentioned in Salimans & Ho (2022). To make better
use of numerical methods, we analyze the differences between Equation (10) and usual differential equations and find two main problems when we directly use numerical methods with diffusion
models.
norm
The first problem is that the neural network θ and Equation (10) are well-defined only in a limited area. Equation 1.0
(2) shows that the data xt is generated along a curve close 0.8
to an arc. According to Figure 2, most of xt is concentrated
in a band with a width of around 0.1, namely the red area 0.6
in Figure 2. This means that the neural network θ cannot get enough examples to fit the noise successfully away 0.4
from this area. Therefore, θ and Equation (10), which contains θ , are only well-defined in this limited area. How- 0.2
0
200
400
600
800
1000
ever, all classical numerical methods generate results along
step
a straight line instead of an arc. The generation process
may generate samples away from the well-defined area and Figure 2: the density distribution of
then introduce new errors. In Section 4.3 we will give more the norm of the data.
visualization results to support this.
The second problem is that Equation (10) is unbounded at most cases. We find that for most linear
variance schedules βt , Equation (10) tends to infinity when t tends to zero (see Appendix A.4), which
does not satisfy the condition of numerical methods mentioned in Section 2.3. This is an apparent
theoretical weakness that previous works have not explored. On the contrary, in the original DDPMs
and DDIMs, the prediction of the sample xt and the noise θ in the data are more and more precise
as the index t tends to zero (see Appendix A.5). This means original diffusion models do not make
a significant error in the last several steps, whereas using numerical methods on Equation (10) does.
This explains why DDIMs are better than higher-order numerical methods.
3.3
P SEUDO N UMERICAL M ETHOD ON M ANIFOLD
The first problem above shows that we should try to solve our problems on certain manifolds.
Here, target manifolds
are the high-density region of the data xt of DDPMs, which is defined by
√
√
xt (x0 , ) = ᾱt x0 + 1 − ᾱt , ∼ N (0, 1). Ernst & Gerhard (1996) show several numerical
methods to solve differential equations on manifolds that have analytic expressions. Unfortunately,
it’s challenging to use the above expression of manifolds. Because we do not know the target x0 in
the reverse process and random items are hard to handle, too.
In this paper, we design a different way that we make our new equation of denoising process more
fits with the equation of original DDIMs to make their results share similar data distribution. Firstly,
we divide the classical numerical methods into two parts: gradient and transfer parts. The gradient
part determines the gradient at each step, while the transfer part generates the result at the next step.
δ
(55ft −59ft−δ +
For example, linear multi-step method can be divided into the gradient part f 0 = 24
0
37ft−2δ − 9ft−3δ ) and the transfer part xt+δ = xt + δf . All classical numerical methods have the
same linear transfer part, while gradient parts are different.
We define those numerical methods which use a nonlinear transfer part as pseudo numerical methods. And an expected transfer part should have the property that when the result from the gradient
part is precise, then the result of the transfer part is as close to the manifold as possible and the error
of this result is as small as possible. We find that Equation (9) satisfies such property.
5
Published as a conference paper at ICLR 2022
Property 3.1 If is the precise noise in xt , then the result of xt−δ from Equation (9) is also precise.
And we put the proof of this property in Appendix A.5. Therefore, we use:
√
ᾱt−δ
(ᾱt−δ − ᾱt )
p
φ(xt , t , t, t − δ) = √
t
xt − √ p
ᾱt
ᾱt ( (1 − ᾱt−δ )ᾱt + (1 − ᾱt )ᾱt−δ )
(11)
as the transfer part and θ as the gradient part. That if θ is precise, the result of xt−δ is also precise,
which means that θ can determine the direction of the denoising process to generate the final results.
Therefore, such a choice also satisfies the definition of a gradient part. Now, we have our gradient
part θ and transfer part φ.
This combination solves the two problems mentioned above successfully. Firstly, our new transfer
parts do not introduce new errors. This property also means that it keeps the results at the next step
on the target manifold because generating samples away is a kind of error. This shows that we solve
the first problem. Secondly, we know that the prediction of θ is more and more precise in the reverse
process in the above subsection. And our new transfer part can generate precise results according
to the precise prediction of θ . Therefore, our generation results are more and more precise using
pseudo numerical methods, while classical numerical methods can introduce obvious error at the last
several steps. This shows that we solve the second problem, too. We also find that their combination
φ(xt , θ (xt , t), t, t − 1) is just the inference equation used by DDIMs, so DDIMs is a simple case
of pseudo numerical methods. Here, we define DDIMs as DDIMs*, emphasizing that it is a pseudo
numerical method.
3.4
G RADIENT PART
Because we split numerical methods into two parts, we can use the same gradient part from different
classical numerical methods freely (e.g., linear multi-step method), although we change the transfer
part of our inference equation. Our theoretical analyses and experiments show that the gradient part
from different classical methods can work well with our new transfer part (see Section 3.6, 4.2). By
using the same gradient part of the linear multi-step method, we have:

 et = θ (xt , t)
Algorithm 1 DDIMs
1
e0t = 24
(55et − 59et−δ + 37et−2δ − 9et−3δ )
1: xT ∼ N (0, I)

xt+δ = φ(xt , e0t , t, t + δ).
2: for t = T − 1, · · · , 1, 0 do
(12) 3: xt = φ(xt+1 , θ (xt+1 , t + 1), t + 1, t)
By using the same gradient part of Runge-Kutta 4: end for
5: return x0
method, we have:
 1
Algorithm 2 PNDMs
et = θ (xt , t)



δ
1
1

1: xT ∼ N (0, I)
=
φ(x
,
e
,
t,
t
+
x
)

t t
t
2


δ
2: for t = T − 1, T − 2, T − 3 do
2
1


et = θ (xt , t + 2 )


3:
xt , et = PRK(xt+1 , t + 1, t)

δ
2
2

 xt = φ(xt , et , t, t + 2 )
4: end for
(13) 5: for t = T − 4, · · · , 1, 0 do
e3t = θ (x2t , t + 2δ )


3
3
6:
xt , et = PLMS(xt+1 , {ep }p>t , t + 1, t)

xt = φ(xt , et , t, t + δ)


7: end for

4
4

et = θ (xt , t + δ)

8: return x0




e0t = 16 (e1t + 2e2t + 2e3t + e4t )


xt−δ = φ(xt , e0t , t, t + δ).
Abbreviate Equation (12) and (13) as xt+δ , et = PLMS(xt , {ep }p<t , t, t + δ), xt+δ , e1t =
PRK(xt , t, t + δ).
Here, we have provided three kinds of pseudo numerical methods. Although advanced numerical
methods can accelerate the denoising process, some may have to compute the gradient part θ more
times at every step, like the Runge-Kutta method. Propagating forward four times along a neural
network makes the denoising process slower. However, we find that the linear multi-step method can
reuse the result of θ four times and only compute θ once at every step. And theoretical analyses tell
us that the Runge-Kutta and linear multi-step method have the same convergence order and similar
results.
6
Published as a conference paper at ICLR 2022
Therefore, we use the gradient part of the linorder
first
non-first
ear multi-step method and our new transfer
φ
linear
forward Euler linear multi-step,
part as our main pseudo numerical methods
Runge-Kutta...
for diffusion models (PNDMs). In Table 1, we
nonlinear
DDIM
PNDM
show the relationship between different numerical methods. Here, we can see PNDMs
combine the benefits of higher-order classical Table 1: The relationship between different numernumerical methods (in the gradient part) and ical methods.
DDIMs (in the transfer part).
3.5
A LGORITHM
We can provide our whole algorithm of the denoising process of DDIMs now. According to Song
et al. (2020a), the algorithm of the original method satisfies Algorithm 1. And our new algorithm of
PNDMs uses the pseudo linear multi-step and pseudo Runge-Kutta method, which satisfies Algorithm 2. Here, we cannot use linear multi-step initially because the linear multi-step method cannot
start automatically, which needs at least three previous steps’ information to generate results. So
we use the Runge-Kutta method to compute the first three steps’ results and then use the linear
multi-step method to calculate the remaining.
We also use the gradient parts of two second-order numerical methods to get another pseudo numerical method. We introduce the details of this method in Appendix A.3. We call it S-PNDMs,
because its gradient part uses information from two steps at every step. Similarly, we also call our
first PNDMs F-PNDMs, which use data from four steps, when we need to distinguish them.
3.6
C ONVERGENCE O RDER
Change the transfer part of numerical methods may introduce unknown error. To determine the
influence of our new transfer part theoretically, we compute the local and global error between the
theoretical result of Equation (10) x(t + δ) and our new methods, we find that x(t + δ) − xDDIM (x +
δ) = O(δ 2 ) and
x(t + δ) − xS/F-PNDM (x + δ) = O(δ 3 ).
(14)
If the target ODE satisfies Lipschitz condition and local error elocal = O(δ k ), then there are C and
h such that the global error eglobal satisfies eglobal ≤ Cδ k (1 + eh + e2h + · · · ) ≤ C 0 δ k−1 . And we
have that the convergence order is equal to the order of the global error. The detailed proof can be
found in Appendix A.6. Therefore, we get the following property:
Property 3.2 S/F-PNDMs have third-order local error and are second-order convergent.
4
E XPERIMENT
4.1
S ETUP
We conduct unconditional image generation experiments on four datasets: Cifar10 (32 × 32)
(Krizhevsky et al., 2009), CelebA (64 × 64) (Liu et al., 2015), LSUN-church (256 × 256) and
LSUN-bedroom (256 × 256) (Yu et al., 2016). According to the analysis in Section 3.1, we can use
pre-trained models from prior works in our experiments. The pre-trained models for Cifar10, LSUNchurch and LSUN-bedroom are taken from Ho et al. (2020) and the pre-trained model for CelebA is
taken from Song et al. (2020a). In these models, the number of total steps N is 1000 and the variance
schedule is linear variance schedule. And we also use a pre-trained model for Cifar10, which uses a
cosine variance schedule from improved denoising diffusion probabilistic models (iDDPMs (Nichol
& Dhariwal, 2021)).
4.2
S AMPLE E FFICIENCY AND Q UALITY
To analyze the acceleration effect, we test Fenchel Inception Distance (FID (Heusel et al., 2018))
on different datasets under different steps and different numerical methods, including DDIMs,
S-PNDMs, F-PNDMs and classical fourth-order numerical methods (FONs) (e.g., Runge-Kutta
7
Published as a conference paper at ICLR 2022
FID
Table 2: Image generation
step
dataset
10
20
50
100
13.4
6.84
13.8
10.9
7.41
7.56
5.00
8.79
5.77
4.61
13.7
13.4
11.6
9.45
5.51
4.67
3.89
6.99
5.26
5.18
3.95
5.86
4.46
3.68
9.17
8.95
8.13
5.69
3.34
4.16
3.69
5.52
4.65
4.34
3.72
4.92
3.94
3.53
6.53
6.36
6.70
4.03
2.81
250
1000
time
model
Cifar10
Cifar10
(linear)
Cifar10
(cosine)
CelebA
CelebA
(linear)
DDIM
PF
DDIM*
FON
S-PNDM
F-PNDM
DDIM
S-PNDM
F-PNDM
DDIM
DDIM*
FON
S-PNDM
F-PNDM
18.5
13.1
11.6
7.03
14.5
8.64
7.05
17.3
16.9
16.0
12.2
7.71
3.71
4.52
4.12
3.91
3.60
4.30
3.71
3.49
4.44
5.14
3.19
2.71
4.04
3.72
4.00
3.71
3.80
3.70
3.69
3.38
3.26
3.51
3.41
4.17
2.99
2.86
0.337
0.390
0.344
0.391
0.505
0.517
0.595
1.237
1.431
1.258
1.433
measured in FID on Cifar10
and CelebA. PFs use black
box ODE solvers and we use
the number of score function evaluations as the step
of PFs. DDIM* is a retest
of DDIM. The bold results
mean the best ones using the
same pretrained model. We
use the 50-step, 512 batch
size experiment on an RTX3090 to test the computational cost and the column
time is the average computational cost per step in seconds. And we put the results
of standard deviation in Appendix A.12
method and linear multi-step method). On Cifar10 and CelebA, we first provide the results of previous works DDIMs. Then, we use the same pre-trained models to test numerical methods mentioned
in this paper and put the results in Cifar10 / CelebA (linear). We also use models from iDDPMs to
test nonlinear variance schedules and put the results in Cifar10 (cosine). Song et al. (2020b) do not
provide detailed FID results of probability flows (PFs) under different steps, so we retest the results
using its pretrained models by ourselves.
Efficiency Our two baselines are DDIM and PF. DDIM is a simple case of pseudo numerical methods, and PF is a case of classical numerical methods. However, PF uses a much bigger model than
DDIM and uses some tricks to improve the sample quality. To ensure the experiment’s fairness, we
use fourth-order numerical methods on Equation (10) and the model from DDIM. In Table 2, we
find that the performance of FON is limited when the number of steps is small. By contrast, our new
methods, including S-PNDM and F-PNDM, can improve the generated results regardless of whether
the number of steps used is large or small. According to Cifar10 / CelebA (linear), F-PNDM can
achieve lower FID than 1000 steps DDIM using only 50 steps, making diffusion models 20x faster
without losing quality.
9
8
DDIM
S-PNDM
F-PNDM
7
FID
We draw a line chart of computation cost with FID according to the results of Cifar10 (linear) above in Figure 3. Because F-PNDM uses the pseudo Runge-Kutta method to
generate the first three steps, it is slower than other methods
at the first several steps. Therefore, S-PNDM can achieve
the best FID initially, then F-PNDM becomes the best and
the acceleration is significant.
6
5
4
Quality When the number of steps is relatively big, the
101
102
103
results of FON become more and more similar to that of
time
pseudo numerical methods. This is because all the methods are solving Equation (10), and their convergent results Figure 3: The FID results under difshould be the same. However, pseudo numerical methods ferent computation costs and different
still work better using a large number of steps empirically. numerical methods on Cifar10. The
F-PNDM can improve the best FID around 0.4 using pre- unit of time is the computational cost
trained models and achieves a new SOTA FID score of 2.71 of 1-step DDIM, which is 0.337s.
on CelebA, which shows that our work can not only accelerate diffusion models but also improve
the sample quality topline. We also notice that the FID results of F-PNDM converge after more than
250 steps. The FID results will fluctuate around a value then. This phenomenon is more pronounced
when we test our methods on LSUN (see Table 5, 6).
According to Cifar10 (cosine), the cosine variance schedule can lower FID using a relatively large
number of steps. More analyses about variance schedule can be found in Appendix A.7. What’s
more, we test our methods on other datasets and provide our FID results in Appendix A.9 and
image results in Appendix A.10. We can draw similar conclusions on our methods’ acceleration and
sampling quality, regardless of the datasets and the size of the images.
8
1.0
1.0
0.8
0.8
0.8
0.6
0.4
0.2
0
200
400
step
600
DDIM-5
FON-5
PNDM-5
DDIM-1000
800
1000
norm
1.0
norm
norm
Published as a conference paper at ICLR 2022
0.6
0.4
DDIM-10
FON-10
PNDM-10
DDIM-1000
800
1000
0.2
0
200
400
step
600
0.4
600
DDIM-20
FON-20
PNDM-20
DDIM-1000
800
1000
0.4
0.3
0.2
0
200
400
step
0.05
0.00
0.10
0.05
0.10
0.10
0.15
0.20
0.25
DDIM-5
FON-5
PNDM-5
DDIM-1000
0.30
0.35
0.7
0.6
x2
0.15
0.15
x2
x2
0.6
0.20
0.30
0.5
0.4
x1
0.3
0.2
0.1
0.25
DDIM-10
FON-10
PNDM-10
DDIM-1000
0.25
0.7
0.6
0.20
DDIM-20
FON-20
PNDM-20
DDIM-1000
0.30
0.5
x1
0.4
0.3
0.2
0.7
0.6
0.5
x1
0.2
Figure 4: The upper part shows the change of norm with the number of steps using different methods
and different steps. The lower part shows the generation curves of two points using different methods
and different steps. DDIM-n means n-step DDIM method. Experiments in this subsection all use
the Cifar10 dataset and we use the 1000-step DDIM’s result as our target result.
4.3
S AMPLE ON M ANIFOLDS
Here, we design visualization experiments to show the effort of our new methods and support our
analyses. Because it is hard to visualize high-dimensional data, we use the change of a global characteristic norm and a local characteristic pixel to show the change of the data under different steps.
For pixel, we randomly choose two positions p1 , p2 . Then for a series of images xT , xT −k , · · · , x0
derived from the reverse process, we denote ytk as the value of xt at position pk . Then we draw a
polyline (yt1 , yt2 )t=T,··· in R2 . For norm, we first count the distribution of the norm of the training
datasets under different steps and use this to make a heat map as the background. After that, we
draw the norm of our generated results using different methods and steps above this heat map.
In Figure 4, we can see that the FON may run far away from the high-density area of the data, which
explains why FON may introduce noticeable noise. However, PNDM can avoid this problem and
appropriately fit the target result. More visualization results supporting our analysis can be found
in Appendix A.11. What’s more, we design a toy example to test our new methods without the
influence of neural networks and get similar conclusions as to the real cases above. We put the
detailed results in Appendix A.8.
5
D ISCUSSION
In this paper, we provided PNDMs, a new numerical method suitable for solving the corresponding
ODEs of DDPMs. PNDMs can generate high-quality images using fewer steps without loss of quality successfully. Based on the idea of this work, further improvement can be explored in our future
works: 1) find a better variance schedule for PNDMs: although we tested PNDMs on linear variance
schedule and cosine variance schedule in this work, there might be another variance schedule more
suitable for our proposed numerical methods. 2) Find higher-order convergent pseudo numerical
methods: we analyzed the convergence order of S/F-PNDMs, which are both second-order convergent. However, F-PNDMs achieve better FID than S-PNDMs in most cases. We think this is because
the result between our transfer part and target ODE has a higher-order error, which limits the convergence order of F-PNDMs. This error from the change of the transfer part is theoretical but does
not influence the quality of images according to the property of Equation (11). However, making
the transfer part higher-order convergent and finding the influence of such change is still exciting
and needs more research. 3) Extend PNDMs to more general applications: when we proved the
convergence order of PNDMs, we found that other kinds of transfer parts can keep the convergence
order unchanged too, which means that pseudo numerical methods can be used on more general
applications, like certain neural ODEs (Chen et al., 2019; Dupont et al., 2019).
9
Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like to thank Qinghai Zhang, Feng Wang, Shuang Hu, Yang Li, Ziyue Jiang and Rongjie
Huang from Zhejiang University for their insightful discussions during the course of this paper.
R EFERENCES
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their
Applications, 12(3):313–326, 1982. ISSN 03044149. doi: 10.1016/0304-4149(82)90051-5.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured
denoising diffusion models in discrete state-spaces, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint, 2020.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. arXiv:1806.07366 [cs, stat], December 2019.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR:
Conditioning Method for Denoising Diffusion Probabilistic Models. CoRR, abs/2108.02938:14,
2021.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented Neural ODEs. arXiv:1904.01681
[cs, stat], October 2019.
Hairer Ernst and Wanner Gerhard. Solving Ordinary Differential Equations II: Stiff and DifferentialAlgebraic Problems. Springer Series in Computational Mathematics 14. Springer-Verlag Berlin
Heidelberg, 2 edition, 1996.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information
processing systems, 27, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.
arXiv:1706.08500 [cs, stat], January 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv, 256,
2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax Flows
and Multinomial Diffusion: Towards Non-Autoregressive Language Models. arXiv preprint,
2021.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. DiffTTS: A Denoising Diffusion Model for Text-to-Speech. arXiv:2104.01409 [cs, eess], April 2021.
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Score Matching
Model for Unbounded Data Score. arXiv:2106.05527 [cs, stat], August 2021.
Zhifeng Kong and Wei Ping.
On Fast Sampling of Diffusion Probabilistic Models.
arXiv:2106.00132 [cs], June 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile
Diffusion Model for Audio Synthesis. arXiv:2009.09761 [cs, eess, stat], March 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
Max W. Y. Lam, Jun Wang, Rongjie Huang, Dan Su, and Dong Yu. Bilateral Denoising Diffusion
Models. arXiv:2108.11514 [cs, eess], August 2021.
10
Published as a conference paper at ICLR 2022
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, and Zhou Zhao. Diffsinger: Singing voice
synthesis via shallow diffusion mechanism. arXiv preprint arXiv:2105.02446, 2, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Shitong Luo and Wei Hu. Diffusion Probabilistic Models for 3D Point Cloud Generation. arXiv
preprint, 2021.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. CoRR,
abs/2102.09672, 2021.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad
Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In
International Conference on Learning Representations, 2022.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. pp. 10, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. arXiv
preprint, pp. 1–19, 2020a.
Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models.
arXiv:2006.09011 [cs, stat], October 2020a.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. arXiv:1907.05600 [cs, stat], October 2020b.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-Based Generative Modeling through Stochastic Differential Equations. pp. 1–32,
2020b.
Sauer Timothy. Numerical Analysis. Pearson, 3 edition, 2017. ISBN 2017028491, 9780134696454,
013469645X.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based Generative Modeling in Latent Space.
arXiv:2106.05931 [cs, stat], June 2021.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to Efficiently
Sample from Diffusion Probabilistic Models. CoRR, abs/2106.03802, 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN:
Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop.
arXiv:1506.03365 [cs], June 2016.
A
A PPENDIX
A.1
R ELATED W ORK
DDPMs have been well developed in the last few years. Some works concentrate on improving the
quality and the speed of DDPMs and making DDPMs more practical. Song et al. (2020a) introduce
new inference equations to accelerate DDPMs. Nichol & Dhariwal (2021), Watson et al. (2021) and
Kong & Ping (2021) choose to find better variance schedules to improve the images quality. Vahdat
et al. (2021) combine the advantages of DDPMs and Variational Autoencoders and get better results.
Furthermore, Kim et al. (2021) try to solve an existing bottleneck that the inference equations of
DDPMs are unbounded in some situations.
Song et al. (2020b) find the similarity between DDPMs and noise conditional score networks (NCSNs (Song & Ermon, 2020b)), which is that they both use a process similar to Langevin dynamics
to produce samples. Therefore, some works (Song & Ermon, 2020a; Kim et al., 2021) that can
11
Published as a conference paper at ICLR 2022
improve the results of NCSNs also can be used in DDPMs. Additional, Song et al. (2020b) combine DDPMs and NCSNs under the framework of neural differential equations (Chen et al., 2019;
Dupont et al., 2019). Therefore, numerical methods widely used in neural differential equations can
also be applied to accelerate DDPMs. Our work successfully combines the advantages of Song et al.
(2020a) and Song et al. (2020b). We use a transfer part from DDIMs and use different gradient parts
from different numerical methods. Although we and Song et al. (2020b) solve certain differential
equations derived from DDPMs, we use different target different equations and different numerical
methods, which get better results.
The application of DDPMs is not limited to unconditional image generation. Some works apply
DDPMs to various types of data successfully, including text-to-speech (Chen et al., 2020; Lam
et al., 2021), singing voice (Liu et al., 2021), 3D Point Cloud (Luo & Hu, 2021), text generation
(Austin et al., 2021). Additionally, DDPMs can also be used to generate conditional samples, too
(Jeong et al., 2021; Choi et al., 2021).
A.2
C ONVERGENT O RDER OF M ETHOD
We use the forward Euler method and linear multi-step method to show what is the order of a
method. Assume that xt is precise and compute the error at xt+δ . For forward Euler method, we
have:
et,δ = x(t + δ) − xt,δ
δ2
= x(t) + δf (x(t), t) + f 00 (c) − (x(t) + δf (xt , t))
(15)
2
δ2
δ 2 00
f (c) ≤ M
2
2
Here, we assume that f 00 is continuous, so f 00 is bounded in a close area. x(t + δ) is the precise
result and xt,δ is the numerical result from t to t + δ.
=
For linear multi-step method, we have:
et,δ =x(t + δ) − xt,δ
δ
δ2
δ4
= x(t) + f (x(t), t) + f 0 (x(t), t) + · · · + f (3) (x(t), t) + O δ 5
1!
2!
4!
−
x(t) + b1 δf (x(t), t) +
4
X
bs δ
s=2
4
X
(−(s − 1)δ)k
k=0
(k)!
!! (16)
f
(k−1)
(x(t), t) + O δ
5
=O(δ 5 )
Here, {bs } satisfy
P4
s=1 bs = 1 and the following equations for j ∈ {1, · · · , 3}:
(−1)j b2 + (−2)j b3 + (−3)j b4 =
1
.
j+1
(17)
We call the error at xt,δ local error, and the error at xt+M δ (M is big enough but finite) global error.
Assume the local error of our method has order k+1 and the target ODE satisfies Lipschitz condition,
then:
x(t + M δ) − xt,M δ ≤|x(t + M δ) − xt+(M −1)δ,δ | + |xt+(M −1)δ,δ − xt,M δ |
≤et+(M −1)δ,δ + eLδ |x(t + (M − 1)δ) − xt,(M −1)δ |
≤et+(M −1)δ,δ + eLδ et+(M −2)δ,δ + · · ·
≤Cδ k+1 1 + eLh + · · · + e(i−1)Lh
=Chk+1
(18)
iLδ
eiLδ − 1
−1
k+1 e
≤
Cδ
Lδ
e −1
Lδ
=O(δ k ).
Therefore, the global error will be one order lower than the local error. From Equation (15), we can
see the forward Euler method has local error O(δ 2 ) and global error O(δ), so we call it the first-order
12
Published as a conference paper at ICLR 2022
numerical method. And the linear multi-step method has local error O(δ 5 ) and global error O(δ 4 ),
and we call it the fourth-order numerical method.
In addition, assuming that a numerical method has kth-order global error, we can compute the convergence speed of this numerical method:
x2δ
(2δ)k
t+T − x(t + T )
=
= 2k .
k
δ→0 xδ
δ
−
x(t
+
T
)
t+T
lim
(19)
Here, xδt+T is the result at t+T and move δ every step. This shows that the fourth-order method can
converge to the exact solution faster than the first-order method when δ → 0, which means that we
can use a bigger iteration interval δ to achieve similar global error and a bigger iteration interval
means that we can iterate fewer times to get results with high quality.
A.3
P SEUDO S ECOND -O RDER M ETHOD
We introduce two second-order numerical methods. First is improved Euler method satisfying:

 k1 = f (xt , t)
(20)
k2 = f (xt + δk1 , t + δ)

xt+δ = xt + 2δ (k1 + k2 )
Second is another linear multi-step method called second-order linear multi-step method satisfying:
δ
xt+δ = xt + (3ft − ft−δ )
2
And the corresponding pseudo improved Euler methods satisfying:


e1t = θ (xt , t)



1
1

 xt = φ(xt , et , t, t + δ)
1
2
et = θ (xt , t + δ)



e0t = 21 (e1t + e2t )



xt+δ = φ(xt , e0t , t, t + δ)
Pseudo second-order linear multi-step method satisfying:

 et = θ (xt , t)
e0 = 12 (3et − et−δ )
 t
xt+δ = φ(xt , e0t , t, t + δ)
Similar to what we do to get F-PNDMs, We combine them to get S-PNDMs. Abbreviate Equation
(22) and (23) as
xt+δ , et = P IE(xt , {ep }p<t , t, t + δ),
xt+δ , e1t = P LM S 0 (xt , t, t + δ).
A.4
(21)
(22)
(23)
Algorithm 3 S-PNDMs
1: xT ∼ N (0, I)
2: for t = T − 1 do
3:
xt , et = P IE(xt+1 , t + 1, t)
4: end for
5: for t = T − 2, · · · , 1, 0 do
6:
xt , et = P LM S 0 (xt+1 , {ep }p>t , t +1, t)
7: end for
8: return x0
T HE E XISTENCE OF A D ERIVATIVE
Because ᾱt is usually obtained by multiplying a linear variance schedule βt . So we have
2
ᾱt = eat +bt+c ,
13
(24)
Published as a conference paper at ICLR 2022
and ᾱ0 = 1, so c = 0. Now we have
xt−δ − xt
δ→0
δ
lim
ᾱt−δ − ᾱt
= lim
δ→0
δ
xt
θ (xt , t)
p
√ √
√
−√ p
ᾱt ( ᾱt−δ + ᾱt )
ᾱt ( (1 − ᾱt−δ )ᾱt + (1 − ᾱt )ᾱt−δ )
2
ᾱt−δ − ᾱt xt
θ (xt , t)
xt
θ (xt , t)
= lim
= (eat +bt )0
− √
− √
δ→0
δ
2ᾱt
2ᾱt
2 1 − ᾱt ᾱt
2 1 − ᾱt ᾱt
xt
θ (xt , t)
1
θ (xt , t)
=(2at + b)ᾱt
− √
= (2at + b)(xt − √
).
2ᾱt
2
2 1 − ᾱt ᾱt
1 − ᾱt
!
(25)
(xt ,t)
To make limδ→0 xt−δδ−xt |t=0 is well-defined, b must equal to zero, or (2at + b) √θ 1−
will tend to
ᾱt
infinity. This is a strong condition that most variance schedules do not satisfy. In practice, DDPMs
can choose the variance schedule very freely. This means that treating DDPMs as ODEs directly is
not proper and has theoretical weakness.
A.5
R ELATIONSHIP BETWEEN t, θ AND xt
Relationship between t and θ . In Figure 5, we can see that the denoising process tends to converge,
whether in the θ domain or the sample/image domain when the step-index tends to zero. Therefore,
we can say that the noise becomes more and more precise when step, namely t, tends to zero.
epsilon
image
delta
102
101
100
0
200
400
step
600
800
1000
Figure 5: The norm δ of the difference between two adjacent terms under different steps
√
√
Relationship between θ and xt To prove Property 3.1, assume that xt = ᾱt x0 + 1 − ᾱt , NN
is the neural network and θ = NN(xt , t). Because we assume that the gradient part is precise, then
we have θ = . Then for all t0 ≤ t, we have:
√
√
1 − ᾱt θ
√
+ 1 − ᾱt0 θ
ᾱt
√
√
√
√
√
ᾱt x0 + 1 − ᾱt − 1 − ᾱt √
= ᾱt0
+ 1 − ᾱt0 θ (xt , t)
ᾱt
√
√
= ᾱt0 x0 + 1 − ᾱt0 .
xt0 =
√
ᾱt0
Here, we can find that xt0 =
A.6
xt −
√
ᾱt0 x0 +
√
(26)
1 − ᾱt0 is also precise, so Property 3.1 is true.
O RDER A NALYSIS OF P SEUDO M ETHOD
For the convenience of theoretical analysis, we generalize the problem. Let φ(x(t), , t, δ) =
f (x(t), t, δ) + g(t, δ)(x(t), t) and we have the property f (x(t), t, 0) = g(t, 0) = 0. Then we
14
Published as a conference paper at ICLR 2022
have:
x(1) = x(0) +
X
(y(t + δ) − y(t))
δ→0
= x(0) +
X
(f (x(t), t, δ) + g(t, δ)(x(t), t))
δ→0
Z 1
= x(0) +
0
(27)
∂f
∂g
(x(t), t, 0) +
(t, 0)(x(t), t) .
∂δ
∂δ
Now, Equation (10) becomes a special case of this more general version and, in this special case, we
have:
!
α(t + δ)
√
− 1 x(t)
f (x(t), t, δ) =
α
p
p
(1 − α(t))α(t + δ)
p
g(t, δ) = 1 − α(t + δ) −
α(t)
p
(28)
Now, we compute the local error of S-PNDMs. We first compute the theoretical and numerical
results of different numerical methods. We have:
x(t + δ)
∂f
∂g
(x(t), t, 0) +
(t, 0)(x(t), t) +
∂δ
∂δ
0
2
δ
∂f
∂g
(x(t), t, 0) +
(t, 0)(x(t), t) + O(δ 3 )
2 ∂δ
∂δ
∂f
∂g
=x(t) + δ
(x(t), t, 0) +
(t, 0)(x(t), t) + O(δ 3 )+
∂δ
∂δ
δ2 ∂ 2 f
∂2f
∂f
∂g
(x(t), t, 0) +
(x(t), t, 0)
(x(t), t, 0) +
(t, 0)(x(t), t)
+
2 ∂δ∂t
∂δ∂x
∂δ
∂δ
δ2 ∂ 2 g
∂g
(t, 0)(x(t), t) +
(t, 0)0 (x(t), t)
2 ∂δ∂t
∂δ
=x(t) + δ
15
(29)
Published as a conference paper at ICLR 2022
and
xS-PNDM (t + δ)
1
=x(t) + f (x(t), t, δ) + g(t, δ) ((x(t), t) + (x(t) + φ(..., t, δ), t + δ))
2
∂f
δ2 ∂ 2 f
=x(t) + δ (x(t), t, 0) +
(x(t), t, 0) + O(δ 3 )+
2
∂δ
2
∂δ
1
δ2 ∂ 2 g
∂g
(t,
0)
((x(t), t) + (x(t + δ) + O(δ 2 ), t + δ))
δ (t, 0) +
2
∂δ
2 ∂δ
2
∂f
δ2 ∂ 2 f
=x(t) + δ (x(t), t, 0) +
(x(t), t, 0) + O(δ 3 )+
2
∂δ
2
∂δ
∂g
1
δ2 ∂ 2 g
δ (t, 0) +
(t, 0)
((x(t), t) + (x(t + δ), t + δ))
∂δ
2 ∂δ 2
2
∂f
δ2 ∂ 2 f
(x(t), t, 0) + O(δ 3 )+
=x(t) + δ (x(t), t, 0) +
2
∂δ
2
∂δ
∂g
δ2 ∂ 2 g
1
(t,
0)
((x(t), t) + (x(t), t) + δ(x(t), t)0 )
δ (t, 0) +
2
∂δ
2 ∂δ
2
∂f
δ2 ∂ 2 f
=x(t) + δ (x(t), t, 0) +
(x(t), t, 0) + O(δ 3 )+
2
∂δ
2
∂δ
∂g
1
δ2 ∂ 2 g
0
δ (t, 0) (x(t), t) + δ(x(t), t) +
(t, 0)(x(t), t)
∂δ
2
2 ∂δ 2
∂f
∂g
=x(t) + δ
(x(t), t, 0) +
(t, 0)(x(t), t) + O(δ 3 )+
∂δ
∂δ
∂g
∂2g
δ2 ∂ 2 f
0
(x(t), t, 0) +
(t, 0)(x(t), t) + 2 (t, 0)(x(t), t) .
2 ∂δ 2
∂δ
∂δ
(30)
Then we compute the difference between the theoretical and numerical results. We have:
x(t + δ) − xS-PNDM (x + δ)
2
δ2
∂2f
∂f
∂ f
∂2f
=
− 2 )(x(t), t, 0) +
(x(t), t, 0) (x(t), t, 0) +
(
2
∂δ∂t
∂δ
∂δ∂x
∂δ
2
2
2
2
∂ g
∂ g
∂ f
∂g
δ
(
− 2 )(t, 0) +
(x(t), t, 0) (t, 0) (x(t), t) + O(δ 3 )
2
∂δ∂t ∂δ
∂δ∂x
∂δ
(31)
In this special case, we compute the derivatives of some items needed in Equation (31). We have:
!
p
(1 − α(t))α(t + δ)
∂g
∂ p
p
(t, δ) =
1 − α(t + δ) −
∂δ
∂δ
α(t)
(32)
p
0
1 − α(t)α0 (t + δ)
−α (t + δ)
p
= p
−
2 1 − α((t + δ))
2 α(t)α(t + δ)
!
!
p
α(t + δ)
∂f
∂
√
(x(t), t, δ) =
− 1 x(t)
∂δ
∂δ
α
(33)
α0 (t + δ)
= p
x(t)
2 α(t)α(t + δ)
∂2f
α00 (t + δ)
−α0 (t + δ)2
(x(t), t, δ)|δ=0 = p
x(t) + p
x(t)|δ=0
2
∂δ
2 α(t)α(t + δ)
4 α(t)α(t + δ)3
(34)
∂2f
α00 (t)
α0 (t)2
(x(t), t, 0) =
x(t) −
x(t)
∂δ∂t
2α(t)
2α(t)2
(35)
∂2f
α0 (t)
(x(t), t, 0) =
∂δ∂x
2α(t)
(36)
16
Published as a conference paper at ICLR 2022
p
1 − α(t)α00 (t + δ)
p
+
2 α(t)α(t + δ)
p
1 − α(t)α0 (t + δ)2
−α0 (t + δ)2
p
|δ=0
p
3 +
4 α(t)α(t + δ)3
4 1 − α(t + δ)
p
1 − α(t)α00 (t)
−α00 (t)
p
p
−
+
=
2 1 − α((t))
2 α(t)α(t)
p
1 − α(t)α0 (t)2
−α0 (t)2
p
+
p
3
4 α(t)α(t)3
4 1 − α(t)
(37)
!
p
1 − α(t)α0 (t)
−α0 (t)
p
p
−
2 1 − α((t))
2 α(t)α(t)
p
1 − α(t)α00 (t)
−α00 (t)
p
= p
−
+
2 1 − α((t))
2 α(t)α(t)
p
1 − α(t)α0 (t)2
α0 (t)2
−α0 (t)2
+ p
p
3 +
2
2α(t)
4 α2 (t)(1 − α(t))
4 1 − α(t)
(38)
∂2g
−α00 (t + δ)
(t, δ)|δ=0 = p
−
2
∂δ
2 1 − α((t + δ))
∂2g
∂
(t, δ)|δ=0 =
∂δ∂t
∂t
Now we can compute the final result of Equation (31). We split it into three parts and the values of
the first two terms. We have:
∂2f
∂2f
∂2f
∂f
(
− 2 )(x(t), t, 0) +
(x(t), t, 0) (x(t), t, 0)
∂δ∂t
∂δ
∂δ∂x
∂δ
00
00
α (t)
α0 (t)2
α (t)
−α0 (t)2
1 α0 (t)
=
x(t) −
x(t)
−
x(t)
+
x(t)
+
x(t)
2α(t)
2α(t)2
2α(t)
4α(t)2
2α(t) 2α(t)
=0
(39)
and
∂2g
∂2f
∂g
∂2g
− 2 )(t, 0) +
(x(t), t, 0) (t, 0)
(
∂δ∂t ∂δ
∂δ∂x
∂δ
!
p
p
0
2
0
2
1 − α(t)α (t)
1 − α(t)α0 (t)
α0 (t)
α (t)
−α0 (t)
p
p
+
−
=
+
4α(t)2
2α(t)
4α(t) 1 − α(t) 2α(t) 2 1 − α((t))
!
−α0 (t)
α0 (t)2
α0 (t)
p
p
=
+
2
4α(t) 1 − α(t) 2α(t) 2 1 − α(t)α(t)
(40)
=0
Finally, we get the final result of Equation (31):
x(t + δ) − xS-PNDM (x + δ) = O(δ 3 )
(41)
And the computation of the convergence order of F-PNDMs is similar, and we ignore it here. Therefore, Property 3.2 is true.
A.7
VARIANCE SCHEDULE
According to Cifar10 (cosine) in Table 2, PNDMs can be used on both linear variance schedule and
cosine variance schedule. However, we also notice cosine variance schedule can make FID lower
when we use relatively big generation steps, but the effort is limited when the number of steps is
small. F-PNDM uses information from four consecutive steps, so the smoothness of the schedule is
more important for F-PNDM than DDIM. According to this experiment, our work can be used with
works that pay attention to variance schedules to improve the acceleration effect further. However,
a variance schedule that fits pseudo numerical methods better remains to be found in further work.
17
Published as a conference paper at ICLR 2022
A.8
T OY EXAMPLE
Here, we design a toy example to test our new methods without the influence of neural networks.
We randomly generate the initial input x1 = (m1 , m2 ), mi ∼ U (0, 1) and use a simple analytic
equations θ (x) = (sin x[0], cos x[1]) to replace the neural networks in real cases. Let φ in Equation
(11) is unchanged and ᾱt = α(t) = 1 − t, then we get:
φ(xt , θ (xt ), t, t − δ)
√
ᾱt−δ
(ᾱt−δ − ᾱt )
p
= √
t
xt − √ p
ᾱt
ᾱt ( (1 − ᾱt−δ )ᾱt + (1 − ᾱt )ᾱt−δ )
p
1 − (t − δ)
δ
p
θ (xt )
xt − √
= √
p
1−t
1−t
(t − δ)(1 − t) + t(1 − (t − δ))
(42)
Here, we use three different numerical methods to generate x0 .
For DDIM, we have:
xt−δ = xt + φ(xt , θ (xt ), t, t − δ)
(43)
For FON, according to Equation (10), we have:
!
(x
)
x
θ
t
t
p
−
e0t = ᾱ0 (t)
2ᾱ(t) 2ᾱ(t) 1 − ᾱ(t)
xt
θ (xt )
√
=−
−
2(1 − t) 2(1 − t) t
δ
xt−δ = xt + (55e0t − 59e0t+δ + 37e0t+2δ − 9e0t+3δ )
24
For F-PNDM, we have:
e0 =
1
(55θ (xt ) − 59θ (xt+δ ) + 37θ (xt+2δ ) − 9θ (xt+3δ ))
24
xt−δ = xt + φ(xt , e0 , t, t − δ)
(44)
(45)
Then we draw the corresponding generation curves in Figure 6. We find that the result is similar
to the real cases. The main difference here is that FON can correct its results while the real case
cannot. The reason is that the gradient is well-defined everywhere, while in real cases, the gradient
is meaningful on the high-density region of the data xt of DDPMs.
A.9
M ORE FID R ESULTS
Here, We provide our more detailed FID results on Cifar10, CelebA, LSUN-church and LSUNbedroom in Table 3, 4, 5, 6.
A.10
M ORE I MAGE R ESULTS
Here, we show more generated images on Cifar10, CelebA, LSUN-church and LSUN-bedroom in
Figure 7, 9, 8, 10, 11, 12.
18
Published as a conference paper at ICLR 2022
DDIM-5
FON-5
PNDM-5
DDIM-1000
0.0
0.5
0.5
1.0
0.5
2.0
2.5
3.0
1.0
x2
1.5
1.5
1.5
2.0
2.0
2.5
2.5
3.0
1.0
1.2
1.6
1.4
1.4
1.6
x1
DDIM-5
FON-5
PNDM-5
DDIM-1000
1.8
DDIM-20
FON-20
PNDM-20
DDIM-1000
0.0
1.0
x2
3.0
0.9
1.0
1.1
1.2
1.3
x1
1.4
1.5
1.0
1.1
1.2
x1
1.3
1.4
1.5
1.2
1.1
1.1
1.0
x2
x2
1.2
1.0
x2
x2
DDIM-10
FON-10
PNDM-10
DDIM-1000
0.0
0.9
0.9
1.0
0.8
0.8
0.5
0.6
0.7
0.8
x1
0.9
1.0
1.1
DDIM-10
FON-10
PNDM-10
DDIM-1000
0.6
DDIM-20
FON-20
PNDM-20
DDIM-1000
0.8
0.7
0.8
0.9
x1
1.0
0.6
0.7
0.8
x1
0.9
1.0
Figure 6: The generation curve of our toy example.
steps
5
10
20
25
40
50
100
125
200
250
500
1000
DDIM
DDIM*
FON
S-PNDM
F-PNDM
44.5
98.0
22.8
13.9
13.4
18.5
13.1
11.6
7.03
10.9
7.41
7.56
5.00
6.84
9.61
6.41
6.79
4.76
7.65
5.50
5.57
4.10
4.67
6.99
5.26
5.18
3.95
4.16
5.52
4.65
4.34
3.72
5.19
4.54
4.18
3.64
4.69
4.23
3.97
3.60
4.52
4.12
3.91
3.60
4.17
3.84
3.81
3.64
4.04
4.00
3.71
3.80
3.70
DDIM*
S-PNDM
F-PNDM
28.7
18.3
18.2
14.5
8.64
7.05
8.79
5.77
4.61
7.83
5.45
4.32
6.41
4.76
3.85
5.86
4.46
3.68
4.92
3.94
3.53
4.75
3.85
3.46
4.42
3.69
3.47
4.30
3.71
3.49
3.98
3.60
3.44
3.69
3.38
3.26
Table 3: Cifar10 image generation measured in FID. The upper part uses linear variance schedule
and the bottom half uses cosine variance schedule. The first line shows the FID provided by Song
et al. (2020a).
steps
5
10
20
25
40
50
100
125
200
250
500
1000
DDIM
DDIM*
FON
S-PNDM
F-PNDM
24.4
60.2
15.2
11.3
17.3
16.9
16.0
12.2
7.71
13.4
11.6
9.45
5.51
13.7
12.3
10.6
8.42
4.75
9.99
8.89
6.50
3.67
9.17
8.95
8.13
5.69
3.34
6.53
6.36
6.70
4.03
2.81
5.74
6.28
3.72
2.75
4.78
5.45
3.30
2.71
4.44
5.14
3.19
2.71
3.75
4.49
3.01
2.77
3.51
3.41
4.17
2.99
2.86
Table 4: CelebA image generation measured in FID. All of them use linear variance schedule.
steps
5
10
20
25
40
50
100
125
200
250
DDIM
DDIM*
S-PNDM
F-PNDM
48.8
20.5
14.8
19.5
18.8
11.8
8.69
12.5
11.7
9.20
9.13
11.0
9.13
9.33
10.1
9.31
9.69
10.8
10.0
9.49
9.89
10.6
9.84
9.82
10.1
9.83
9.88
9.99
9.85
10.0
10.1
9.88
10.0
10.1
Table 5: LSUN-church image generation measured in FID. All of them use linear variance schedule.
19
Published as a conference paper at ICLR 2022
steps
5
10
20
25
40
50
100
125
200
250
DDIM
DDIM*
S-PNDM
F-PNDM
51.3
18.1
12.6
17.0
16.4
10.2
6.99
8.89
8.47
6.50
5.68
7.41
6.02
5.74
6.27
5.74
6.17
6.75
6.05
5.81
6.44
6.62
5.97
6.29
6.91
6.03
6.44
6.96
6.23
6.69
7.03
6.32
6.75
6.92
Table 6: LSUN-bedroom image generation measured in FID. All of them use linear variance schedule.
Figure 7: 5, 10, 20, 50, 100, 250, 500-steps generated results using DDIMs, classical numerical
methods and PNDMs on Cifar10.
Figure 8: 5, 10, 20, 50, 100, 250, 500-steps generated results using DDIMs, classical numerical
methods and PNDMs on CelebA.
20
Published as a conference paper at ICLR 2022
Figure 9: Generated images of PNDMs on Cifar10.
Figure 10: Generated images of PNDMs on CelebA.
21
Published as a conference paper at ICLR 2022
Figure 11: 5, 10, 20, 50, 100-steps generated results using DDIMs, classical numerical methods and
PNDMs on LSUN-church.
22
Published as a conference paper at ICLR 2022
Figure 12: 5, 10, 20, 50, 100-steps generated results using DDIMs, classical numerical methods and
PNDMs on LSUN-bedroom.
23
Published as a conference paper at ICLR 2022
A.11
M ORE V ISUALIZATION R ESULTS
Here, we put more visualization results similar to Figure 4 in Figure 13.
1.2
1.2
DDIM-5
FON-5
PNDM-5
DDIM-1000
1.0
0.8
y
y
y
0.6
0.6
0.4
0.4
0.4
0.8
0.7
0.6
x
0.5
0.2
0.4
DDIM-25
FON-25
PNDM-25
DDIM-1000
1.0
0.85
0.80
0.75
0.70
0.65
x
0.60
0.55
0.85
DDIM-40
FON-40
PNDM-40
DDIM-1000
1.0
0.8
0.6
0.4
0.4
0.4
x
0.70
0.65
0.60
0.85
0.80
0.75
x
0.75
0.70
0.65
0.60
0.85
0.65
0.60
0.60
0.60
0.55
0.55
0.55
y
0.70
0.65
y
0.70
0.65
DDIM-5
FON-5
PNDM-5
DDIM-1000
0.40
0.35
0.7
0.45
0.40
0.8
0.9
1.0
x
0.75
1.1
1.2
0.35
DDIM-10
FON-10
PNDM-10
DDIM-1000
0.8
0.40
0.9
x
1.0
0.35
1.1
0.65
0.60
0.60
0.60
0.55
0.55
0.55
DDIM-25
FON-25
PNDM-25
DDIM-1000
0.80
0.85
0.90
0.45
0.40
0.95
x
1.00
1.05
1.10
0.35
0.80
1.15
0.75
x
0.70
0.65
0.60
0.80
0.85
0.90
0.95
x
1.00
1.05
1.10
1.15
1.10
1.15
0.50
0.50
0.40
0.35
y
0.70
0.65
y
0.70
0.65
0.45
0.60
DDIM-50
FON-50
PNDM-50
DDIM-1000
DDIM-20
FON-20
PNDM-20
DDIM-1000
0.45
0.70
0.50
0.65
0.50
0.50
0.45
0.80
0.75
0.70
0.50
0.70
x
y
y
y
0.6
0.75
0.75
0.8
0.6
0.80
0.80
1.0
0.8
0.85
y
0.8
0.6
0.9
DDIM-20
FON-20
PNDM-20
DDIM-1000
1.0
0.8
0.2
y
DDIM-10
FON-10
PNDM-10
DDIM-1000
1.0
DDIM-40
FON-40
PNDM-40
DDIM-1000
0.85
0.90
DDIM-50
FON-50
PNDM-50
DDIM-1000
0.45
0.40
0.95
1.00
1.05
x
1.10
1.15
0.35
0.85
0.90
0.95
1.00
x
1.05
Figure 13: Visualization results under 5, 10, 20, 25, 40 and 50 steps.
A.12
FID RESULT WITH S TANDARD D EVIATION
Here, we report the mean and standard deviation of FID results, tested over four sampling runs.
FID
step
dataset
10
20
50
100
250
1000
18.50±.06
13.00±.11
11.58±.10
6.12±.07
10.86±.08
7.33±.06
7.53±.07
5.04±.07
6.95±.04
5.24±.05
5.15±.05
4.01±.02
5.49±.06
4.64±.04
4.34±.03
3.75±.04
4.52±.02
4.12±.03
3.93±.02
3.67±.03
4.02±.04
3.73±.03
3.83±.03
3.78±.04
model
Cifar10
(linear)
DDIM*
FON
S-PNDM
F-PNDM
Table 7: Image generation measured in FID on Cifar10. DDIM* means a kind of pseudo numerical
method and also a retest of DDIM.
24