Optimum quantizer design
The common quantizer optimality conditions were set by Lloyd (1957). These were
necessary and sufficient conditions for a fixed-rate quantizer to be optimal.
Note: The optimality only holds for the fixed rate quantizers, where we assume that
we use the same amount of bits to represent each quantization level. Actually, this
is a bad assumption because one can also assign different number of bits to
different codebook entries. Normally, the reason to perform entropy coding after the
quantization step is to assign less bits to frequently occuring codebook values and
more bits to seldom occuring ones.
Usually, the quantizer optimality and bit allocation optimality is not done at
simultaneously, so we will only concentrate on the fixed rate quantizer optimality
now.
Lloyd's conditions are:
The quantizer interval boundaries (called partitioning) must be optimal for a
given set of reproduction levels.
The set of reproduction levels (the codebook values) must be optimal for a
given interval partitioning.
The first condition is commonly referred as the encoder optimality (because we
have to optimize the boundaries according to the input range), and the second
condition is called decoder optimality, because it optimizes the reproduction levels,
which is an issue at the decoder side.
The encoder optimality requires:
Minimum distortion (or nearest neighbor encoding)
The decoder optimality requires:
Centroid values for the reproduction
If the two conditions are met at the same time, then the optimal fixed rate
quantizer is obtained.
This optimality is obvious if one understands the following minimization of the
distortion due to the centroid and nearest neighbor rules as given in two steps:
.
First, assume that the codebook ( ) are fixed. The input x is obviously nearest to
one of the reproduction levels. The quantization result of the input x can be at most
as small as that minimum distance. Hence,
. Taking expected
values of both sides, we get an average expression:
The lower bound is achieved only by selecting the quantization intervals in such a
way that the minimum distance output is selected as the quantizer output:
which means,
(x is coded into i) only if
for all k. The overall
encoder rule is ,therefore:
This optimal encoder is a nearest-neighbor rule.
. Now let's optimize the decoder (the second condition). Assume, this time, that
the interval partitioning (encoder) is fixed. Let us see what happens in the MSE
distance case (The MSE quantizer):
We want to minimize
this is equivalent to minimizing
The optimum
separately for each i.
values which minimizes the above are
(from
stochastic process). This expectation is the conditional mean value of x when x is
known to be in the region i. The optimum decoder is, therefore, the weighted
(weighted by the PDF) mean (or average) of each region. We call this mean:
centroid.
In most of the practical cases, the PDF of the input is unknown. Therefore, we
approximate the PDF of x by the experimental samples of it. The easiest way to
understand a numerical optimal quantizer design is by giving an example:
Numerical exercise: Suppose we have the following numbers obtained from a
random process: x = (-1.1,-1.0,-0.9, 0.8 1.0). Using this much data, let us try to
obtain the optimal two level. We will see more advanced techniques for obtaining
numerical solutions. However, from this much information, we can start by
observing that there are two types of outcomes. One is around -1 and the other is
around 1. Let us, therefore, start with the intervals
=(- ,0) and =(0, ). The
expected values inside
and
can be calculated by taking the average of the
samples inside each interval:
and
Therefore, the optimum quantizer output levels are determined.
As a matter of fact, the interval boundaries should also be changed according to the
optimal output levels. Luckily, the interval boundaries are almost the same and
consistent with our initial assumption. Normally, we must select the region
boundary to be the half way between the two quantization output levels:
=(- ,-0.05) and
0.9.
=(-0.05,
) because -0.05 is the half way between -1 and
If we continuously change the boundaries and the corresponding output levels, we
come up with a famous iterative algorithm called the Lloyd-Max quantizer design
algorithm. We will explain the details of the Lloyd-Max quantizer design procedure
later in this lecture, but you may just want to see if the averages inside the new
intervals have changed, or not. Since the averages have not changed, the output
levels are the same. Therefore, the region boundary stays the same. In fact, if the
boundaris or the quantizer output values do not change, the Lloyd-Max quantizer
design procedure stops. For this example, the procedure stopped in one iteration!
Optimality conditions for the MSE quantizer:
Suppose that q(x) is a quantizer which satisfies the above two conditions (centroid
and nearest neighbor conditions) to form a minimum MSE quantizer. Define the
quantization error
. Then the following equations hold:
1.
2. E{ |
}=0
3. E{ }=0
4. E{q(x) }=E{
5. E{
}=
=
}=0
-
6. E{X }= Let us prove conditions 1,2, and 4.
Proof of 1:
We have
but we also know that
from the probability theory. If we substitute this to the
previous equation, we have
which is the
desired equation.
Proof of 2 :
By definition, we have
. Substituting the definition
of the error signal, we get
. We distribute
this integral as a difference of two integrals as
. The first part is exactl
equal to
, therefore
.
Proof of 4 :
By definition
. Again, by splitting into a
difference of two integrals, we get
. Notice that the first integral is
equal to
from property 1. We also know that the second integral is equal to 1. As
a result we get
which is the desired equation.
Homework 2 : Prove the properties 3, 5, and 6 of the minimum MSE quantizer:
3. E{ }=0
5. E{
}=
=
-
6. E{X }= -
The above optimality properties are satisfied according to the minimization of the
distortion with respect to both:
Intervals:
, or the boundaries of these intervals, say
Therefore we have:
.
=0 for i=2,3,...,L
The output levels:
Therefore we have:
=0 for i=1,2,...,L
Assume that the left-most boundary is
and the right-most boundary is
. The above two conditions provide the nearest neighbor and centroid
solutions as:
Eq. Opt-1.
Eq. Opt-2.
Lloyd-Max quantizer design
Based on the optimality conditions given above, an iterative algorithm which
optimizes the encoder and decoder parts one after the other has been developed.
There are two variants of the iterative algorithm, but we will introduce only the
more popular one here.
Algorithm :
1. Start with initial reproduction levels
2. Find the optimum interval boundaries ( )corresponding to the
to Eq. Opt-1.
3. Recalculate the reproduction levels 's according to the found
Eq. Opt-2.
4. If new 's are very different from old 's, then go to 2.
's according
's according to
This is a very popular algorithm, also known as "clustering".
Notice that Eq. Opt-2 assumes the knowledge of the pdf of the random variable.
This is not true in almost all of the cases. If we have numerical data, instead of
equation Eq. Opt-2, we calculate the centroid of the data samples inside the interval
. The calculation of the centroid is very simple; just average all the data that is
inside the current interval. The following numerical example illustrates step-by-step
how the Lloyd-Max algorithm converges to the optimal result.
Numerical exercise: Let us start with the data set:
x=[-0.9 -0.8 -0.7 -0.6 -0.2 -0.2 0.1 0.1 0.7 0.7 1.0] and design a 3-level quantizer.
Initially, let us start with output levels -1, 0, and 1 (which seem quite
reasonable).
The interval boundaries corresponding to these
values are [-inf, -0.5, 0.5,
inf]. Notice that intervals are half ways between the
values (satisfying Eq.
Opt-1), except the boundaries that reach to infinity.
With the new boundaries, we have the following situation:
Samples -0.9, -0.8, -0.7, and -0.6 are in interval 1.
Samples -0.2, -0.2, -0.1, and -0.1 are in interval 2.
Samples 0.7, 0.7, and 1.0 are in interval 3.
Therefore:
The centroid of interval 1 is (-0.9 - 0.8 - 0.7 - 0.6)/4 = -0.75
The centroid of interval 2 is (-0.2 - 0.2 + 0.1 + 0.1)/4 = -0.05
The centroid of interval 3 is (0.7 + 0.7 + 1.0)/3 = 0.8
We update the
values with the above centroids. According to these new
values, we calculate the boundaries again:
boundary 1 = -inf.
boundary 2 = (-0.75 - 0.05)/2 = -0.4
boundary 3 = (-0.05+0.8)/2 = 0.375
boundary 4 = inf.
According to the new boundaries, we have the following grouping:
Samples -0.9, -0.8, -0.7, and -0.6 are in interval 1.
Samples -0.2, -0.2, -0.1, and -0.1 are in interval 2.
Samples 0.7, 0.7, and 1.0are in interval 3.
As you see, the clustering has not changed, therefore we will obtain the same
centroid values for the intervals. As a result, the algorithm must stop at this
point. The results are:
and
which form
a 3-level quantizer.
Please click here to download an example Matlab function which obtains the
minimum MSE scalar quantizer for a given set of data (in_sig) and a given number
of reproduction levels (L), and then quantizes the data and produces the quantized
output. After downloading, save it under your Matlab work directory. The usage is
explained in the file.
Notice that, sometimes the algorithm produces less number of reproduction levels
than L. This occurs when the initial levels or boundaries are not selected very well.
In general, the iterative algorithm may not converge to the optimum quantizer
levels, but it may stop at a locally optimum solution. You can experiment with this
function by generating various signals, e.g. a sinusoid, a saw-tooth wave, etc. and
using the signals as inputs to the above function. An example usage is:
>> in_sig=128*sin(0:0.01:20);
>> [sig_out, x, y] = lloyd_max_scalar(in_sig, 5);
>> plot(sig_out);
The above example obtains a 5-level quantizer with boundaries in vector x, and
reproduction levels in vector y. You can, then, see the output by plotting it. Notice
that, the input and output signals can also be 2D signals, which are images. So if
you have a gray-scale (monochrome) image, you can see its quantized version.
Click here do download the zipped version (121Kb) of a small test image, called
lena_small. After unzipping to the work directory, you can use this data instead of
in_sig, and obtain a quantized image at the output. To see the 2D quantization
result with 5 levels, you can type:
>> load lena_small; %The file must be saved in the same directory.
>> [sig_out, x, y] = lloyd_max_scalar(lena_small, 5);
>> image(sig_out); colormap(gray(255));
I strongly recommend that you change the quantization levels and observe how the
quality improves with larger quantization levels. You can measure the distortion
with the Matlab programs that you have prepared as the first homework of last
week (the MSE calculation) with some modifications to support 2D signals (images).
The basic optimality discussions end here. We will now proceed to the Vector
Quantization (VQ) which is formed by quantizing groups of data instead of scalars
one by one. It is noticeable that the VQ optimality conditions are just generalized
versions of the scalar quantizer optimality conditions.
An important issue regarding quantization is that, neither scalar nor vector
quantization is commonly used "as is" inside a coding scheme. Usully, there are are
predictive coding and entropy coding blocks attached inside the coding scheme, and
the overall compression success is determined by their hybrid use. However, we will
cover the predictive coding and entropy coding issues separately in this course.