Parameter estimation of two dimensional two component
Gaussian Mixtures
Nilantha Katugampala and Roland Wilson
0
Abstract
Multiresolution Gaussian Mixture Models (MGMM) can be used to represent image
and video data in video annotation and retrieval. Preliminary experiments were
carried out to estimate the model parameters for two-dimensional data. An iterative
algorithm similar to Expectation-Maximisation (EM) is used to estimate the model
parameters. The suitability of Akaike's Information Criterion (AIC) as a measure of
model fit is also evaluated. AIC was successful for most of the synthetic data sets
used in the experiments, however further work is required to develop a more
consistent criterion for model fit.
1
1. Introduction
Multiresolution Gaussian Mixture Models (MGMM) has been proposed for modelling
image and video data in video annotation and retrieval [1]. Preliminary experiments
were carried out to estimate the model parameters for two-dimensional data, and to
obtain a measure of model fit.
 0
Two-dimensional Gaussian data of 100 points were generated with mean   and
 0
 1 0
 [2]. Two different Gaussian populations of 100 points each were
covariance 
 0 1
obtained by applying different transformations to the initial data. Transformed
populations were merged to obtain a combined population of 200 points [3]. Then
extracting the original populations from the combined population was attempted by
using an iterative algorithm as follows:
1. Initialise µ1 , µ 2 , Σ1 , and Σ 2 of two-dimensional Gaussian populations.
2. Evaluate
1
′
a. g1 (xi ) =
exp − ( xi − µ1 ) Σ1−1 ( xi − µ1 ) 2
1
2



(2π ) 2 Σ1 2
b. g 2 ( xi ) =
1
(2π ) 2 Σ 2
2
1
2
′
exp − ( xi − µ 2 ) Σ −21 (xi − µ 2 ) 2


for all points, xi , 1 ≤ i ≤ 200 of the combined population.
3. Classify xi ; if g 2 ( xi ) > g1 (xi ) then xi belongs to g 2 else xi belongs to g1 .
4. Recompute µ1 , µ 2 , Σ1 , and Σ 2 from the classified xi , and go to step 2.
The above procedure was carried out until no further changes of xi between g1 and
g 2 were observed, i.e. convergence was reached. Where µ1 , µ 2 , Σ1 , and Σ 2 represent
the mean points and the covariances of the Gaussian populations.
The algorithm was tested with five different initial combined populations, each
obtained by merging two transformed Gaussian populations. For each initial
combined population two sets of experiments were carried out by differently
initialising µ1 , µ 2 , Σ1 , and Σ 2 .
Experiment A
100 random pairs of initial mean points, µ1 and µ 2 are used. The initial mean points
are uniformly distributed in a square area around the mean of the combined
population. The length, a of the square is given by equation 1. The Initial covariance
matrices, Σ1 , and Σ 2 are set equal to the covariance matrix of the combined
population, S g . The set of experiments carried out with these initial conditions are
referred to as Experiment A.
2
a = 6 Sg
1
4
(1)
Experiment B
Separation was also carried out 100 times with random pairs of initial covariances, Σ1 ,
and Σ 2 . The initial covariances are computed by scaling S g by a randomly selected
factor from the range 0.01, 0.02, 0.03, …, 1.00. The initial mean points, µ1 and µ 2
for this second set of experiments are set equal to the mean of the combined
population. The set of experiments carried out with these initial conditions are
referred to as Experiment B.
Akaike’s Information Criterion (AIC)
AIC [4] is computed as a measure of model fit to compare how well each separated
solution performs against their combined counterparts. The difference of the AIC
values by modelling the combined population using one Gaussian and a mixture of
two Gaussians is estimated. Two versions are derived with different assumptions,
when there are two Gaussian components. In Equation 2 (AIC1) a data point is
included in the likelihood estimation of a particular Gaussian component only if that
data point contributed to the estimation of its parameters, i.e. belonged to that
component in the classification step of the algorithm described above. In contrast
equation 3 (AIC2) evaluates the likelihood over all the data points for the complete
Gaussian mixture model. In either case positive values favour two components.

N  N
N 
N
 N
AIC1 =  ln S g + 5 −  1 ln S g1 − N1 ln 1  + 2 ln S g 2 − N 2 ln 2  + 11
N 2
 N 
2
 2

(2)
 N
  N

AIC 2 = − ∑ ln g (xi ) + 5 − − ∑ ln[π 1 g1 ( xi ) + π 2 g 2 (xi )] + 11
 i =1
  i =1

(3)
Further reading
The application of MGMM in computer vision as a representation of image data and
motion is described in [5]. The technique is illustrated by applying for image
segmentation. The use of Expectation Maximisation (EM) Algorithm to estimate the
parameters of finite mixtures is demonstrated in [6], and applied to image
segmentation. AIC and Minimum Description Length (MDL) information criteria are
employed to determine the number of segments in an image. An algorithm, which
integrates the EM algorithm and the information criterion, Minimum Message Length
(MML) has been, reported [7]. The inclusion of the information criterion within the
parameter estimation process increases the ability of the algorithm to escape from
local maxima. An introduction to model selection intended for nonspecialists is given
in [8]. The introduction is extended to explanations of AIC and Bayesian Information
3
Criterion (BIC). Application of AIC and MDL for selecting the optimal number of
components for a Gaussian mixture model is described in [9]. An intuitive derivation
of AIC is given in [10]. A geometrical interpretation of AIC is given in [11].
The following sections describe the details of the experiments and their results. The
final section draws some conclusions from the results of the experiments.
4
2. Similar mean points and covariances of different radii
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 2.1 Initial populations
The two initial populations are shown in blue and green in Figure 2.1. The mean of
the blue population is depicted by ‘+’ and the mean of the green population by ‘*’.
The dashed ellipses represent the covariances of the individual populations. The major
axes are equal to the square roots of the corresponding eigenvalues. The solid ellipse
and the black dot depict the covariance and the mean of the combined population
respectively. Table 2.1 shows the statistics of the initial populations.
Table 2.1 Statistics of the initial populations
Population
Blue
Green
Combined
Mean
-0.13641
-0.73186
0.00183
0.002167
-0.06729
-0.36485
Covariance
17.89735 -3.77466
-3.77466 28.06227
0.691264 -0.06027
-0.06027 4.258283
9.299083 -1.8921
-1.8921 16.29498
Samples
100
100
200
The two initial populations were merged and extracting the original populations using
the algorithm described in section 1. Introduction was carried out with initial
conditions of Experiments A and B.
There were 43 different solutions or convergences from both Experiments A and B.
Experiment A gave rise to 36 solutions and Experiment B gave rise to 8 solutions, i.e.
one solution came from both Experiments A and B. 4 pairs of initial mean points of
Experiment A did not separate the combined population, i.e. insufficient number of
points (less than three) in one component and the rest are in the other. The following
section presents some of the selected converged solutions. The solutions are selected
such that they resulted from more initial conditions for, either Experiment A or B, or
the aggregate of both of them, if the solution resulted from Experiments A and B.
5
2.1 Examples of separated populations
Example 1: resulted from Experiment A
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 2.2 Separated populations
Figure 2.2 depicts a selected converged solution for the initial populations shown in
Figure 2.1. Table 2.2 shows the statistics of the converged populations. 15 pairs of
initial mean points of Experiment A converged to this solution, with average number
of iterations for convergence 22.266667 and standard deviation 4.767482. Some of
these initial pairs of mean points are depicted in Figure 2.3. The ‘+’ and ‘*’ in each
colour represent each pair of random initial mean points.
Table 2.2 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
0.108565
1.997297
-0.336643
-3.982810
-0.06729
-0.36485
Covariance
5.230056 -0.372275
-0.372275 6.420510
15.411473 -5.830666
-5.830666 9.783339
9.299083 -1.8921
-1.8921 16.29498
Samples
121
79
200
6
10
10
8
5
6
4
0
2
0
-5
-2
-4
-10
-6
-15
-10
-8
-6
-4
-2
0
2
4
6
8
10
-8
-8
-6
-4
-2
0
2
4
6
Figure 2.3 Examples of successful initial pairs of mean points
The estimated AIC values are:
AIC1 = -40.873256
AIC2 = -6.097164
Example 2: resulted from Experiment A
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 2.4 Separated populations
7
8
10
Table 2.3 Statistics of the separated populations
Population
Mean
-0.845951
-4.734508
0.243083
1.376908
-0.06729
-0.36485
Blue
Green
Combined
Covariance
17.220419 -9.174362
-9.174362 11.459727
5.803619 -0.886202
-0.886202 7.577729
9.299083 -1.8921
-1.8921 16.29498
Samples
57
143
200
Figure 2.4 depicts another selected converged solution for the initial populations
shown in Figure 2.1. Table 2.3 shows the statistics of the converged populations. 9
pairs of initial mean points of Experiment A converged to this solution, with average
number of iterations for convergence 9.111111 and standard deviation 2.131481.
These initial pairs of mean points are depicted in Figure 2.5. The ‘+’ and ‘*’ in each
colour represent each pair of random initial mean points.
10
6
8
4
6
2
4
0
2
-2
0
-4
-2
-6
-4
-8
-6
-10
-8
-12
-10
-8
-6
-4
-2
0
2
4
6
-12
-8
-6
-4
-2
0
2
4
Figure 2.5 Examples of successful initial pairs of mean points
The estimated AIC values are:
AIC1 = -29.856318
AIC2 = -2.940483
8
6
Example 3: resulted from Experiment B
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 2.6 Separated populations
Figure 2.6 depicts another selected converged solution for the initial populations
shown in Figure 2.1. Table 2.4 shows the statistics of the converged populations. 25
pairs of initial covariances of Experiment B converged to this solution, with average
number of iterations for convergence 6.080000 and standard deviation 0.890842.
Some of these initial pairs of covariances are depicted in Figure 2.7. Each colour
represents a pair.
Table 2.4 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
-0.022878
-0.125641
-0.113518
-0.613813
-0.06729
-0.36485
Covariance
0.530273 -0.234742
-0.234742 3.001635
18.421613 -3.639661
-3.639661 30.009360
9.299083 -1.8921
-1.8921 16.29498
Samples
102
98
200
The estimated AIC values are:
AIC1 = 24.941031
AIC2 = 48.409106
9
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-5
-4
-3
-2
-1
0
1
2
3
4
-8
-5
5
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-5
-4
-3
-2
-1
0
1
2
3
4
-8
-5
5
-4
-3
-2
-1
0
1
2
3
4
5
-4
-3
-2
-1
0
1
2
3
4
5
Figure 2.7 Examples of successful initial pairs of covariances
Example 4: resulted from Experiment B
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 2.8 Separated populations
10
Table 2.5 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
0.006140
-0.088675
-0.146843
-0.664030
-0.06729
-0.36485
Covariance
0.563759 -0.177817
-0.177817 3.020742
18.750181 -3.794999
-3.794999 30.503256
9.299083 -1.8921
-1.8921 16.29498
Samples
104
96
200
Figure 2.8 depicts another selected converged solution for the initial populations
shown in Figure 2.1. Table 2.5 shows the statistics of the converged populations. 19
pairs of initial covariances of Experiment B converged to this solution, with average
number of iterations for convergence 3.526316 and standard deviation 0.499307.
Some of these initial pairs of covariances are depicted in Figure 2.9. Each colour
represents a pair.
The estimated AIC values are:
AIC1 = 24.978408
AIC2 = 48.466776
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-5
-4
-3
-2
-1
0
1
2
3
4
5
-8
-5
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-5
-4
-3
-2
-1
0
1
2
3
4
5
-8
-5
-4
-3
-2
-1
0
1
2
3
4
5
-4
-3
-2
-1
0
1
2
3
4
5
Figure 2.9 Examples of successful initial pairs of covariances
11
Example 5: resulted from Experiments A and B
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 2.10 Separated populations
Table 2.6 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
-0.011093
-0.229970
-0.135979
-0.529694
-0.06729
-0.36485
Covariance
0.584931 -0.062541
-0.062541 3.609172
19.941135 -4.148803
-4.148803 31.750435
9.299083 -1.8921
-1.8921 16.29498
Samples
110
90
200
Figure 2.10 depicts another selected converged solution for the initial populations
shown in Figure 2.1. Table 2.6 shows the statistics of the converged populations. 10
pairs of initial mean points of Experiment A converged to this solution, with average
number of iterations for convergence 17.500000 and standard deviation 2.655184.
These initial pairs of mean points are depicted in Figure 2.11. The ‘+’ and ‘*’ in each
colour represent each pair of random initial mean points. 36 pairs of initial
covariances of Experiment B also converged to this solution, with average number of
iterations for convergence 5.361111 and standard deviation 0.535038. Some of these
initial pairs of covariances are depicted in Figure 2.12. Each colour represents a pair.
12
10
8
8
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-12
-10
-8
-6
-4
-2
0
2
4
6
-10
-15
8
-10
-5
0
5
10
Figure 2.11 Examples of successful initial pairs of mean points
The estimated AIC values are:
AIC1 = 26.022754
AIC2 = 48.602174
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-5
-4
-3
-2
-1
0
1
2
3
4
5
-8
-5
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-5
-4
-3
-2
-1
0
1
2
3
4
5
-8
-5
-4
-3
-2
-1
0
1
2
3
4
5
-4
-3
-2
-1
0
1
2
3
4
5
Figure 2.12 Examples of successful initial pairs of covariances
13
The 4 pairs of initial mean points of Experiment A that did not separate the combined
population in Figure 2.1 are depicted in Figure 2.13. These initial conditions resulted
in insufficient number of points (less than three) in one component and the rest are in
the other.
10
5
0
-5
-10
-15
-10
-8
-6
-4
-2
0
2
4
6
Figure 2.13 Examples of unsuccessful initial pairs of mean points
14
3. Non overlapping populations
10
5
0
-5
-10
-15
-25
-20
-15
-10
-5
0
5
10
Figure 3.1 Initial populations
The two initial populations are shown in blue and green in Figure 3.1. The mean of
the blue population is depicted by ‘+’ and the mean of the green population by ‘*’.
The dashed ellipses represent the covariances of the individual populations. The major
axes are equal to the square roots of the corresponding eigenvalues. The solid ellipse
and the black dot depict the covariance and the mean of the combined population
respectively. Table 3.1 shows the statistics of the initial populations.
Table 3.1 Statistics of the initial populations
Population
Mean
-0.289687
Blue
0.522580
-12.319016
Green
-12.247319
-6.304351
Combined
-5.862370
Covariance
5.255479 -7.694632
-7.694632 15.509217
17.290340 6.193848
6.193848 3.241064
47.449096 37.652935
37.652935 50.142720
Samples
100
100
200
The two initial populations were merged and extracting the original populations using
the algorithm described in section 1. Introduction was carried out with initial
conditions of Experiments A and B.
There was only 1 solution or convergence from both Experiments A and B. In fact the
solution is identical to the initial populations. 10 pairs of initial mean points of
Experiment A and 8 pairs of initial covariances of Experiment B did not separate the
combined population, i.e. insufficient number of points (less than three) in one
component and the rest are in the other. The following section presents the converged
solution.
15
3.1 The Separated population resulted from Experiments A and B
10
5
0
-5
-10
-15
-25
-20
-15
-10
-5
0
5
10
Figure 3.2 Separated populations
Table 3.2 Statistics of the separated populations
Population
Mean
-0.289687
Blue
0.522580
-12.319016
Green
-12.247319
-6.304351
Combined
-5.862370
Covariance
5.255479 -7.694632
-7.694632 15.509217
17.290340 6.193848
6.193848 3.241064
47.449096 37.652935
37.652935 50.142720
Samples
100
100
200
Figure 3.2 depicts the converged solution for the initial populations shown in Figure
3.1. Table 3.2 shows the statistics of the converged populations. 90 pairs of initial
mean points of Experiment A converged to this solution, with average number of
iterations for convergence 8.111111 and standard deviation 3.790176. Some of these
initial pairs of mean points are depicted in Figure 3.3. The ‘+’ and ‘*’ in each colour
represent each pair of random initial mean points. 92 pairs of initial covariances of
Experiment B also converged to this solution, with average number of iterations for
convergence 8.576087 and standard deviation 1.854687. Some of these initial pairs of
covariances are depicted in Figure 3.4. Each colour represents a pair.
16
15
15
10
10
5
5
0
0
-5
-5
-10
-10
-15
-15
-20
-25
-20
-15
-10
-5
0
5
-20
-25
-20
-15
-10
-5
0
5
10
Figure 3.3 Examples of successful initial pairs of mean points
5
5
0
0
-5
-5
-10
-10
-15
-20
-15
-10
-5
0
5
-15
-20
5
5
0
0
-5
-5
-10
-10
-15
-20
-15
-10
-5
0
5
-15
-20
-15
-10
-5
0
5
-15
-10
-5
0
5
Figure 3.4 Examples of successful initial pairs of covariances
The estimated AIC values are:
AIC1 = 243.378149
AIC2 = 243.388687
17
The 10 pairs of initial mean points of Experiment A that did not separate the
combined population in Figure 2.1 are depicted in Figure 3.5. These initial conditions
resulted in insufficient number of points (less than three) in one component and the
rest are in the other.
15
10
10
5
5
0
0
-5
-5
-10
-10
-15
-15
-20
-20
-25
-25
-20
-15
-10
-5
0
5
-25
-25
10
-20
-15
-10
-5
0
5
10
Figure 3.5 Examples of unsuccessful initial pairs of mean points
The 8 pairs of initial covariances of Experiment B that did not separate the combined
population in Figure 2.1 are depicted in Figure 3.6. These initial conditions resulted in
insufficient number of points (less than three) in one component and the rest are in the
other.
5
5
0
0
-5
-5
-10
-10
-15
-20
-15
-10
-5
0
5
-15
-20
5
5
0
0
-5
-5
-10
-10
-15
-20
-15
-10
-5
0
5
-15
-20
-15
-10
-5
0
5
-15
-10
-5
0
5
Figure 3.6 Examples of unsuccessful initial pairs of covariances
18
4. Overlapping populations
5
0
-5
-10
-15
-20
-15
-10
-5
0
5
Figure 4.1 Initial populations
The two initial populations are shown in blue and green in Figure 4.1. The mean of
the blue population is depicted by ‘+’ and the mean of the green population by ‘*’.
The dashed ellipses represent the covariances of the individual populations. The major
axes are equal to the square roots of the corresponding eigenvalues. The solid ellipse
and the black dot depict the covariance and the mean of the combined population
respectively. Table 4.1 shows the statistics of the initial populations.
Table 4.1 Statistics of the initial populations
Population
Blue
Green
Combined
Mean
0.130864
-0.468696
-7.992005
-9.997833
-3.930571
-5.233264
Covariance
5.349469 -7.930679
-7.930679 15.716679
19.316026 8.396025
8.396025 4.258283
28.827996 19.583653
19.583653 32.688591
Samples
100
100
200
The two initial populations were merged and extracting the original populations using
the algorithm described in section 1. Introduction was carried out with initial
conditions of Experiments A and B.
There were 4 different solutions or convergences from both Experiments A and B.
Experiment A gave rise to 4 solutions and Experiment B gave rise to 1 solution, i.e.
one solution came from both Experiments A and B. 11 pairs of initial mean points of
Experiment A and 8 pairs of initial covariances of Experiment B did not separate the
combined population, i.e. insufficient number of points (less than three) in one
component and the rest are in the other. The following section presents the converged
solutions.
19
4.1 The separated populations
Example 1: resulted from Experiment A
5
0
-5
-10
-15
-20
-15
-10
-5
0
5
Figure 4.2 Separated populations
Figure 4.2 depicts a converged solution for the initial populations shown in Figure
4.1. Table 4.2 shows the statistics of the converged populations. 4 pairs of initial mean
points of Experiment A converged to this solution, with average number of iterations
for convergence 2.000000 and standard deviation 0.000000. These initial pairs of
mean points are depicted in Figure 4.3. The ‘+’ and ‘*’ in each colour represent each
pair of random initial mean points.
Table 4.2 Statistics of the separated populations
Population
Mean
5.236881
Blue
-10.222348
-4.117661
Green
-5.131446
-3.930571
Combined
-5.233264
Covariance
0.188004 -0.110598
-0.110598 0.382498
27.662336 20.938038
20.938038 32.829553
28.827996 19.583653
19.583653 32.688591
Samples
4
196
200
The estimated AIC values are:
AIC1 = 9.700011
AIC2 = 9.723727
20
5
0
-5
-10
-15
-20
-15
-10
-5
0
5
10
Figure 4.3 Examples of successful initial pairs of mean points
Example 2: resulted from Experiment A
5
0
-5
-10
-15
-20
-15
-10
-5
0
5
Figure 4.4 Separated populations
Figure 4.4 depicts another converged solution for the initial populations shown in
Figure 4.1. Table 4.3 shows the statistics of the converged populations. 3 pairs of
initial mean points of Experiment A converged to this solution, with average number
of iterations for convergence 8.333333 and standard deviation 0.942809. These initial
pairs of mean points are depicted in Figure 4.5. The ‘+’ and ‘*’ in each colour
represent each pair of random initial mean points.
The estimated AIC values are:
AIC1 = -12.832056
AIC2 = 10.115188
21
Table 4.3 Statistics of the separated populations
Population
Mean
-6.278740
-4.993368
-0.016954
-5.633091
-3.930571
-5.233264
Blue
Green
Combined
Covariance
25.652553 30.724216
30.724216 47.086077
9.614170 3.519682
3.519682 8.437002
28.827996 19.583653
19.583653 32.688591
Samples
125
75
200
5
0
-5
-10
-15
-20
-20
-15
-10
-5
0
5
10
15
Figure 4.5 Examples of successful initial pairs of mean points
Example 3: resulted from Experiment A
5
0
-5
-10
-15
-20
-15
-10
-5
0
5
Figure 4.6 Separated populations
Figure 4.6 depicts another converged solution for the initial populations shown in
Figure 4.1. Table 4.4 shows the statistics of the converged populations. 1 pair of
22
initial mean points of Experiment A converged to this solution, with average number
of iterations for convergence 2.000000. This initial pair of mean points is depicted in
Figure 4.7.
Table 4.4 Statistics of the separated populations
Population
Mean
5.354657
Blue
-10.571228
-4.071970
Green
-5.151976
-3.930571
Combined
-5.233264
Covariance
0.195188 0.016894
0.016894 0.023128
27.931109 20.647903
20.647903 32.745510
28.827996 19.583653
19.583653 32.688591
Samples
3
197
200
The estimated AIC values are:
AIC1 = 9.403133
AIC2 = 9.407357
5
0
-5
-10
-15
-20
-14
-12
-10
-8
-6
-4
-2
0
2
4
Figure 4.7 An Example of a successful initial pair of mean points
23
Example 4: resulted from Experiments A and B
5
0
-5
-10
-15
-20
-15
-10
-5
0
5
Figure 4.8 Separated populations
Table 4.5 Statistics of the separated populations
Population
Mean
0.182389
Blue
-0.550476
-8.211406
Green
-10.107187
-3.930571
Combined
-5.233264
Covariance
5.380257 -7.985082
-7.985082 15.743110
17.300327 7.366931
7.366931 3.747047
28.827996 19.583653
19.583653 32.688591
Samples
102
98
200
Figure 4.8 depicts another converged solution for the initial populations shown in
Figure 4.1. Table 4.5 shows the statistics of the converged populations. 81pairs of
initial mean points of Experiment A converged to this solution, with average number
of iterations for convergence 11.259259 and standard deviation 6.653279. Some of
these initial pairs of mean points are depicted in Figure 4.9. The ‘+’ and ‘*’ in each
colour represent each pair of random initial mean points. 92 pairs of initial
covariances of Experiment B also converged to this solution, with average number of
iterations for convergence 9.195652 and standard deviation 0.679500. Some of these
initial pairs of covariances are depicted in Figure 4.10. Each colour represents a pair.
The estimated AIC values are:
AIC1 = 217.402522
AIC2 = 218.516788
24
10
10
5
5
0
0
-5
-5
-10
-10
-15
-15
-20
-15
-10
-5
0
5
10
-20
-20
-15
-10
-5
0
5
10
Figure 4.9 Examples of successful initial pairs of mean points
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-10
-10
-12
-12
-14
-14
-12
-10
-8
-6
-4
-2
0
2
4
-14
-14
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-10
-10
-12
-12
-14
-14
-12
-10
-8
-6
-4
-2
0
2
4
-14
-14
-12
-10
-8
-6
-4
-2
0
2
4
-12
-10
-8
-6
-4
-2
0
2
4
Figure 4.10 Examples of successful initial pairs of covariances
25
The 11 pairs of initial mean points of Experiment A that did not separate the
combined population in Figure 4.1 are depicted in Figure 4.11. These initial
conditions resulted in insufficient number of points (less than three) in one component
and the rest are in the other.
10
10
5
5
0
0
-5
-5
-10
-10
-15
-15
-20
-20
-15
-10
-5
0
5
10
-20
-20
-15
-10
-5
0
5
10
Figure 4.11 Examples of unsuccessful initial pairs of mean points
The 8 pairs of initial covariances of Experiment B that did no separate the combined
population in Figure 4.1 are depicted in Figure 4.12.
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-10
-10
-12
-12
-14
-14
-12
-10
-8
-6
-4
-2
0
2
4
-14
-14
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-10
-10
-12
-12
-14
-14
-12
-10
-8
-6
-4
-2
0
2
4
-14
-14
-12
-10
-8
-6
-4
-2
0
2
4
-12
-10
-8
-6
-4
-2
0
2
4
Figure 4.12 Example of unsuccessful initial pairs of covariances
26
5. Similar mean points and covariances of different directions
10
5
0
-5
-10
-10
-5
0
5
10
15
Figure 5.1 Initial populations
The two initial populations are shown in blue and green in Figure 5.1. The mean of
the blue population is depicted by ‘+’ and the mean of the green population by ‘*’.
The dashed ellipses represent the covariances of the individual populations. The major
axes are equal to the square roots of the corresponding eigenvalues. The solid ellipse
and the black dot depict the covariance and the mean of the combined population
respectively. Table 5.1 shows the statistics of the initial populations.
Table 5.1 Statistics of the initial populations
Population
Blue
Green
Combined
Mean
-0.029255
-0.130021
-0.182076
-0.175361
-0.105665
-0.152691
Covariance
6.619194 -8.915960
-8.915960 15.605562
25.260471 9.415431
9.415431 4.310522
15.945671 0.251468
0.251468 9.958556
Samples
100
100
200
The two initial populations were merged and extracting the original populations using
the algorithm described in section 1. Introduction was carried out with initial
conditions of Experiments A and B.
There were 20 different solutions or convergences from both Experiments A and B.
Experiment A gave rise to 19 solutions and Experiment B gave rise to 2 solutions, i.e.
one solution came from both Experiments A and B. 2 pairs of initial mean points of
Experiment A did not separate, i.e. insufficient number of points (less than three) in
one component and the rest are in the other. The following section presents some of
the selected converged solutions. The solutions are selected such that they resulted
from more initial conditions for, either Experiment A or B, or the aggregate of both of
them, if the solution resulted from Experiments A and B.
27
5.1 Examples of separated populations
Example 1: resulted from Experiment A
10
5
0
-5
-10
-10
-5
0
5
10
15
Figure 5.2 Separated populations
Figure 5.2 depicts a selected converged solution for the initial populations shown in
Figure 5.1. Table 5.2 shows the statistics of the converged populations. 19 pairs of
initial mean points of Experiment A converged to this solution, with average number
of iterations for convergence 16.157895 and standard deviation 1.724839. Some of
these initial pairs of mean points are depicted in Figure 5.3. The ‘+’ and ‘*’ in each
colour represent each pair of random initial mean points.
Table 5.2 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
-0.262281
-0.102120
0.047849
-0.202260
-0.105665
-0.152691
Covariance
25.529872 9.592733
9.592733 4.250640
6.503647 -8.889448
-8.889448 15.548479
15.945671 0.251468
0.251468 9.958556
Samples
99
101
200
The estimated AIC values are:
AIC1 = 67.017429
AIC2 = 92.308354
28
10
15
8
10
6
4
5
2
0
0
-2
-4
-5
-6
-8
-10
-5
0
5
10
-10
-15
15
-10
-5
0
5
Figure 5.3 Examples of successful initial pairs of mean points
Example 2: resulted from Experiment A
10
5
0
-5
-10
-10
-5
0
5
10
15
Figure 5.4 Separated populations
Figure 5.4 depicts another selected converged solution for the initial populations
shown in Figure 5.1. Table 5.3 shows the statistics of the converged populations. 13
pairs of initial mean points of Experiment A converged to this solution, with average
number of iterations for convergence 9.769231 and standard deviation 2.606319.
Some of these initial pairs of mean points are depicted in Figure 5.5. The ‘+’ and ‘*’
in each colour represent each pair of random initial mean points.
29
10
Table 5.3 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
0.767138
1.876953
-1.361650
-3.073399
-0.105665
-0.152691
Covariance
13.033658 -1.151955
-1.151955 4.235560
17.462402 -3.946541
-3.946541 3.735555
15.945671 0.251468
0.251468 9.958556
Samples
118
82
200
The estimated AIC values are:
AIC1 = -29.976402
AIC2 = -8.092659
8
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-10
-10
-12
-10
-8
-6
-4
-2
0
2
4
6
-12
-15
-10
-5
0
5
Figure 5.5 Examples of successful initial pairs of mean points
30
10
15
Example 3: resulted from Experiment A
10
5
0
-5
-10
-10
-5
0
5
10
15
Figure 5.6 Separated populations
Figure 5.6 depicts another selected converged solution for the initial populations
shown in Figure 5.1. Table 5.4 shows the statistics of the converged populations. 11
pairs of initial mean points of Experiment A converged to this solution, with average
number of iterations for convegence 11.636364 and standard deviation 2.993105.
These initial pairs of mean points are depicted in Figure 5.7. The ‘+’ and ‘*’ in each
colour represent each pair of random initial mean points.
Table 5.4 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
Covariance
2.505050 7.502316 2.186188
-0.884802 2.186188 8.855087
-3.566380 6.126537 3.578999
0.817782 3.578999 9.768981
-0.105665 15.945671 0.251468
-0.152691 0.251468 9.958556
Samples
114
86
200
The estimated AIC values are:
AIC1 = -36.462414
AIC2 = -12.850395
31
10
10
8
8
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-10
-15
-10
-5
0
5
10
-8
-15
15
-10
-5
0
5
Figure 5.7 Examples of successful initial pairs of mean points
Example 4: resulted from Experiment B
10
5
0
-5
-10
-10
-5
0
5
10
15
Figure 5.8 Separated populations
Figure 5.8 depicts another selected converged solution for the initial populations
shown in Figure 5.1. Table 5.5 shows the statistics of the converged populations. 13
pairs of initial covariances of Experiment B converged to this solution, with average
number of iterations for convergence 3.307692 and standard deviation 0.461538.
Some of these initial pairs of covariances are depicted in Figure 5.9. Each colour
represents a pair.
32
10
Table 5.5 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
-0.101504
-0.285712
-0.108440
-0.064010
-0.105665
-0.152691
Covariance
2.085454 -0.271056
-0.271056 1.333963
25.185796 0.600432
0.600432 15.688624
15.945671 0.251468
0.251468 9.958556
Samples
80
120
200
The estimated AIC values are:
AIC1 = -32.433444
AIC2 = -0.615810
4
4
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
-5
-6
-4
-2
0
2
4
6
-5
-6
4
4
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
-5
-6
-4
-2
0
2
4
6
-5
-6
-4
-2
0
2
4
6
-4
-2
0
2
4
6
Figure 5.9 Examples of successful initial pairs of covariances
33
Example 5: resulted from Experiments A and B
10
5
0
-5
-10
-10
-5
0
5
10
15
Figure 5.10 Separated populations
Figure 5.10 depicts another selected converged solution for the initial populations
shown in Figure 5.1. Table 5.6 shows the statistics of the converged populations. 17
pairs of initial mean points of Experiment A converged to this solution, with average
number of iterations for convergence 21.529412 and standard deviation 4.827832.
Some of these initial pairs of mean points are depicted in Figure 5.11. The ‘+’ and ‘*’
in each colour represent each pair of random initial mean points. 87 pairs of initial
covariances of Experiment B also converged to this solution, with average number of
iterations for convergence 25.241379 and standard deviation 1.694925. Some of these
initial pairs of covariances are depicted in Figure 5.12. Each colour represents a pair.
Table 5.6 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
-0.312857
-0.115919
0.078070
-0.185300
-0.105665
-0.152691
Covariance
26.943785 10.072134
10.072134 4.463628
6.120799 -8.444677
-8.444677 14.829154
15.945671 0.251468
0.251468 9.958556
Samples
94
106
200
34
10
10
8
8
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-10
-10
-8
-6
-4
-2
0
2
4
6
8
-10
-10
10
-8
-6
-4
-2
0
2
4
Figure 5.11 Examples of successful initial pairs of mean points
The estimated AIC values are:
AIC1 = 67.207818
AIC2 = 92.509745
4
4
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
-5
-6
-4
-2
0
2
4
6
-5
-6
4
4
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
-5
-6
-4
-2
0
2
4
6
-5
-6
-4
-2
0
2
4
6
-4
-2
0
2
4
6
Figure 5.12 Examples of successful initial pairs of covariances
35
6
The 2 pairs of initial mean points of Experiment A that did not separated the
combined population in Figure 5.1 are depicted in Figure 5.13. These initial
conditions resulted in insufficient number of points (less than three) in one component
and the rest are in the other.
10
5
0
-5
-8
-6
-4
-2
0
2
4
6
8
10
12
Figure 5.13 Examples of unsuccessful initial pairs of mean points
36
6. Parallel populations
10
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 6.1 Initial populations
The two initial populations are shown in blue and green in Figure 6.1. The mean of
the blue population is depicted by ‘+’ and the mean of the green population by ‘*’.
The dashed ellipses represent the covariances of the individual populations. The major
axes are equal to the square roots of the corresponding eigenvalues. The solid ellipse
and the black dot depict the covariance and the mean of the combined population
respectively. Table 6.1 shows the statistics of the initial populations.
Table 6.1 Statistics of the initial populations
Population
Blue
Green
Combined
Mean
-7.997678
-0.406998
8.036238
-0.621375
0.019280
-0.514187
Covariance
0.767930 0.019526
0.019526 21.853479
0.918393 -0.007768
-0.007768 24.077201
65.114783 -0.853445
-0.853445 22.976829
Samples
100
100
200
The two initial populations were merged and extracting the original populations using
the algorithm described in section 1. Introduction was carried out with initial
conditions of Experiments A and B.
There were 28 different solutions or convergences from both Experiments A and B.
Experiment A gave rise to 12 solutions and Experiment B gave rise to 16 solutions,
i.e. none of the solutions came from both Experiments A and B. 6 pairs of initial mean
points of Experiment A and 41 initial covariances of Experiment B did not separate
the combined population, i.e. insufficient number of points (less than three) in one
component and the rest are in the other. The following section presents some of the
selected converged solutions. The solutions are selected such that they resulted from
more initial conditions for either Experiment A or B.
37
6.1 Examples of separated populations
Example 1: resulted from Experiment A
10
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 6.2 Separated populations
Figure 6.2 depicts a selected converged solution for the initial populations shown in
Figure 6.1. In fact the solution is identical to the original populations. Tabel 6.2 shows
the statistics of the converged populations. 75 pairs of initial mean points of
Experiment A converged to this solution, with average number of iterations for
convergence 7.840000 and standard deviation 4.242766. Some of these initial pairs of
mean points are depicted in Figure 6.3. The ‘+’ and ‘*’ in each colour represent each
pair of random initial mean points.
Table 6.2 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
8.036238
-0.621375
-7.997678
-0.406998
0.019280
-0.514187
Covariance
0.918393 -0.007768
-0.007768 24.077201
0.767930 0.019526
0.019526 21.853479
65.114783 -0.853445
-0.853445 22.976829
Samples
100
100
200
38
20
15
15
10
10
5
5
0
0
-5
-5
-10
-10
-15
-20
-20
-15
-10
-5
0
5
10
15
-15
-15
20
-10
-5
0
5
10
15
Figure 6.3 Examples of successful initial pairs of mean points
The estimated AIC values are:
AIC1 = 290.564960
AIC2 = 290.564960
Example 2: resulted from Experiment A
10
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 6.4 Separated populations
39
Figure 6.4 depicts another selected converged solution for the initial populations
shown in Figure 6.1. Tabel 6.3 shows the statistics of the converged populations. 6
pairs of initial mean points of Experiment A converged to this solution, with average
number of interations for convergence 7.000000 and sandard deviation 4.760952.
These initial pairs of mean points are depicted in Figure 6.5. The ‘+’ and ‘*’ in each
colour represent each pair of random initial mean points.
Table 6.3 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
-1.173667
-4.355067
1.165445
3.176071
0.019280
-0.514187
Covariance
63.754093 -5.457491
-5.457491 7.628010
63.741106 -5.061874
-5.061874 9.931892
65.114783 -0.853445
-0.853445 22.976829
Samples
98
102
200
15
10
5
0
-5
-10
-15
-20
-15
-10
-5
0
5
10
15
Figure 6.5 Examples of successful initial pairs of mean points
The estimated AIC values are:
AIC1 = -40.506219
AIC2 = -14.646831
40
Example 3: resulted from Experiment B
10
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 6.6 Separated populations
Figure 6.6 depicts another selected converged solution for the initial populations
shown in Figure 6.1. Table 6.4 shows the statistics of the converged populations. 16
pairs of initial covariances of Experiment B converged to this solution, with average
number of iterations for convergence 12.750000 and standard deviation 0.661438.
Some of these initial pairs of covariances are depicted in Figure 6.7. Each colour
represents a pair.
Table 6.4 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
-1.367471
0.047295
1.748823
-1.214462
0.019280
-0.514187
Covariance
63.201476 -1.388535
-1.388535 38.484512
62.111278 1.996177
1.996177 2.752211
65.114783 -0.853445
-0.853445 22.976829
Samples
111
89
200
The estimated AIC values are:
AIC1 = -72.807954
AIC2 = -23.447301
41
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-10
-8
-6
-4
-2
0
2
4
6
8
-6
-10
10
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-10
-8
-6
-4
-2
0
2
4
6
8
-6
-10
10
-8
-6
-4
-2
0
2
4
6
8
10
-8
-6
-4
-2
0
2
4
6
8
10
Figure 6.7 Examples of successful initial pairs of covariances
Example 4: resulted from Experiment B
10
5
0
-5
-10
-15
-10
-5
0
5
10
15
Figure 6.8 Separated populations
42
Figure 6.8 depicts another selected converged solution for the initial populaions
shown in Figure 6.1. Table 6.5 shows the statistics of the converged populations. 7
pairs of initial covariances of Experiment B converged to this solution, with average
number of iterations for convergence 12.857143 and standard deviation 2.166536.
These initial pairs of covariances are depicted in Figure 6.9. Each colour represents a
pair.
Table 6.5 Statistics of the separated populations
Population
Blue
Green
Combined
Mean
1.340049
-4.028986
-1.500314
3.529722
0.019280
-0.514187
Covariance
63.036136 3.710449
3.710449 8.182409
63.190146 5.381791
5.381791 9.431642
65.114783 -0.853445
-0.853445 22.976829
Samples
107
93
200
The estimated AIC values are:
AIC1 = -40.649783
AIC2 = -13.701798
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-10
-8
-6
-4
-2
0
2
4
6
8
10
-6
-10
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-10
-8
-6
-4
-2
0
2
4
6
8
10
-6
-10
-8
-6
-4
-2
0
2
4
6
8
10
-8
-6
-4
-2
0
2
4
6
8
10
Figure 6.9 Examples of successful initial pairs of cavariances
43
The 6 pairs of initial mean points of Experiment A that did not separate the combined
population in Figure 6.1 are depicted in Figure 6.10. These initial conditions resulted
in insufficient number of points (less than three) in one component and the rest are in
the other.
20
15
10
5
0
-5
-10
-15
-20
-20
-15
-10
-5
0
5
10
15
20
Figure 6.10 Examples of unsuccessful initial pairs of mean points
8 of the 41 pairs of initial covariances of Experiment B that did no separate the
combined population in Figure 6.1 are depicted in Figure 6.11.
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-10
-8
-6
-4
-2
0
2
4
6
8
10
-6
-10
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-10
-8
-6
-4
-2
0
2
4
6
8
10
-6
-10
-8
-6
-4
-2
0
2
4
6
8
10
-8
-6
-4
-2
0
2
4
6
8
10
Figure 6.11 Example of unsuccessful initial pairs of covariances
44
7. Conclusions
The combined population presented in section 3. Non overlapping populations
resulted in a unique solution for all the initial conditions of Experiments A and B,
excluding the initial conditions, which did not separate the population. All the other
combined populations resulted in more than one solution. However the combined
population in section 3 is the easiest to separate, since it consists of individual
populations with different mean points as well as covariances, without any overlap of
the data points. As the overlap between the individual populations increases there are
more solutions. However not all of them are equally acceptable.
An example of more solutions without any overlap is given in section 6. Parallel
populations. In this example the y coordinates of the mean points are equal as well as
the covariances. Thus an acceptable solution may only be achieved by initial mean
points with offsets in the x direction.
The initial conditions, which did not separate the combined populations had one
Gaussian distribution with a higher probability than the other one for all the points.
This is possible due to one mean point being further away than the other one from
both individual populations in the case of Experiment A, and one covariance being
concentrated in an area without any points in the case of Experiment B.
Positive AIC values do not necessarily indicate a good model fit. Example 2 of
section 4. Overlapping populations demonstrates a very poor fit of the model
parameters for the given data. However AIC2 is positive. AIC1 is always less than or
equal to AIC2 and in this example it is negative, indicating the poor fit. A negative
AIC1 reliably indicates a poor fit. However neither a positive AIC1 nor a positive
AIC2 reliably indicate a good fit. Examples 1 and 3 of section 4. Overlapping
populations demonstrates poor model fits, and both AIC1 and AIC2 are positive.
However in these examples one population has only about 2% or less of the total data
points. Therefore a combination of AIC and the proportions of the data points have
the potential to develop a reliable measure of model fit. The AIC values of clearly
good model fits are remarkably higher than the ambiguous cases, usually greater than
50. Furthermore AIC1 and AIC2 become closer as the overlap between the two
Gaussian components of the mixture model reduces.
Therefore an improved criterion may be used to obtain a measure of model fit and to
identify acceptable solutions. However there is no guarantee that the best possible
solution is among the solutions achieved in the process since the dependence of
convergences on the initial conditions.
45
8. Appendix
Matlab code 1
% Generates the two initial Gaussian populations and save them in
% gauss1.txt and gauss2.txt
% The mean points and covariances of initial populations as well as %
the combined population are saved in meancov.txt
% The combined population is separated using an iterative algorithm,
% with 200 different initial conditions
% The initial and final means and covariances, number of iterations %
of convergence, number of samples in converged populations and the %
AIC values are % saved in results.txt
% All the final converged solutions are also saved in the files
% f<i>gauss1.txt and f<i>gauss2.txt, where 1 <= i <= 200
clear; hf = 1;
% handle of the current figure
n1 = 100;
component
n2 = 100;
component
n = n1+n2;
N_itr = 100;
classification
x1 = randn(2,n1);
x2 = randn(2,n2);
% number of data points in the first
% number of data points in the second
% number of iterations in the
% 2*n1 Gaussian matrix
% 2*n2 Gaussian matrix
As1 =
Ah1 =
theta
Ar1 =
[1,0;0,5];
%
[1,0;2,1];
%
= 0;
%
[cos(theta), -sin(theta);
scaling
shearing
rotation
sin(theta), cos(theta)];
As2 =
Ah2 =
theta
Ar2 =
[1,0;0,5];
%
[1,2;0,1];
%
= 0;
%
[cos(theta), -sin(theta);
scaling
shearing
rotation
sin(theta), cos(theta)];
c1 = [-8,0;0,0];
c2 = [8,0;0,0];
mu1 = ones(2,n1);
mu1 = c1*mu1;
mu2 = ones(2,n2);
mu2 = c2*mu2;
% translation
% generate two 2-D Gaussian populations
g1 = Ar1*As1*x1 + mu1;
g2 = Ar2*As2*x2 + mu2;
% concatenate g1 and g2 to produce 2*(n = n1+n2) matrix G
G = [g1,g2];
% set the initial conditions using mean and covariance of G
muG = mean(G,2);
covG = cov(G',1);
% maximum likelihood covariance instead
of unbiased one
muG1 = mean(g1,2);
covG1 = cov(g1',1);
of unbiased one
muG2 = mean(g2,2);
% maximum likelihood covariance instead
46
covG2 = cov(g2',1);
of unbiased one
% maximum likelihood covariance instead
% plot the second raw of g1 versus the first raw of g1
x1 = [1,0];
x1 = x1*g1;
% first raw of g1
x2 = [0,1];
x2 = x2*g1;
% second raw of g1
figure(hf);
plot(x1,x2, 'b.');
hf = hf + 1;
% plot the second raw of g2 versus the first raw of g2
x1 = [1,0];
x1 = x1*g2;
% first raw of g2
x2 = [0,1];
x2 = x2*g2;
% second raw of g2
figure(hf);
plot(x1,x2, 'g.');
hf = hf + 1;
% plot the second raw of G versus the first raw of G
figure(hf); hold on; plot(x1,x2, 'g.');
x1 = [1,0]; x1 = x1*g1; x2 = [0,1]; x2 = x2*g1;
plot(x1,x2, 'b.');
% plot an ellipse to represent the covarariance of G
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2), 'k.');
axis equal; hold off; hf = hf + 1;
% save the initial populations, means, and covariances
f_meancov = fopen('data\meancov.txt', 'wt');
fprintf(f_meancov, '%f\n%f\n\n%f\n%f\n\n%f\n%f', muG1, muG2, muG);
fprintf(f_meancov, '\n\n\n%f %f\n%f %f\n\n%f %f\n%f %f\n\n%f %f\n%f
%f', covG1', covG2', covG');
fclose(f_meancov);
f_gauss1 = fopen('data\gauss1.txt', 'wt');
fprintf(f_gauss1, '%f %f\n', g1);
fclose(f_gauss1);
f_gauss2 = fopen('data\gauss2.txt', 'wt');
fprintf(f_gauss2, '%f %f\n', g2);
fclose(f_gauss2);
r = det(covG)^0.25;
combined population
r = 6*r;
% an estimate of the radius of the
f_results = fopen('data\results.txt', 'wt');
% run with different initial conditions
for ii = 1:200,
if ii > 100
muG1 = muG;
muG2 = muG;
covG1 = covG*ceil(100*rand(1))/100;
covG2 = covG*ceil(100*rand(1))/100;
else
muG1 = muG + [r*(rand(1)-0.5);r*(rand(1)-0.5)];
muG2 = muG + [r*(rand(1)-0.5);r*(rand(1)-0.5)];
covG1 = covG;
covG2 = covG;
47
end
fprintf(f_results, '%f\n%f\n\n%f\n%f', muG1, muG2);
fprintf(f_results, '\n\n\n%f %f\n%f %f\n\n%f %f\n%f %f', covG1',
covG2');
% do N_itr iterations of the classification
cl_old = zeros(n,1);
% memory for the previous
classification
for a = 1:N_itr,
% constant factors of the 2-D Gaussian density functions
fac1 = 1/(2*pi*sqrt(det(covG1)));
fac2 = 1/(2*pi*sqrt(det(covG2)));
exp1 = -covG1\eye(2)/2;
% constant factors of the exponent
exp2 = -covG2\eye(2)/2;
% inv(A) = A\eye(size(A)); about 2-3
times faster
MUG1 = ones(1,n); MUG1 = muG1*MUG1;
MUG2 = ones(1,n); MUG2 = muG2*MUG2;
% classify according to the probabilities
p1 = fac1*exp(diag((G-MUG1)'*exp1*(G-MUG1)));
p2 = fac2*exp(diag((G-MUG2)'*exp2*(G-MUG2)));
cl = p2 > p1;
k = sum(cl); j = n - k;
G1 = zeros(2,j); G2 = zeros(2,k);
j = 1; k = 1;
for i = 1:n,
if cl(i)
G2(2*k-1) = G(2*i-1);
G2(2*k) = G(2*i);
k = k + 1;
else
G1(2*j-1) = G(2*i-1);
G1(2*j) = G(2*i);
j = j + 1;
end
end
% check for empty populations and not enough points to compute a
covariance
if prod(size(G1)) < 6 | prod(size(G2)) < 6
G1 = G;
G2 = G;
muG1 = muG;
muG2 = muG;
covG1 = covG;
covG2 = covG;
j = 1;
k = 1;
break;
end
% recompute the means and covariances
muG1 = mean(G1,2);
muG2 = mean(G2,2);
covG1 = cov(G1',1);
% maximum likelihood covariance instead
of unbiased one
covG2 = cov(G2',1);
% check for singularities in covariance matrices
if cond(covG1) > 1000 | cond(covG2) > 1000
G1 = G;
G2 = G;
muG1 = muG;
muG2 = muG;
covG1 = covG;
covG2 = covG;
48
j = 1;
break;
k = 1;
end
% check for no changes in the points
cl_old = abs(cl_old - cl);
if sum(cl_old) == 0
break;
end
cl_old = cl;
end
% end of the main classification loop
N1 = j - 1; N2 = k - 1; N = N1+N2;
L = log(det(covG)); L1 = log(det(covG1)); L2 = log(det(covG2));
% The lower AIC gives the better solution, common factor 2 is not
used
AIC_1 = N/2*L + 5;
% AIC for one component
AIC_2 = N1/2*L1 - N1*log(N1/N) + N2/2*L2 - N2*log(N2/N) + 11;
% AIC for two components
% AIC +ve: two components; -ve: one component
AICs = AIC_1 - AIC_2;
if ~(AICs <= 100 | AICs > 100)
AICs = 10000;
end
% log likelihood of the combined population
fac = 1/(2*pi*sqrt(det(covG)));
fac_exp = -covG\eye(2)/2;
% constant factors of the exponent
MUG = ones(1,n); MUG = muG*MUG;
p = fac*exp(diag((G-MUG)'*fac_exp*(G-MUG)));
L = sum(log(p));
AIC_1 = -L + 5;
% log likelihoods of the classified populations
fac1 = 1/(2*pi*sqrt(det(covG1))) * N1/N;
fac2 = 1/(2*pi*sqrt(det(covG2))) * N2/N;
exp1 = -covG1\eye(2)/2;
% constant factors of the exponent
exp2 = -covG2\eye(2)/2;
% inv(A) = A\eye(size(A)); about 2-3
times faster
MUG1 = ones(1,n); MUG1 = muG1*MUG1;
MUG2 = ones(1,n); MUG2 = muG2*MUG2;
p1 = fac1*exp(diag((G-MUG1)'*exp1*(G-MUG1)));
p2 = fac2*exp(diag((G-MUG2)'*exp2*(G-MUG2)));
Ls = sum(log(p1+p2));
AIC_2 = -Ls + 11;
AICc = AIC_1 - AIC_2;
if ~(AICc <= 100 | AICc > 100)
AICc = 10000;
end
% save final results, populations, means, covariances, and AIC
fprintf(f_results, '\n\n\n%f\n%f\n\n%f\n%f', muG1, muG2);
fprintf(f_results, '\n\n\n%f %f\n%f %f\n\n%f %f\n%f %f', covG1',
covG2');
fprintf(f_results, '\n\n\n%d %d %d', a, N1, N2);
fprintf(f_results, '\n\n\n%f %f', AICs, AICc);
49
fprintf(f_results,
'\n\n****************************************\n\n');
f_fgauss1 = fopen(strcat('data\f', int2str(ii), 'gauss1.txt'), 'wt');
fprintf(f_fgauss1, '%f %f\n', G1);
fclose(f_fgauss1);
f_fgauss2 = fopen(strcat('data\f', int2str(ii), 'gauss2.txt'), 'wt');
fprintf(f_fgauss2, '%f %f\n', G2);
fclose(f_fgauss2);
end
fclose(f_results);
Matlab code 2
% Analyses the file results.txt and identifies the different
% solutions
% A new results file nresults.txt is created with only one entry for
% each different solution
% The first occurrence of the solution in the 200 initial
% conditions, number of iterations for that occurrence, number of
% samples in the converged solutions, the final means and
% covariances, and the AIC values are copied from results.txt
% The number of initial conditions which did not separate the
% initial populations, and the total number of solutions are saved
% in nnresults.txt
% For each solution the total number of initial conditions, initial %
conditions from Experiments A and B, the corresponding average
% number of iterations for convergence and the standard deviation
% values, and the first occurrence of the solution in the 200
% initial conditions are also saved
% The initial mean points of Experiment A which did not separate the
% populations are saved in nmeans.txt
% The initial covariances of Experiment B which did not separate the
% populations are saved in ncovs.txt
% The successful initial mean points and covariances of each
% solution are saved in the files imeans<i>.txt and icovs<i>.txt,
% where 1 <= i <= number of different solutions
clear;
j = 0;
k = 0;
v_itr = zeros(0); itr_index = zeros(0);
mu_itr = zeros(0); mu_index = zeros(0);
cov_itr = zeros(0); cov_index = zeros(0);
fp_arr = zeros(0); fp_cov = zeros(0); file_list = zeros(0);
mu1Array = zeros(2,0); mu2Array = zeros(2,0);
f_results = fopen('data\results.txt', 'rt');
f_nresults = fopen('data\nresults.txt', 'wt');
f_nmeans = fopen('data\nmeans.txt', 'wt');
f_ncovs = fopen('data\ncovs.txt', 'wt');
for ii = 1:200,
muG1i = fscanf(f_results, '%f', 2);
muG2i = fscanf(f_results, '%f', 2);
covG1i = fscanf(f_results, '%f', [2,2]);
covG1i = covG1i';
50
covG2i = fscanf(f_results, '%f', [2,2]);
muG1 = fscanf(f_results, '%f', 2);
muG2 = fscanf(f_results, '%f', 2);
covG1 = fscanf(f_results, '%f', [2,2]);
covG2 = fscanf(f_results, '%f', [2,2]);
itr = fscanf(f_results, '%d', 1);
N1 = fscanf(f_results, '%d', 1);
N2 = fscanf(f_results, '%d', 1);
AICs = fscanf(f_results, '%f', 1);
AICc = fscanf(f_results, '%f', 1);
s = fscanf(f_results,'%c', 44);
covG2i = covG2i';
covG1 = covG1';
covG2 = covG2';
if itr == 100 fprintf(1, '100 Iterations for file: %d\n', ii); end
tArray1 = muG1*ones(1,size(mu1Array,2));
tArray2 = muG2*ones(1,size(mu2Array,2));
diff1
diff2
diff3
diff4
=
=
=
=
sum(mu1Array
sum(mu2Array
sum(mu1Array
sum(mu2Array
-
tArray1,
tArray2,
tArray2,
tArray1,
1);
1);
1);
1);
n = 0;
for m = 1:k,
if (diff1(m) == 0 & diff2(m) == 0) | (diff3(m) == 0 & diff4(m)
== 0)
n = m;
end
end
if N1 == 0
j = j + 1;
if ii > 100
fprintf(f_ncovs, '%f %f %f %f %f %f %f %f\n', covG1i',
covG2i');
else
fprintf(f_nmeans, '%f %f %f %f\n', muG1i, muG2i);
end
elseif n > 0
if ii > 100
fprintf(fp_cov(n), '%f %f %f %f %f %f %f %f\n', covG1i',
covG2i');
cov_itr = [cov_itr, itr]; cov_index = [cov_index, n];
else
fprintf(fp_arr(n), '%f %f %f %f\n', muG1i, muG2i);
mu_itr = [mu_itr, itr]; mu_index = [mu_index, n];
end
v_itr = [v_itr, itr]; itr_index = [itr_index, n];
else
k = k + 1; fp_arr = [fp_arr, 0]; fp_cov = [fp_cov, 0];
file_list = [file_list, ii];
fp_arr(k) = fopen(strcat('data\imeans', int2str(k), '.txt'),
'wt');
fp_cov(k) = fopen(strcat('data\icovs', int2str(k), '.txt'),
'wt');
if ii > 100
fprintf(fp_cov(k), '%f %f %f %f %f %f %f %f\n', covG1i',
covG2i');
cov_itr = [cov_itr, itr]; cov_index = [cov_index, k];
else
51
fprintf(fp_arr(k), '%f %f %f %f\n', muG1i, muG2i);
mu_itr = [mu_itr, itr]; mu_index = [mu_index, k];
end
v_itr = [v_itr, itr]; itr_index = [itr_index, k];
mu1Array = [mu1Array, muG1];
mu2Array = [mu2Array, muG2];
fprintf(f_nresults, '%3d %3d', ii, itr);
fprintf(f_nresults, '\n\n%3d %3d', N1, N2);
fprintf(f_nresults, '\n\n\n%10f\n%10f\n\n%10f\n%10f', muG1,
muG2);
fprintf(f_nresults, '\n\n\n%10f %10f\n%10f %10f\n\n%10f
%10f\n%10f %10f', covG1', covG2');
fprintf(f_nresults, '\n\n\n%10f %10f', AICs, AICc);
fprintf(f_nresults,
'\n\n****************************************\n\n');
end
end
f_nnresults = fopen('data\nnresults.txt', 'wt');
fprintf(f_nnresults, '%d %d\n\n', j, k);
for ii = 1:k,
vv_itr = zeros(0); n = 0;
for j = 1:size(itr_index,2)
if itr_index(j) == ii
vv_itr = [vv_itr, v_itr(j)];
n = n + 1;
end
end
vmu_itr = zeros(0); mu_n = 0;
for j = 1:size(mu_index,2)
if mu_index(j) == ii
vmu_itr = [vmu_itr, mu_itr(j)];
mu_n = mu_n + 1;
end
end
if size(vmu_itr,2) == 0 vmu_itr = 0; end
vcov_itr = zeros(0); cov_n = 0;
for j = 1:size(cov_index,2)
if cov_index(j) == ii
vcov_itr = [vcov_itr, cov_itr(j)];
cov_n = cov_n + 1;
end
end
if size(vcov_itr,2) == 0 vcov_itr = 0; end
fprintf(f_nnresults,
fprintf(f_nnresults,
sqrt(var(vv_itr,1)));
fprintf(f_nnresults,
sqrt(var(vmu_itr,1)));
fprintf(f_nnresults,
sqrt(var(vcov_itr,1)));
fprintf(f_nnresults,
end
'%3d %2d %2d ', n, mu_n, cov_n);
'| %10f %10f | ', mean(vv_itr),
'%10f %10f | ', mean(vmu_itr),
'%10f %10f | ', mean(vcov_itr),
'%3d\n\n', file_list(ii));
52
fclose(f_nnresults);
fclose(f_results);
fclose(f_nresults);
fclose(f_nmeans);
fclose(f_ncovs);
for ii = 1:k,
fclose(fp_arr(ii));
fclose(fp_cov(ii));
end
Matlab code 3
% Plot the initial populations and results of Experiment A
clear; hf = 1;
f_nnresults = fopen('data\nnresults.txt', 'rt');
nn = fscanf(f_nnresults, '%d', 2); ss = nn(2);
fprintf(1, 'Number of initial conditions which did not separate:
%d\n', nn(1));
fprintf(1, 'Total number of solutions: %d\n', ss);
nn = fscanf(f_nnresults, '%d%d%d%c%c%f%f%c%c%f%f%c%c%f%f%c%c%d',
[18,inf]);
fclose(f_nnresults);
if ss ~= size(nn,2)
fprintf(2, 'Error in nnresults.txt\n');
return;
end
tmp_nn = nn; lnn = zeros(0);
for ii = 1:4,
rr = 0;
for jj = 1:ss,
if (tmp_nn(2, jj) > rr)
kk = jj; rr = tmp_nn(2, kk);
end
end
if rr > 0
lnn = [lnn, kk]; tmp_nn(2, kk) = 0;
else
break;
end
end
ss = size(lnn, 2);
f_gauss1 = fopen('data\gauss1.txt', 'rt');
g1 = fscanf(f_gauss1, '%f', [2,inf]);
fclose(f_gauss1);
f_gauss2 = fopen('data\gauss2.txt', 'rt');
g2 = fscanf(f_gauss2, '%f', [2,inf]);
fclose(f_gauss1);
% concatenate g1 and g2 to produce 2*(n = n1+n2) matrix G
G = [g1,g2];
muG = mean(G,2); covG = cov(G',1);
muG1 = mean(g1,2); covG1 = cov(g1',1);
muG2 = mean(g2,2); covG2 = cov(g2',1);
53
% plot the second raw of G versus the first raw of G
figure(hf); box on; hold on;
x1 = [1,0]; x1 = x1*g1; x2 = [0,1]; x2 = x2*g1; plot(x1,x2, 'b.');
x1 = [1,0]; x1 = x1*g2; x2 = [0,1]; x2 = x2*g2; plot(x1,x2, 'g.');
% plot ellipses to represent the covarariance of G
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2), 'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2), 'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2), 'k*');
axis equal; hold off; hf = hf + 1;
% plot the means
fprintf('\nMeans Covs 1st_file
hf\n');
for kk = 1:ss,
cc = lnn(kk);
fprintf(1, '%d\t%d\t%d\t%d\n', nn(2,cc), nn(3,cc), nn(18,cc), hf);
f_imeans = fopen(strcat('data\imeans', int2str(cc), '.txt'), 'rt');
muGi = fscanf(f_imeans, '%f', [4,inf]);
fclose(f_imeans);
a = min(24, size(muGi,2));
if a > 0
x1 = [1,0,0,0]; x1 = x1*muGi; x2 = [0,1,0,0]; x2 = x2*muGi;
x3 = [0,0,1,0]; x3 = x3*muGi; x4 = [0,0,0,1]; x4 = x4*muGi;
end
for k = 1:a,
switch round(6*(k/6 - floor(k/6)))
case 1, figure(hf); box on; hold on;
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
case
case
case
case
case
2,
3,
4,
5,
0,
plot(x1(k),
plot(x1(k),
plot(x1(k),
plot(x1(k),
plot(x1(k),
plot(x1(k),
x2(k),
x2(k),
x2(k),
x2(k),
x2(k),
x2(k),
'm+',
'c+',
'r+',
'g+',
'b+',
'y+',
x3(k),
x3(k),
x3(k),
x3(k),
x3(k),
x3(k),
x4(k),
x4(k),
x4(k),
x4(k),
x4(k),
x4(k),
'm*');
'c*');
'r*');
'g*');
'b*');
'y*');
end
end
end
% plot the means, which didn't work
f_nmeans = fopen('data\nmeans.txt', 'rt');
muGn = fscanf(f_nmeans, '%f', [4,inf]);
fclose(f_nmeans);
a = min(12, size(muGn,2));
if a > 0
fprintf(1, 'Figures of means which did not work starts from:
%d\n', hf);
x1 = [1,0,0,0]; x1 = x1*muGn; x2 = [0,1,0,0]; x2 = x2*muGn;
x3 = [0,0,1,0]; x3 = x3*muGn; x4 = [0,0,0,1]; x4 = x4*muGn;
end
for k = 1:a,
54
switch round(6*(k/6 - floor(k/6)))
case 1, figure(hf); box on; hold on;
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
case
case
case
case
case
2,
3,
4,
5,
0,
plot(x1(k),
plot(x1(k),
plot(x1(k),
plot(x1(k),
plot(x1(k),
plot(x1(k),
x2(k),
x2(k),
x2(k),
x2(k),
x2(k),
x2(k),
'm+',
'c+',
'r+',
'g+',
'b+',
'y+',
x3(k),
x3(k),
x3(k),
x3(k),
x3(k),
x3(k),
x4(k),
x4(k),
x4(k),
x4(k),
x4(k),
x4(k),
'm*');
'c*');
'r*');
'g*');
'b*');
'y*');
end
end
for kk = 1:ss,
cc = nn(18, lnn(kk));
f_f1gauss1 = fopen(strcat('data\f', int2str(cc), 'gauss1.txt'),
'rt');
fgauss1 = fscanf(f_f1gauss1, '%f', [2,inf]);
fclose(f_f1gauss1);
f_f1gauss2 = fopen(strcat('data\f', int2str(cc), 'gauss2.txt'),
'rt');
fgauss2 = fscanf(f_f1gauss2, '%f', [2,inf]);
fclose(f_f1gauss2);
muG1 = mean(fgauss1,2); covG1 = cov(fgauss1',1);
muG2 = mean(fgauss2,2); covG2 = cov(fgauss2',1);
figure(hf); box on; hold on;
x1 = [1,0]; x1 = x1*fgauss1; x2 = [0,1]; x2 = x2*fgauss1; plot(x1,x2,
'b.');
x1 = [1,0]; x1 = x1*fgauss2; x2 = [0,1]; x2 = x2*fgauss2; plot(x1,x2,
'g.');
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2), 'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2), 'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2), 'k*');
axis equal; hold off; hf = hf + 1;
end
Matlab code 4
% Plot the initial populations and results of Experiment B
clear; hf = 1;
f_nnresults = fopen('data\nnresults.txt', 'rt');
nn = fscanf(f_nnresults, '%d', 2); ss = nn(2);
fprintf(1, 'Number of initial conditions which did not separate:
%d\n', nn(1));
fprintf(1, 'Total number of solutions: %d\n', ss);
55
nn = fscanf(f_nnresults, '%d%d%d%c%c%f%f%c%c%f%f%c%c%f%f%c%c%d',
[18,inf]);
fclose(f_nnresults);
if ss ~= size(nn,2)
fprintf(2, 'Error in nnresults.txt\n');
return;
end
tmp_nn = nn; lnn = zeros(0);
for ii = 1:4,
rr = 0;
for jj = 1:ss,
if (tmp_nn(3, jj) > rr)
kk = jj; rr = tmp_nn(3, kk);
end
end
if rr > 0
lnn = [lnn, kk]; tmp_nn(3, kk) = 0;
else
break;
end
end
ss = size(lnn, 2);
f_gauss1 = fopen('data\gauss1.txt', 'rt');
g1 = fscanf(f_gauss1, '%f', [2,inf]);
fclose(f_gauss1);
f_gauss2 = fopen('data\gauss2.txt', 'rt');
g2 = fscanf(f_gauss2, '%f', [2,inf]);
fclose(f_gauss1);
% concatenate g1 and g2 to produce 2*(n = n1+n2) matrix G
G = [g1,g2];
muG = mean(G,2); covG = cov(G',1);
muG1 = mean(g1,2); covG1 = cov(g1',1);
muG2 = mean(g2,2); covG2 = cov(g2',1);
% plot the second raw of G versus the first raw of G
figure(hf); box on; hold on;
x1 = [1,0]; x1 = x1*g1; x2 = [0,1]; x2 = x2*g1; plot(x1,x2, 'b.');
x1 = [1,0]; x1 = x1*g2; x2 = [0,1]; x2 = x2*g2; plot(x1,x2, 'g.');
% plot ellipses to represent the covarariance of G
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2), 'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2), 'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2), 'k*');
axis equal; hold off; hf = hf + 1;
% plot the covariances
fprintf('\nMeans Covs 1st_file
hf\n');
for kk = 1:ss,
cc = lnn(kk);
fprintf(1, '%d\t%d\t%d\t%d\n', nn(2,cc), nn(3,cc), nn(18,cc), hf);
f_icovs = fopen(strcat('data\icovs', int2str(cc), '.txt'), 'rt');
covGi = fscanf(f_icovs, '%f', [8,inf]);
fclose(f_icovs);
a = min(8, size(covGi,2));
for k = 1:a
switch round(6*(k/6 - floor(k/6)))
case 1, figure(hf); box on; hold on;
56
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
plotellipse(muG, covGi, k, 'm--');
case 2, plotellipse(muG, covGi, k, 'c--');
case 3, figure(hf); box on; hold on;
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
plotellipse(muG, covGi, k, 'r--');
case 4, plotellipse(muG, covGi, k, 'g--');
case 5, figure(hf); box on; hold on;
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
plotellipse(muG, covGi, k, 'b--');
case 0, plotellipse(muG, covGi, k, 'y--');
end
end
end
% plot the covariances, which didn't work
f_ncovs = fopen('data\ncovs.txt', 'rt');
covGn = fscanf(f_ncovs, '%f', [8,inf]);
fclose(f_ncovs);
a = min(8, size(covGn,2));
if a > 0
fprintf(1, 'Figures of covariances which did not work starts
from: %d\n', hf);
end
for k = 1:a
switch round(6*(k/6 - floor(k/6)))
case 1, figure(hf); box on; hold on;
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
plotellipse(muG, covGn, k, 'm--');
case 2, plotellipse(muG, covGn, k, 'c--');
case 3, figure(hf); box on; hold on;
57
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
plotellipse(muG, covGn, k, 'r--');
case 4, plotellipse(muG, covGn, k, 'g--');
case 5, figure(hf); box on; hold on;
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2),
'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2),
'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2),
'k*');
hf = hf + 1;
plotellipse(muG, covGn, k, 'b--');
case 0, plotellipse(muG, covGn, k, 'y--');
end
end
for kk = 1:ss,
cc = nn(18, lnn(kk));
f_f1gauss1 = fopen(strcat('data\f', int2str(cc), 'gauss1.txt'),
'rt');
fgauss1 = fscanf(f_f1gauss1, '%f', [2,inf]);
fclose(f_f1gauss1);
f_f1gauss2 = fopen(strcat('data\f', int2str(cc), 'gauss2.txt'),
'rt');
fgauss2 = fscanf(f_f1gauss2, '%f', [2,inf]);
fclose(f_f1gauss2);
muG1 = mean(fgauss1,2); covG1 = cov(fgauss1',1);
muG2 = mean(fgauss2,2); covG2 = cov(fgauss2',1);
figure(hf); box on; hold on;
x1 = [1,0]; x1 = x1*fgauss1; x2 = [0,1]; x2 = x2*fgauss1; plot(x1,x2,
'b.');
x1 = [1,0]; x1 = x1*fgauss2; x2 = [0,1]; x2 = x2*fgauss2; plot(x1,x2,
'g.');
ellipse(muG, covG, 'k-', hf); plot(muG(1), muG(2), 'k.');
ellipse(muG1, covG1, 'k--', hf); plot(muG1(1), muG1(2), 'k+');
ellipse(muG2, covG2, 'k--', hf); plot(muG2(1), muG2(2), 'k*');
axis equal; hold off; hf = hf + 1;
end
Auxiliary functions 1
% Plot an ellipse to represent the population with,
% mean muG and covariance covG
function ellipse(muG, covG, style, hf)
[v, d] = eig(covG); c = 1;
58
x = linspace(-sqrt(d(1,1))*c, sqrt(d(1,1))*c, 1000);
yp = real(sqrt((c^2 - x.^2/d(1,1))*d(2,2)));
xp = [x;yp]; yn = -yp; xn = [x;yn];
ev = v(1,1) + i*v(2,1); theta = angle(ev);
R = [cos(theta), -sin(theta); sin(theta), cos(theta)];
mu = ones(2,1000); muGG = [muG(1),0;0,muG(2)]; mu = muGG*mu;
xp = R*xp + mu; xn = R*xn + mu;
xg1 = [1,0]; xg1p = xg1*xp;
xg2 = [0,1]; xg2p = xg2*xp;
xg1 = [1,0]; xg1n = xg1*xn;
xg2 = [0,1]; xg2n = xg2*xn;
figure(hf); hold on;
plot(xg1p, xg2p, style); plot(xg1n, xg2n, style);
Auxiliary functions 2
% Plot two ellipses to represent the populations with,
% mean muG and covariances stored in the matrix covGi
function plotellipse(muG, covGi, k, style)
covG1 = [covGi(1, k), covGi(2, k); covGi(3, k) ,covGi(4, k)];
covG2 = [covGi(5, k), covGi(6, k); covGi(7, k) ,covGi(8, k)];
muG1 = muG; muG2 = muG;
[v, d] = eig(covG1); c = 1;
x = linspace(-sqrt(d(1,1))*c, sqrt(d(1,1))*c, 1000);
yp = real(sqrt((c^2 - x.^2/d(1,1))*d(2,2)));
xp = [x;yp]; yn = -yp; xn = [x;yn];
ev = v(1,1) + i*v(2,1); theta = angle(ev);
R = [cos(theta), -sin(theta); sin(theta), cos(theta)];
mu = ones(2,1000); muGG = [muG1(1),0;0,muG1(2)]; mu = muGG*mu;
xp = R*xp + mu; xn = R*xn + mu;
xg11 = [1,0]; xg11p = xg11*xp;
xg12 = [0,1]; xg12p = xg12*xp;
xg11 = [1,0]; xg11n = xg11*xn;
xg12 = [0,1]; xg12n = xg12*xn;
[v, d] = eig(covG2); c = 1;
x = linspace(-sqrt(d(1,1))*c, sqrt(d(1,1))*c, 1000);
yp = real(sqrt((c^2 - x.^2/d(1,1))*d(2,2)));
xp = [x;yp]; yn = -yp; xn = [x;yn];
ev = v(1,1) + i*v(2,1); theta = angle(ev);
R = [cos(theta), -sin(theta); sin(theta), cos(theta)];
mu = ones(2,1000); muGG = [muG2(1),0;0,muG2(2)]; mu = muGG*mu;
xp = R*xp + mu; xn = R*xn + mu;
xg21 = [1,0]; xg21p = xg21*xp;
xg22 = [0,1]; xg22p = xg22*xp;
xg21 = [1,0]; xg21n = xg21*xn;
xg22 = [0,1]; xg22n = xg22*xn;
plot(xg11p, xg12p, style); plot(xg11n, xg12n, style);
plot(xg21p, xg22p, style); plot(xg21n, xg22n, style);
59
9. References
[1] Roland Wilson: ‘Video annotation and retrieval using multiresolution gaussian
mixture models’, EPSRC Project proposal, 2000
[2] Richard Johnson and Dean Wichern: ‘Applied multivariate statistical analysis’,
Prentice Hall, 1992
[3] D. M. Titterington, A. F. M. Smith, and U. E. Makov: ‘Statistical analysis of finite
mixture distributions’, Wiley, 1985.
[4] Hirotugu Akaike: ‘A new look at the statistical model identification’, IEEE Trans.
Automatic Control, 1974, vol-19, no-6, pp. 716-723
[5] Roland Wilson: ‘MGMM: Multiresolution Gaussian mixture models for computer
vision’, Proc. IEEE International Conference on Pattern Recognition, 2000, pp.
212-215
[6] Z. Liang, R. J. Jaszczak, and R. E. Coleman: ‘Parameter estimation of finite
mixtures using the EM algorithm and information criteria with application to
medical image processing’, IEEE Trans. Nuclear Science, 1992, vol-39, no-4, pp.
1126-1133
[7] Mario Figueiredo and Anil Jain: ‘Unsupervised selection and estimation of finite
mixture models’, Proc. IEEE International Conference on Pattern Recognition,
2000, pp. 87-90
[8] Walter Zucchini: ‘An introduction to model selection’, Journal of Mathematical
Psychology, 2000, vol-44, no-1, pp. 41-61
[9] Patricia McKenzie and Michael Alder: ‘Selecting the optimal number of
components for a Gaussian mixture model’, Proc. IEEE International Symposium
on Information Theory, 1994, pp. 393
[10] Patrick L. Smith: ‘On Akaike’s model fit criterion’, IEEE Trans. Automatic
Control, 1987, vol-32, no-12, pp. 1100-1101
[11] Toshifumi Matsuoka and Tad J. Ulrych: ‘Information theory measures with
application to model identification’, IEEE Trans. Acoustics, Speech, and Signal
processing, 1986, vol-34, no-3, pp. 511-517
60