11.8. How and why shall we use automatic pattern generation for a
Hopfield network?
(Translation by Marek Zachara; mzachara@agh.edu.pl)
As I have mentioned before, while creating the patterns that are to be memorized in the network you shall
observe the value of the scalar product - to achieve a set of patterns with minimized “crosstalk”. You already
know that a reasonable set of patterns shall have these values as low as possible (each pattern against each other).
The ideal solution is to have patterns that have zero value of the scalar product with each other. Such set of
patterns with zero values of the scalar products between each other is called an orthogonal design. An
orthogonal design has a number of great theoretical features, making mathematicians close to ecstatic. It is
possible to write a couple of heavy volumes about it, filled solely with formulas and seasoned with words and
phrases like orthogonal coordinate system, transformation invariance, decorrelation of variables, canonical
components' transformation and diagonalizing of a covariance matrix (all of these do have an important
meaning and are related to the program we discuss), but here and now you really do not need to know about
them. You do need to know however that getting closer to this ideal solution will result in the learning process
and the performance of the network becoming more optimal.
Unfortunately, a creating 'manually' such an orthogonal set of patterns is virtually impossible. You can
try to get close to the ideal, but you will find out that the images of characters inputted with the keyboard will
always be somehow correlated, so their scalar product will not be zero. You can however delegate this task to the
program Example12b. This program can support you in your work by creating orthogonal patterns by itself. In
the top part of the screen there is a grouping filed Orthogonal and pseudorandom patterns, and inside it – an
input field Number of pattern(s), where you can enter a number of patterns the program is required to generate
for you. Remember to confirm your decision by pressing the button labeled Generate. Make sure that the
number of patterns you request in the Number of pattern(s) field added up to the number of patterns you
provided earlier do not exceed the program's limit (20 patterns). Pressing Generate will result in additional
thumbnails of orthogonal patterns (as many as you requested) appearing in the Input pattern(s) for teaching or
recalling window (just like it works with the Add button). You can start generating and adding automatically
generated patterns right from the beginning (Fig. 11.34) or you can try a combined approach – first adding one or
a few patterns by yourself and then passing the control to the machine. (Fig. 11.35, 11.36)
Fig. 11.34. A set of orthogonal patterns that were generated automatically
Rozdział 11, str. 1/5
Fig. 11.35. A set of pseudo-random patterns generated based on initially supplied character X
Fig. 11.36. A set of pseudo-random patterns generated based on initially supplied characters X, Y and Z
In the first case (meaning you request all the patterns, right from the beginning to be generated automatically),
the program creates a set of pairwise orthogonal patterns, which will greatly improve the performance of the
memory. The patters will be simpler and more regular in case of a fully automated pattern generation than in a
case of generation based on a manually inputted character, because the patterns generated in the later scenario
are not pairwise orthogonal, but pseudo-random instead (see Fig. 11.35 and 11.36). In the second and third case,
i.e. when you provide a character with a keyboard before requesting an automated pattern generation, the
program will create a number of pseudo-random patterns. You may notice the low values of the scalar products.
Orthogonal patterns are very useful when the network is employed as an associative memory, because
utilizing of orthogonal patterns (but also pseudo-random patterns) allows for reconstruction of the original input
image, even if it was severely distorted (see Fig. 11.37, 11.38). This of course applies to each of the memorized
patterns, however it is rather hard to verify if a pseudo-random pattern has been correctly reconstructed, as
pseudo-random patterns - generally speaking - look rather exotic.
Fig. 11.37. Example of a correct recalling of a distorted pattern, using a set of orthogonal patterns
Rozdział 11, str. 2/5
Fig. 11.38. Example of a correct recalling of a letter Z pattern that had 26 randomly selected pixels changed
(28% noise)
However, reconstruction of correct images by the Hopfield's network with memorized orthogonal or pseudorandom symbols, even for severely distorted patterns does have its limits (see Fig. 11.39, 11.40). If you distort
the pattern too much, the ability to reconstruct it will be irrecoverably lost. This phenomenon has a critical
characteristic. For example, in Fig. 11.38 and 11.40 you can see that a network with pseudo-random patterns can
quickly and effectively recall a pattern of a letter Z which had 26 and 24 pixels changed, respectively.
Fig. 11.39. Example of an imperfect recalling of an orthogonal pattern that had 19 randomly selected pixels
changed (20% noise)
Rozdział 11, str. 3/5
Fig. 11.40. Correct recalling of a letter Z pattern that had 24 randomly selected pixels changed (25% noise)
You can repeat these experiments several times and you will get a different result with each attempt, because
each time the distorted pattern will be different and the pixels to be changed are picked up randomly. Quite often
it is just a matter of one pixel (e.g. damaging 25 instead of 24 pixels in the original image) that will result in an
inability to reconstruct the image, because of persistent small artifacts that appear in the resultant image (Fig.
11.41)
Fig. 11.41. Completely incorrect recalling of a letter Z pattern that had 25 randomly selected pixels changed
(26% noise)
Rozdział 11, str. 4/5
With an increased damage to the input pattern, the reconstruction process becomes longer and the final result
decreases in quality, until at some point it disintegrates and the networks outputs only pure garbage. Even then it
may happen though – by a pure luck – that some heavily distorted patterns are brilliantly reconstructed (Fig.
11.42)
Fig. 11.42. Incidentally correct recalling of a letter Z pattern
If you would like to experiment with a set of patterns that you have created (and stored on your hard drive) – you
can do so by using the grouping field Load file with patterns. On pressing Load button, a new window will
appear that will allow you to select the file with patterns that is to be loaded into the program. If the file (with a
.PTN extension) you provide exists and can be loaded, the program will list it in the editable field Path: and the
thumbnails of the patterns that are present in this file will show up in the Input pattern(s) for teaching or
recalling window.
Rozdział 11, str. 5/5