6.12. What is hiding in hidden layers?
(Translation by Marta Kraszewska; martak@agh.edu.pl)
Briefly we can say that the hidden layers are the tool used for processing input signals (received by the input
layer) to the output layer in such way that the required response (the solving problem) can be easily found. The
effect of these additional (not always present) layers is not possible to be directly observed by the user, who
makes the network tasks, and then observes that the network performs them correctly. He has no access nor to
enter neurons of these layers (you have to use the input layer neurons for the transmission of signals to them)
neither to their outputs (effects of hidden layer neurons are revealed only indirectly, through the responses they
produce and are sent by the output layer neurons). It follows that - activity of neurons in the intermediate layers
is not directly visible to the network’s user - and in this sense they are hidden.
Although hidden - these additional layers in neural networks play a very important and responsible function.
They create an additional structure that processes information, and the most simple way to discuss their role is to
provide the example of network implementation in frequently occurring pattern recognition task. In such
networks, a digital image is given at the input (the first layer of the network). It cannot be a picture of a typical
camera or traditional image loaded into the scanner or Frame Grabber. The reason is very simple: in such
system supplied by simple image at the input, rather than some of its more sophisticated description (see Figure
2.30), the input layer neurons corresponds to its size and organization (grouping of neurons in the corresponding
rows and columns) the organization of the image. Simply each point of the image is assigned to the input of
neuron network, which analyzes and reports its status. Image from a digital camera or a scanner is a set of pixels,
but those pixels are millions. It is impossible to build a network with millions of neurons on the input, especially
that such network would have billions (!) connections for which you would need to set the value of weight. It is
impossible to achieve such a structure.
However, you can imagine a system that receives very simplistic digital image on the input, consisting of a small
number of pixels, arranged in the form of simple characters - such as letters (Figure 2.39).
These neurons
represent the
recognition of
individual
characters
A
B
C
Fig. 2.39. An example of a network that recognizes simple images
Rozdział 2, str. 1/3
At the output of the network, it is expected to have decisions informing about what has been diagnosed - for
example, it can be organized so that the output of individual neurons contractually assign certain decisions - for
example, “A was diagnosed”, “B was diagnosed”, etc., and the size of signals at these outputs will be
interpreted in terms of the degree of certainty of the appropriate decision. It is easy to note that it is possible,
therefore, that ambiguous responses will be given by the network ("something is related to the extent of 0.7 to
letter A, but the degree of 0.4 points with B"). It is one of the interesting and useful features of neural networks
that can be therefore associated with a fuzzy logic systems (fuzzy) - more comprehensive discussion of this
aspect goes beyond the book.
In the presented neural network, hidden layer neurons play as intermediaries - they have direct access to the
input data and they can see the image presented at the input, and based on their output further layers take certain
decisions on the diagnosis of one or the other image. Therefore, it is believed that the role of the hidden layer
neurons is to devise such a pre-processed sets of input data to be used by the output layer neurons in the
determination of the final result.
Usefulness of the intermediate layers is based on the fact, that in general making certain input transformation
causes that the solution of the task given to the network becomes much easier to achieve than in the case of
attempts to solve the same task in a direct way. For example, considering a pattern recognition task, it is difficult
to find a rule that allows to determine what the object is shown when analyzing only bright and dark pixels in the
image (and only that information has a network without hidden layer). Finding the correct recognition rule which
functions at that low level is difficult because the diagnosed objects can occur in a number of varieties which
differ somewhat in shape, but have the same meaning. The same object (for example the letter A) can in fact
look a little different when it is scanned from the manuscript, otherwise when it comes to printed text, and
another way when, for example, was photographed on fluttering banner. Even the letter printed using exactly the
same characters can be represented by two completely different images to the pixel values in some selected
points, for example, if it is moved (such trivial fault, which is not disturbing for the man who reads the text, can
change the digital picture of character in completely dramatic way if we have to pay attention to that specific
pixel is white and a black). As if that was not enough - there is an additional difficulty with the digital images of
low resolution which is based on the fact that completely different objects can have very large sets of identical
pixels on digital images.
Expectation that the neural network with "one jump" overcomes all these difficulties, which are visible between
"raw" image and the final decision of the recognition of what the image shows - might be unrealistic in that case.
This reasoning is correct, because the relevant experience confirms that none of the learning process will be able
to make a simple network with no hidden layer to ensure that it will be able to solve the task. Such a network
without hidden layers would produce such an amazing mechanism by which the time to understand some
selected and established sets of pixels as belonging to different objects, and other times would link different sets
pixels with the same object. This simply cannot be done.
What is not possible to be done by a network with fewer layers, in general can be performed by a network
containing a hidden layer. In such tasks, neurons in this layer would find certain auxiliary values, that
significantly facilitate to solve the task. For example considering pattern recognition task, hidden layer neurons
can detect and encode the structure describing the general characteristics of the image and objects seen on that
image. These characteristics should be more responsive to the demands of the final image recognition, than the
original image - for example, can be to some extent independent of the position and the scale of recognized
objects. Examples of features that describe the objects in images of letters (to facilitate their recognition) may be:

the presence or absence of closed contour drawing letters (they have them letters O, D, A, and a few
others, the letters like I, T, S, N, etc. do not have these characteristics)

whether the character has a notch at the bottom (A, X, K), or at the top (U, Y, V, K), and maybe from
the side (E, F)

whether more rounded shape (O, S, C) or severe (E, W, T, A)
Obviously you can find much more the differentiating features of letters and at the same time insensitive to
common factors affecting their recognition (change fonts or handwriting, change the size of the letters, the tilt,
pan, etc.). However it should be emphasized, that the creator of the neural network does not have to explicitly
ask what characteristics of the picture may be found, because the network itself can acquire relevant skills during
the learning process. Of course there is no guarantee that the neural network designed to recognize letters, learns
Rozdział 2, str. 2/3
its hidden layers to detect and signal precisely those characteristics that were mentioned above. You can even bet
a large sum that those characteristics mentioned by me to differentiate letters, neural network would not detect
by itself. But the experience of many researchers confirm that the neural network can recognize the
characteristics in the recognition tasks, that effectively facilitate recognition - although in general, the creator of
the network cannot understand what is the sense of this or another characteristics encoded in the hidden layer.
The use of hidden layer is intentional in every respect, because the neural network might show much more
intelligent forms of behavior, however, the chance need to be given by taking into account the structure of the
elements that precisely can extract "descriptive image features". It is worth noting that the nature of the extracted
features is determined by the structure of the network, only through the process of learning. If we imagine the
network to recognize images, all features that will be extracted by hidden layer neurons, might adapt
automatically to the type of recognizable images. If we ask the network for the detection of aerial photographs of
masked rocket launchers, it becomes clear that the main task of the hidden layer will become independent of the
position of the object, as the shape of the suspect can occur in any region of the image and should always be
recognized in the same way. If the network has to recognize letters, it should not lose information about their
position (a network that will tell you that scanned letter A is somewhere on the page is not very useful as we
need to know precisely where it is, and even more - in what context it presents), so in this case the objective of
the hidden layer is to extract features that may allow for reliable identification of letters, regardless of its size and
whatever typeface (font). What's interesting - both of these tasks can be solved (after appropriate training) using
the same network - although, of course, the network learnt to recognize tanks cannot read the writing, and learnt
to identify fingerprints cannot handle face recognition.
Rozdział 2, str. 3/3