2.11. Is having one network with multiple outputs better than having
multiple networks with one output?
(Translation by Lukasz Brodziak lukasz.brodziak@gmail.com)
There is another issue connected to the output layer of a neural network which can be solved in two
ways and the choice of the way is a free decision of the network creator. It is known that it is always
possible to create a network which will have any amount of outputs – as many as much data
we want to get on the output as a result of solving the problem. It is not always an optimal solution
because the teaching process of multiple-output network has to lead, while estimating weight values
inside the network, to specific compromises which always have negative effect on the result.
An example of above mentioned compromises can be a situation when while setting
a problem for a specific neuron of the hidden layer one has to take into account (which shows
during its weight factors estimation) what role will this neuron have in the calculation of the values
of several output neurons to which it sends its output signal when it will be calculated. It may
happen that the role of a given hidden neuron, optimal from point of view of calculating with its aid
one given output signal from whole network, will be significantly different then its role optimal for
some other output. In such case the teaching process will change the weight values in this hidden
neuron adapting it each time to a different role – this will have an effect in long and less successful
teaching. That is why it is better to divide a complex problem and instead of one network with
multiple outputs build several networks which will use the same set of input data but have separate
hidden layers and single outputs.
Fig. 2.37. Comparison of one network with two outputs versus two separate networks
The rule suggested in Figure 2.37 cannot be treated as a dogma because sometimes it appears that
multiple-output network learns better than the one with single output. This, at first glance, paradox
result can be substantiated with a fact that potential „conflict”, described in paragraph above,
connected to functioning of the hidden neurons and estimation of their roles while calculating
several different output values may never occur. On the contrary sometimes during the teaching
process one can observe an effect of specific synergy which consists in the fact that while setting
(during the teaching process) the parameters of a hidden neuron which are optimal from the point of
view of the „interests” of several output layer neurons, one can achieve success faster and more
efficiently than while doing it in separate networks individually tuned for each output signal
separately. That is why one should not assume that for sure one solution is better than the other, it is
better to test them both and then making a choice. My own observation, based on experience in
building hundreds of networks for various uses and on consulting tens of works by my students,
shows that significantly more often the collection of networks with single output is an optimal
solution – although biological neural networks are more often organized as multi-output aggregates.
You already know that every feedforward network has to have at least two of above
mentioned layers – input and output. Yet, there are networks (see fig. 2.38) – they are called singlelayered networks because they have only one learning layer. It is of course an output layer because
as you already know the input layer of any network does not learn.
Fig. 2.38. Single-layered network
Yet still many networks (especially those solving more complex problems) must contain additional
layers connecting input and output layer. Those layers put between input and output are called
hidden layers. The name sounds quite mysterious that is why you may feel troubled while getting in
touch with hidden layer problem for the first time. That is why I would like to explain to you what
those layers are and in what sense they are „hidden”.