advertisement

2.2 How to make an artificial neuron? (translation by Agata Barabasz agata.barabasz@op.pl) The basic “building materials” that we use to create a neural network are artificial neurons. Now we will try to learn about them more precisely. In the previous chapter you have seen some pictures illustrating the shape of a biological neuron, but it will not harm to recall one more picture, so see in picture 2.4, how an exemplary neuron (in simplification) looks like. Picture 2.4. The orientation structure of a biological nerve cell (a neuron). So that you do not think that all real neurons look exactly like that, in picture 2.5 I am showing you one more illustration of a real biological neuron, dissected free of a rat’s cerebral cortex. Picture 2.5 The view of a microscopic preparation of a real neuron It is hard in this picture to guess, which of the many visible on it fibres is an axon, which is always single and as the only one delivers signals from the given neuron to all the others, and which are performing the role of dendrites. Nevertheless, this is also a real biological neuron, thus also such a cell as this one our artificial neuron has to map well, and of which we will take care now more precisely. Artificial building neurons used in the networks technique are of course very simplified models of nerve cells, that occur in nature. A structure of an artificial neuron best illustrates the scheme presented in picture 2.6. Comparing this illustration with pictures 2.4 or 2.5 you will realise how far neural networks’ researchers simplify biological reality. Picture 2.6 A general scheme of an artificial neuron shows the degree of it’s simplification However, in spite of this simplifications artificial neurons keep all these features, which are valid from the point of view of tasks we want to entrust them within built networks, being the computer science tools, rather than biology models. Firstly, they are characterised by having many inputs and one output. The input signals xi (i = 1,2,…,n) and the output signal y may take on only numerical values, generally of the range from 0 to 1 ( sometimes also from –1 to + 1), whereas the fact that within the tasks being solved by networks they represent some information (e.g. as the output of a decision, who has been recognised by the neural network, which has been analysing someone’s photo), is the result of a specific agreement. Generally particular meanings are ascribed to network’s input and output signals in such a way that the most crucial is this, on which input or output a given signal has occurred (each input and output is associated with a specific meaning of a signal), additionally signals scaling is used, so selected that signal values that would be circulating in a network, would not be out of an agreed range – e.g. from 0 to 1. Secondly – artificial neurons perform specific activities on signals, which they receive on inputs, as a consequence they produce signals (only one by each single neuron), which are present on their outputs and are sent forward (to other neurons, or onto this network’s output, as the solution of a raised problem). Network’s assignment, reduced to the functioning of it’s basic element, which is a neuron, is based on this that it transforms an input data xi into a result y applying rules resulting from that how it has been built, and what has been taught. Considered up to this point neuron’s properties have been illustrated on picture 2.7. x1 x2 y ... xn Picture 2.7 Basic signals occurring in a neuron Thirdly – neurons may learn. This purpose serve wi coefficients called synaptic weights. As you certainly remember from the previous chapter – these reflect rather complicated biochemical and bioelectric processes, which take place in real biological neuron’s synapses. From further considerations point of view the most significant is that synaptic weights can be modified (i.e. their values can be changed), x1 x2 xn w1 w2 y ... wn Picture 2.8 Adding to a neuron’s structure adjustable weight’s coefficients makes it a learnable unit what constitutes a basis for teaching networks. A scheme of a neuron capable of learning has been shown on picture 2.8. Summing up this discourse it could be ascertained that artificial neurons can be treated as elementary processors with the following features: each neuron receives many input signals xi and on their basis determines it’s own “answer” y, that is produces one output signal; with each separated neuron’s input is connected a parameter called weight wi . This name means that it expresses a degree of significance of an information arriving to this neuron through just this input; a signal, coming in through a particular input is first modified with the use of the weight of that given input. Most often a modification is based on this that a signal is simply multiplied through the weight of a given input, so in consequence in further calculations it is already participating in the modified form: strengthened ( if the weight is greater than 1) or restrained (if the weight’s value is less than 1). A signal from a particular input may occur even in the form opposite in relation to signals from the other inputs, if it’s weight has a negative value. Inputs with negative weights are among neural networks users defined as so called inhibitory inputs, whereas these with positive weights are called excitatory inputs. input signals (modified by adequate weights) are aggregated in a neuron (see picture 2.9). Once again considering networks in general, many ways of input signals aggregation may be given, nevertheless most often it is based on this that signals are simply summed up giving as the result some helpful internal signal, called a cumulative neuron stimulation or a postsynaptic stimulation. This signal may be also defined as a net value. x1 x2 xn w1 w2 ... wn s g wi , xi y i 1,,n Picture 2.9. An aggregation of input data as the first of neuron’s internal functions to so created sum of signals the neuron adds sometimes (not in all networks’ types, but generally often) some extra component independent of input signals, called a bias. A bias, if it is taken into account, also undergoes a learning process, that is why sometimes one can imagine, that a bias is an extra synaptic weight associated with the input, on which it is provided an internal signal of constant value equal to 1. A bias role lies in this that thanks to it’s presence during a learning process a neuron’s properties may be formed in a much more free way (without having it the aggregation function characteristics always must pass through the beginning of the coordinate system, what sometimes is a burdensome “ anchor”). A scheme of a neuron, in which a bias has been taken into account, is shown in picture 2.10; Picture 2.10. The application of the additional parameter, which is bias A sum of internal signals multiplied by weights plus (possibly) a bias may be sometimes sent directly to it’s axon and treated as a neuron’s output signal. In many types of networks that is enough. In this way work so called linear networks (for example a net named ADALINE = ADaptive LINEar). However, in networks with richer abilities (for example in very popular networks called MLP from the words Multi–Layer Perceptron) a neuron’s output signal is calculated by means of some nonlinear function. This function in the whole book we will be designating with the symbol ƒ( ) or φ ( ). A scheme of a neuron including both an input signals’ aggregation and an output signal’s generation is presented in picture 2.11; Picture 2.11. The full number of neuron’s internal functions a function φ ( ) is called a characteristic of a neuron (a transfer function). There are known many different neuron’s characteristics, what illustrates picture 2.12 Some of them are chosen in a such way that artificial neuron’s behaviour would be the most similar to a real biological neuron’s behaviour (a sigmoid function), but they also could be selected in such manner, which would assure the maximum efficiency of computations carried on by a neural network (a Gauss function). In all the cases function φ ( ) constitutes an important element going between a joint stimulation of a neuron and it’s output signal; Picture 2.12. Some of the more often used neuron’s characteristics a knowledge of the input signals, weights’ coefficients, inputs aggregation method and neuron’s characteristic, allow to unequivocally define at any time it’s output signal, with usual assuming that (in contrast to what takes place in real neurons) this process occurs immediately. Thanks to this in artificial neural networks changes of input signals are practically immediately appearing on output. Of course this is a clearly theoretical assumption, because after input signals change even in electronic realization some time would be needed for establishing the right value of an output signal by an adequate integrated circuit.. . Much more time would be necessary to achieve the same effect in a net working as a simulation model, because a computer imitating network activities must then calculate all values of all signals on all neurons outputs of this network, what even on very fast computers could take a lot of time. While speaking about a prompt neuron’s action I mean that considering network’s functioning we will not pay attention to a factor, which is a time of neuron’s reaction, because this will be insignificant for us. A complete structure of a single neuron is presented in picture 2.13. x1 x2 w1 w2 wn xn es y neuron liniowy linear neuron Picture 2.13. Structure of a neuron as a processor, which is the basis for building neural networks A neuron presented in this picture is the most typical “material”, which is used for creating a network. More precisely – such typical “material” is a neuron of a network defined as MLP (Multi–Layer Perceptron), the most crucial elements of which I have collected and presented in picture 2.14. It is visible in this picture that neuron MLP is characterised by the aggregation function consisting of simple summing up the input signals multiplied by weights, and uses a nonlinear transfer function with a distinctive sigmoid shape. x1 x2 xn w1 w2 ... wn y:=1/(1+exp(-0.5*x)) 1.1 0.9 n s wx i 1 i y 0.7 0.5 i 0.3 0.1 -0.1 -10 -5 0 5 10 Picture 2.14 The most popular component of neural networks – the MLP type neuron However sometimes in neural networks for special purposes there are used so called radial neurons. They have an atypical method of input data aggregation, and they also use an untypical characteristic (a Gauss’s one) and are taught in an unusual way. At this moment I do not intend to elaborate on a subject of this specific neurons, which are used mainly to create special networks called RBF (Radial Basis Functions), but in picture 2.15 I present a scheme of such a radial neuron, to enable you to make a comparison with discussed earlier a typical neuron shown in picture 2.14. 1 t1 x1 r1 f ... xn t n f y x-t r n In this type of neuron the aggregation of the input signals consists of evaluating the distance between the present input vector X and the centroid of a certain subset T determined during a teaching process Also a nonlinear transfer function in these neurons has a different form – ”bell-shaped” gaussoid - i.e. it is a non-monotonic function. Picture 2.15 A structure and peculiar properties of a radial neuron, denoted also as RBF