PS Einführung in die Bioinformatik 1999 Growing Artificial Neural Networks

Wolfgang Klier  Peter Palfrader  Bernhard Stader

(wklier|ppalfrad|bstader)@cosy.sbg.ac.at 1999-03-22 Gruppe 6 15:00 Roland Schwaiger

Abstract

In this document we review the current state of research in the area of artificial neural networks(ANNs) with respect to methods for growing them on a biologically plausible basis.

1 Introduction

According to [3] neural network theory is relatively old – the first publications are dating back to 1890 – but draws growing attention in recent years in connection with ANNs modeled in computers. Many people think of androides or artificial brains when hearing neural network, but taking into consideration that the human brain consists of 10^{10} − 10^{11} neurons, recent research is aimed at much simpler goals. Computers are exact machines what is appreciated in many cases, but for other problems this is not very well suited. ANNs provide a way to cope even with incomplete or inexact data as it is encountered in pattern recognition applications for example.

1.1 Topologies of ANNs

1.1.1 Neurons in Nature vs. Artificial neurons

Our nervous system consists of neuron cells which are interconnected via synapses. In [4] the authors name three types of synapses: In most cases synapses are located between a cell’s axon and another cell’s dendrites or between an axon and a cell’s body but there also exist synapses that connect two axons. A nerve cell receives excitatory and inhibitory stimuli which result in one combined stimulus (excitatory stimuli strengthen it; inhibitory ones weaken it). If the resulting stimulus exceeds a certain value – called the threshold value of this cell – the neuron fires an impulse down its axon, called a spike. Note that the grade of excitement has no effect on the strength of the spike but controls the rate at which the neuron fires. If the threshold is not reached, nothing is propagated to the other cells. It’s an all or nothing principle.

Computer scientists try to simulate this behavior in different grades of detail. The basic artificial neuron as described in [3], gets several input values which are added up. These values are often multiplied by so called connection weights (negative weights for inhibitory, positive weights for excitatory connections), which represent the different grades of influence the preceeding cells have. This sum can be propagated as is or it must exceed a threshold to propagate a value. At any point in this process additional functions can be inserted to transform the signal an some way.

1.1.2 Basic ANNs

A basic neural network consists of input neurons, hidden neurons and output neurons according to [3]. The different types of neurons don’t have to be separate units, but one neuron can fulfill several functions. The input neurons have no incoming connections and are turned on and off according to some input pattern. The hidden neurons do the real work. They have incoming and outgoing connections and so all the possibilities of information processing such as input weights, thresholds and other functions can be facilitated. The output neurons main purpose is to represent the networks resulting pattern. Therefore the neurons have no outgoing connections instead their state is read out directly. But the output neurons also do the final processing of the signals from the hidden neurons.

The network can sometimes be dynamic, i.e. it can grow or shrink during learning. Growing can take place on several levels:

• at the cell level where the connections between cells develop by growing the dendrites and axons of the artificial neurons.

• at the network level where additional neurons and connections are inserted as they are needed.

2 Background

2.1 Function of Neurons in Nature

2.1.1 Development of Neural Structures in Vertebrates

As [4] describes the nervous system develops out of the neural plate in the embryo. After the primary cell division the neural cells migrate to their final positions, where some divide further forming special brain structures. They are guided by the Glia-Cells which are long cells spanning the neural pipe. After reaching their final position the neurons begin growing the dendrites and the axon and develop the electrical properties needed for propagating stimuli.

Axon growth is most likely directed by chemical gradients, probably supported by special pioneer cells which take the right place in early stages of development where distances are small and act as guideposts later on. The nerve growth factor plays an important role in this step as well as during the following maturing phase when the nervous system consolidates itself. Neurons which have not formed enough connections up to this point die making room for more important ones to develop further. This process allows better adaption of neural structures to different requirements.

Further changes occur when the brain is learning. Synapses that are used for processing a certain information are strengthened while others degenerate leaving memory traces which represent the newly acquired knowledge. These traces are very stable once they have formed completely, allowing remembering the information for a long time.

2.2 The artificial neuron

According to [5] the original neural network was based on the work of Warren McCulloch and Walter Pitts in 1943. They built up an artificial neuron, which structure is loosely based on the nerve cell of the brain, i.e. it is based on the theory that a nerve’s action potential only fires in an all-or-nothing manner if the threshold for that nerve has been exceeded. Thus it sends an impulse to neurons connected via it’s axon. An artificial neuron – shown in figure 1 – fires not continously, but at any time t if and only if (e-i) exceeds h, where e is the number of excitatory synapses at that time, i is the number of the inhibitory ones and h is the firing threshold for that neuron.

Figure 1: Analog Artifical Neuron

They used three different kinds of neurons: Receptor (input neurons), central or inner neurons, effector (output neurons). The Receptor receives the impulse to fire from a sensor. The central neuron connects the receptor and other neurons with the effector and other neurons. The effector receives its impulses directly from the receptor or from the inner neurons.

There are many different solutions for ANNs, like the two layer perceptron from Rosenblatt or the the Multi Layer Perceptron (MLP) which is now considered the standard form of ANNs.

The two layer perceptron is much alike the MLP, but with one big difference. As the name suggests it consists of only two layers: the input and the output layer. The MLP has a hidden layer between the input and the output layer. The MLPs are divided into two groups: nets with and without feedback loops.

A Glossary

connection weights CWs control the influence of the neurons on each other.

hidden layer Group of artificial neurons which are not accessible directly but propagate signals through the ANN.

input layer Group of artificial neurons which normally have no incoming connections but are activated externally.

neuron, natural special kind of cell able to propagate and process electric signals. They form the nervous system in animals.

neuron, artificial processing unit in an ANN, also called a node.

node see: neuron, artificial

output layer Group of artificial neurons which normally have no outgoing connections but represent the output of the ANN.

pattern A set of values which represent the input or output of an ANN.

synapses connection between two natural neurons. threshold A value which has to be exceeded to generate some action.

Protocol

1999-Mar-09

Location: Room Nr.: 2.30 Time: 15:00 - 15:30 Preliminary discussion Tasks:

• General outline of this paper

• Familiarize with the topic

1999-Mar-16

Location: Room Nr.: 2.30 Time: 15:00 - 15:30 Discussing current paper version. Tasks:

• Connection of neurons in nature (graphic)

• Growing of connections in nature

• Construction of connection in ANNs

• Make a glossary

References

[1] Jari Vaario, Akira Onisuka and Katuso Shimohara. Formation of Neural Structures. Technical Paper, ATR Human Information Processing Laboratories.

[2] Dietmar Heinke and Fred H. Hamker. Comparing Neural Networks: A Benchmark on Growing Neural Gas, Growing Cell Structures, and Fuzzy ARTMAP. TU Ilmenau 1998.

[3] David D. Olmsted. History and Principles of Neural Networks. http://www.neurocomputing.org/history.htm.

[4] Hermann Haken, Maria Haken-Krell. Entstehung von biologischer Infomation und Ordnung. Wissenschaftliche Buchgesellschaft Darmstadt 1995.

[5] Human Brain Project. Consciousness: The Development of Neural Networks. http://sig.biostr.washington.edu/projects/brain/