Growing Artificial Neural Networks

Wolfgang Klier  Peter Palfrader  Bernhard Stader

(wklier|ppalfrad|bstader)@cosy.sbg.ac.at

March 15, 1999

Abstract

In this document we will describe the current state of research in the field of artificial neural networks and especially methods for growing them automatically.

1 Introduction

Neural network theory is relatively old,the first publications dating back to 1890, but draws growing attention in recent years in connection with artificial neural networks modelled in computers. Many people think of androides or arificial brains when hearing neural network, but recent research is aimed at much simpler goals. Computers are exact mashines what is appreciated in many cases, but for other problems this is not very well suited. Artificial neural networks provide a way to cope even with incomplete or unexact data as it is encountered in pattern recognition applications for example.

1.1 Topologie of Neural Networks

1.1.1 Neurons in Nature vs. Artificial neurons

Our nervous system consists of neuron cells which are interconnected via synapses. A nerve cell receives excitory and inhibitory stimuli via the dendrites, which are added up at the cells body. If the resulting stimulus exceeds a certain value the neuron fires an impulse down its axon (called an action potential), which is propagated to other cells. The grade of excitement controlls the rate at which the neuron fires not the strength, so even our brain uses a binary system.

Computer scientists try to simulate this behaviour in different grades of detail. The basic artificial neuron gets several input values which are added up. These values are often multiplied by so called connection weights, which represent the different grades of influence the preceeding cells have. This sum can be propagated as is or it must exceed a threshold to propagate a value. At any point in this process additional functions can be inserted to transform the signal an some way.

1.1.2 Basic Neural Networks

A basic neural network consists of an input layer, one or more hidden layers and an output layer. In the input layer the neurons have no incoming connections and are turned on and off according to some input pattern. The hidden layer(s) do the real work as here the full potential with input weigths, thresholds and other functions can be facilitated. The output layers main purpose is to show the networks resulting pattern. Therefore the neurons usually have no outgoing connections instead their state is read out directly. But the output layer also does the final processing of the signals from the hidden layer(s).

The hidden layer can sometimes be dynamic, i.e. it can grow or shrink during learning. Growing can take place on several levels:

• at the cell level where the connections between cells develop by growing the dentrites and axons of the artificial neurons.

• at the network level where additional neurons and connections are inserted as they are needed.

References

[1] Jari Vaario, Akira Onisuka and Katuso Shimohara. Formation of Neural Structures. Technical Paper, ATR Human Information Processing Laboratories.

[2] Dietmar Heinke and Fred H. Hamker. Comparing Neural Networks: A Benchmark on Growing Neural Gas, Growing Cell Structures, and Fuzzy ARTMAP. TU Ilmenau 1998.

[3] David D. Olmsted. History and Principles of Neural Networks. http://www.neurocomputing.org/history.htm.