<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\documentclass[12pt,fleqn,a4paper]{report}

%\usepackage{german}
\usepackage{amsfonts}
\usepackage{latexsym}
\usepackage{xspace}
\usepackage{epsfig}
\usepackage{psfrag}
\usepackage{graphicx}
\usepackage{amssymb} % Mathematik-Symbole

\textheight     680pt
\textwidth      460pt
\topmargin      -40pt
\oddsidemargin    5pt
\evensidemargin   5pt
\parindent        0pt

\begin{document}
\begin{titlepage}
\centerline{\large PS Einf\"{u}hrung in die Bioinformatik SS2001}
\vspace{0.5cm}
\hspace {2.3cm}
NC-006-2001
\hspace{5.2cm}
\today
\vspace{2cm}

\centerline{\Large\bf Sensitivity Analysis Of Neurons In ANN }
\vspace{0.2cm}
\centerline{\Large\bf }
\vspace{1cm}
\centerline{\bf Report}
\centerline{\bf }
\vspace {1cm}
\centerline{\large Sebastian Gumpold \hfill Clemens Bauer \hfill Annemarie Mayer}
\vspace{0.2cm}
\centerline{sgumpold@cosy.sbg.ac.at \hfill cbauer@cosy.sbg.ac.at \hfill ammayer@cosy.sbg.ac.at}
\vspace {1cm}
\centerline{Academic Supervisor Mag. Dipl.-Ing. Schwaiger Roland}
\vspace{1cm}
\centerline{\large Department of Computer Science}
\vspace{0.2cm}
\centerline{\large University of Salzburg}
\vspace{8cm}
Correspondence to: \\
\centerline{Universit\"{a}t Salzburg} \\
\centerline{Institut f\"{u}r Computerwissenschaften und Systemanalyse} \\
\centerline{Jakob--Haringer--Stra\ss e 2} \\
\centerline{A--5020 Salzburg} \\
\centerline{Austria}
\clearpage
\end{titlepage}

\title{\Large\bf Sensitivity Analysis Of Neurons In ANN }
\author{{\large Sebastian Gumpold, Annemarie Mayer and Clemens Bauer}\\
{\small Department of Computer Science} \\
{\small University of Salzburg} \\
{\small A--5020 Salzburg, Austria} \\
{\small \{sgumpold\}, \{ammayer\}, \{cbauer\}@cosy.sbg.ac.at}}
\date{}

\maketitle

\begin{abstract}

{\sl 
In this paper we present methods to determine the optimal size and
structure of a Feed Forward Neural Network (FFNN) for a specific problem.\\
Using these methods a network's complexity can often be minimized by
pruning hidden neurons and/or connections without loss of function.
We will also show how the optimal structure of an ANN can be reached by
adding hidden neurons to an existing network structure.
As a third way to optimize a neural net, evolutionary design is discussed.\\
One specific aim of this paper is to explain how to (automatically) determine 
the ``optimal'' structure of a FFNN to people who are currently learning about
neural networks.\\
Some of the methods dealed with in our paper we also implemented in a neural
net visualisation program called \emph{NetVis}.
}

\end{abstract}


\tableofcontents


\chapter{Introduction}

\section{A short Introduction to ANN}

\begin{quote}
A Neural Network is an interconnected assembly of simple processing elements,
\emph{units} or \emph{nodes}, whose functionality is loosely based on the
animal neuron. The processing ability of the network is stored in the inter-unit
connection strengths, or \emph{weights}, obtained by a process of adaptation to,
or \emph{learning} from, a set of training patterns. \cite{GUR}
\end{quote} 
In our context we are only dealing with \emph{Feed Forward Neural Networks}
(FFNN), which will be discussed in detail in the following subsection.\\

\subsection{Feed Forward Neural Networks (FFNN)}

An ANN consists of a finite set of neurons, which are organized in layers:
\begin{itemize}
  \item one input layer
  \item $n$ hidden layer(s), where $n \in \mathbb{N}_{0}$
  \item one output layer
\end{itemize}
The input data is presented to the input neurons.
All neurons perform specific operations on the data and transfer the results to the next
layer (except the output layer, as it is the last layer in the network). In our case the
input neuron's operation is the \emph{id-function}.\\
The connections
among the neurons are weighted, that means, values are changed by some factor. This goes
on until the output layer is reached, and the ``result'' of the computation is expressed
by the numerical values that are presented by the output neurons.\\
In a FFNN a neuron in layer $l$ can only get input from neurons which are situated in
layer(s) $L_{j}$ $(0 \leq L_{j} \leq (l-1))$, where layer 0 denotes the input layer.
And the output produced by the neuron can only be forwarded to neurons which
are in layer(s) $L_{j}$ $((l+1) \leq L_{j} \leq (n+1))$, where layer (n+1) is the output layer.\\

Let us consider a single neuron in the hidden layer $l$.
This neuron has $n_{i}$ input connections from neurons which are situated in the previous
layer(s) $L_{j}$ $(0 \leq L_{j} \leq (l-1))$.
All those connections are weighted with the weights $w_{1}$ to $w_{n_{i}}$. \\

Computation is performed in the following way:


\begin{picture}(400,170)(0,0)

% Neuron (Kreis): Laeßt sich leider nicht groesser darstellen!
\put(200,100){\circle{40}} % senkrechte Linien im Neuron:
\put(193,118.5){\line(0,-1){37}} \put(207,118.5){\line(0,-1){37}}

% Input-Connections:
\put(182,109){\line(-4,1){100}}
\put(180,100){\line(-1,0){100}}
\put(182,91){\line(-4,-1){100}}

% Output-Connections:
\put(218,109){\line(4,1){100}}
\put(220,100){\line(1,0){100}}
\put(218,91){\line(4,-1){100}}

% Beschriftung:
% 1. des Kreises:
\put(185,95){$I$}
\put(195,95){$A$}
\put(209,95){$O$}

% 2. der Input-Connection-Gewichte
\put(100,130){$w_{1}$}
\put(100,103){$w_{2}$}
\put(100,79){$w_{n_i}$}

% 3. der Output-connections-Gewichte
\put(280,130){$w_{1}$}
\put(280,103){$w_{2}$}
\put(280,79){$w_{n_o}$}

% 4. der Input-Connections (Name)
\put(67,135){$x_{1}$}
\put(67,100){$x_{2}$}
\put(67,65){$x_{n_i}$}

% 5. der Output-Connections (Name)
\put(321,135){$y_{1}$}
\put(321,100){$y_{2}$}
\put(321,65){$y_{n_i}$}

% 6. die 'Punkterl' ...
\put(69,80){\shortstack{.\\.\\.}} % fuer die x(e)
\put(103,87){\shortstack{.\\.\\.}} % fuer die Gewichte von x
\put(323,80){\shortstack{.\\.\\.}} % fuer die y(e)
\put(283,87){\shortstack{.\\.\\.}} % fuer die Gewichte von y

% 7. die untenstehenden L_j, ...:
\put(55,30){$L_j, j \in [0,l-1]$}
\put(197,30){$l$}
\put(280,30){$L_j, j \in [l+1,n+1]$}

% 8. die Trennline zwischen den 3 Layern:
\put(150,20){\dashbox{3.5}(0,125)}
\put(250,20){\dashbox{3.5}(0,125)}

% Der Theta-Input:
\put(200,120){\line(-4,1){118}}
\put(103,146){$\theta$}
\put(37,146){$x_0 = -1$}



\end{picture}



\begin{description} \label{pa:comp}
  \item{\textbf{The Input Function $I$:}}
     The summed input that is presented to our neuron is:
     \begin{equation}
     I_{simple} = w_{1}x_{1} + w_{2}x_{2} + \cdots + w_{n_{i}}x_{n_{i}}
     \end{equation}
     where $n_{i}$ is the number of input connections, $x_{i}$ is the signal of the $i$th
     input neuron, and $w_{i}$ is the weight of the $i$th connection.\\
     As an addition, we have a constant weight $\theta$ (the \emph{threshold}), which has
     the value $-1$. \\
     Including $\theta$, $I$ can also be written as:
     \begin{equation} \label{eq:input}
     I = \sum_{k=0}^{n_{i}} w_{k}x_{k}
     \end{equation}
  \item{\textbf{The Activation Function $A$:}}
     Then, an Activation Function is applied on the weighted sum as seen in \ref{eq:input}.
     Mostly, the activation function is the sigmoid function denoted by
     \begin{equation}
     A = \frac {1} {1 + e^{-(I_{simple} + \theta)}} = \frac {1} {1 + e^{-I}}
     \end{equation}
     where $\theta$ denotes the threshold.
  \item{\textbf{The Output Function $O$:}}
     Normally, the output function is the identical function $O(x) = x$, that means,
     it has no further effect on the value computed by the activation function.
\end{description}

Finally, the value computed by our neuron is forwarded to all other neurons in layer(s) $L_{j}$
$((l+1) \leq L_{j} \leq (n+1))$ which are connected to our neuron.
The output connections $y_{1}$ to $y_{n_{o}}$ are weighted similar to the input
connections with the weights $w_{1}$ to $w_{n_{o}}$.

\subsection{The Backpropagation Learning Algorithm}

Although there are many different kinds of learning rules used by ANN we concentrate on the
\emph{Backpropagation Algorithm}, because it is the most widely used learning algorithm for
FFNN.\\
Backpropagation is a training method that uses the \emph{Generalized Delta Rule}. Its basic
idea is to perform a gradient descent on the total squared error of the network output 
(considered as a function of the weights).
It was first described by \emph{Werbos} %\cite{WER}
and made popular by \emph{Rumelhart}'s,
\emph{Hinton}'s and \emph{Williams}' paper \cite{RHW}. \\

ANN learning generally occurs by applying a list of patterns (a \emph{patternset}) on the
network. We can separate this set in
\begin{enumerate}
  \item a \emph{training} patternset, which is used to train the network
  \item a \emph{validation} patternset, to evaluate the training progress during training
  \item a \emph{test} patternset, to test the performance of the net after training
\end{enumerate}
This is called \emph{cross validation}.\\
The pattern of input values is presented to the input nodes, and the computed values
(which are expressed by the output nodes) are compared to the pattern of output values. If those
values differ for an output neuron and the desired output, we can compute the squared error $E$
for a single node
\begin{equation}
E = \sum_{p} \frac {1} {2} (t_{p} - y)^{2}
\end{equation}
where p is one specific pattern in the patternset, $t$ is the target function and $y$ is the
'result function' of the node (the final output of one node, depending on the input, activation
and output function as mentioned in \ref{pa:comp}). \\
Now, for computing the error for a layer of nodes, we have to sum the error $E_{p}$ (for a single
pattern) over the number n of nodes
\begin{equation}
E_{p} = \frac {1} {2} \sum_{j=1}^{n} (t_{p}^{j} - y^{j})^{2}
\end{equation}
As the result function depends on the weights $w_{k}$ (see \ref{eq:input}), we can analyze
the weight change in the output layer:
\begin{equation} \label{eq:bp_output}
\Delta w_{k}^{j} = \alpha \sigma' (a^{j}) (t^{j} - y^{j}) x_{k}^{j}
\end{equation}
where superscript $j$ denotes which node is being described. The term $(t^{j} - y^{j})$ represents
a measure of the error on the $j$th node. The term $\sigma' (a^{j})$ relates to how quickly (rate
of change) the activation function can change the output (and hence the error). If this is large,
we can expect a rapid change in error for a given change in activation. The factor of $x_{k}^{j}$
is related to the amount that the $k$th input has affected the activation. If it is large, the
weight of the $k$th input should be changed in order to get a large change in error. \\
For the $i$th node in a hidden layer, the formula looks like this:
\begin{equation} \label{eq:bp_hidden}
\delta^{i} = \sigma' (a^{k}) \sum_{j \in I_{i}} \delta^{j} w_{i}^{j}
\end{equation}
where $I_{i}$ is the set of nodes which take an input from the hidden node $k$, and the term
$\delta^{k}$ is an abbreviation for $\sigma' (a^{k}) (t^{k} - y^{k})$.\\
As a conclusion, we can give a ``pseudo code´´ for the backpropagation algorithm:\\
\begin{verbatim}
repeat
   for each training pattern
      present the pattern at the input layer
      apply the target pattern to the output layer
      for each output node
         calculate and reduce the error according to 1.6
      end for
      for each hidden node
         calculate and reduce the error according to 1.7
         propagating from the output nodes back towards this node
      end for
   end for
until the error is ``acceptably low''
      (or the network does not converge)
\end{verbatim}

\section{Overview to Sensitivity Analysis}

If a Neural Network is welltrained (the error function converges and the network
is not ``overtrained''), then the net should be able to solve a specific
problem.
In other words, the underlying trend in the data is captured by the network 
\footnote{see \cite{GUR}, chapter 4, p.8}. \\
If we now want to know why the network recognizes the problem structure
it is trained on, we need to perform \emph{Sensitivity Analysis} on it. 
This helps us to analyze the inner structure and working method of the ANN, 
and further, to manually or automatically apply a better structure to it.\\
And, as \emph{Hashem} states it in \cite{HASHEM}:
\begin{quote}
In general, one of the key factors that affect the success of process 
modeling, is the ability to extract information about the model structure
and the relationships between its inputs and outputs from the trained
network. Such information is essential for model validation and for 
process optimization and control. Moreover, in some cases where the
original process is not well understood, this information can be employed
as a basis for the analysis of the process and in determining the most
significant factors that affect it.
\end{quote}

\section{Methods Overview}

\subsection{Pruning Methods}

According to \cite{TF97}, five connection removal methods are presented.
\begin{enumerate}
  \item
      \begin{enumerate}
        \item The simplest method is to select the connections with weights near
              to zero ($w_i \approx 0$) and to erase them.\\

        \item \label{ref:method1}
              As an addition, if the connection's average contribution is added
              to the bias (= threshold) of the neuron that receives input from
              the erased connection.
      \end{enumerate}

  \item The \emph{smallest variance method} proposed by J. Sietsma and R.J.F.
        Dow \cite{SD91} applies the idea of the ``smallest contribution
        variance''
        \[ min(\sigma_{P}( \frac{1}{P} \sum_{p=1}^{P} c^{p} )) \]
        where $c$ is the output of the connection, $P$ is the size of the
        training set, $p$ is a single pattern and $\sigma$ is the contribution
        variance. Then the mean
        output of the connection to be erased is added to the bias
        of the receiving neuron.
  \item E.D. Karnin \cite{KAR90} determines the sensitivity $s$ of a connection by:

     \begin{equation}
     s = \sum_{n=1}^{N} (\Delta w_{n})^2 \frac {w_{n}} {\eta(w_{n} - w_{0})}
     \ \ $if$ \  w_{n} \neq w_{0} $, otherwise $\ s = 0
     \end{equation}

        where $w_{n}$ is the weight in the current training epoch $n$, $w_{0}$ the
        initial weight and $\Delta w_{n}$ the weight change in the $n$-th epoch. $N$
        is the number of training epochs, and $\eta$ is a factor that specifies the
        ``importance'' of the change between the initial weight and the current weight
        in the current training epoch. If $\eta$ is high, a little weight change has
        a big effect on the sensitivity value $s$. \\
        The connections with the lowest sensitivity $s$ are pruned.

  \item M.C. Mozer and P. Smolensky \cite{MOZ89} weight removal method is called
        \emph{skeletonization}. If a neuron is removed the error induced by this
        removal is computed by multiplying the neurons output with an \emph{additional
        strength}.\\
        Setting an additional strength to zero is the same as removing a neuron.
        If the derivative of the error function to these
        additional strengths are small, the neuron is removed.
  \item The 5th sensitivity measure is part of a pruning method called \emph{autoprune}
        \cite{FIN93}. There the probability that a weight becomes zero is tested
        statistically. If this probability is high, then the connection will be removed.\\
        An extension to this pruning method can be found in \cite{PRE95}. The
        socalled pruning method \emph{$\lambda$-prune} estimates how many neurons
        should be removed at each step.
\end{enumerate}


\subsection{Growing Methods}

Growing Methods (or Constructive Methods) start with a minimal network structure
(only input and output neurons) and then add neurons according to specific
methods to gain the ``optimal'' network size.
According to \cite{SMI91}, methods are summarized in the following way:
\begin{enumerate}
  \item{The Upstart Algorithm} \cite{FRE90} \\
       The first step is to train the minimal network until no further error
       reduction can be reached. Then it is determined for which patterns the output
       is incorrectly responding. The attempt is to add ``daughter units'' to the 
       incorrectly responding output unit, which are supposed to correct the input
       of the output unit. Again,
       the new network is trained, and this cycle goes on until the error is
       low enough.
  \item{Cascade Correlation} \cite{FL90} \\
       This algorithm adds a hidden unit (the \emph{candidate unit}) to a trained, 
       but not ``big enough'' network in a way that it gets input from all input 
       units and all (until then) existing hidden units and delivers its output 
       to all output units. The basic idea is to equalize the output of the candidate
       unit and the error of the actual output units for a specific pattern $p$.
       After the connections from the input and hidden units to the candidate unit
       are trained, they are ``frozen'' (that means, they can not be changed during
       further training), and then the connections from the candidate unit to the
       output units are trained. \\
       The cascade-correlation algorithm produces very deep networks (that is, they 
       have got a large number of hidden layers, but usually very few neurons in 
       each of them).
  \item{The Tiling Algorithm} \cite{MN89} \\
       This algorithm adds a new layer containing an output unit (or a set of output 
       units) to the ``top'' (the former output layer) of the existing net. These units
       are either connected only to the former output units and the input units, or
       to additional ``ancillary units''.       
  \item{Dynamic Node Creation} \cite{ASH89} \\
       This method is more empirical than the others, as it adds a new hidden
       neuron (if the previous rate decrease of the net's error falls below a
       certain value) with randomized initial weights connecting between the
       input and output layers. It is left to the user to adjust certain parameters,
       such as the minimum period between node additions, when to stop adding nodes,
       or what too slow a rate is.
\end{enumerate}


%\subsection{Evolutionary Design Methods}



\section{Experimenting with SNNS}

\begin{enumerate}
\item Starting SNNS

\item Creating an XOR network

First, we chose ``bignet'' to create a feedworward network.\\
In the opening dialog box we defined the planes for our network,
such that we had two neurons in both the input layer and the
hidden layer (x=1, y=2) and one neuron in the output layer
(x=1, y=1). Then we selected ``full connection''
to connect each neuron to every neuron in the following layer.\\
Finally, we created our net by selecting ``create net''.

\item Displaying the network

The command ``display'' opens a window showing the neurons in
the specific layers. To also display the connections, we had
to set the option ``links on'' under ``setup''.

\item Training the network

As a starting point, we loaded the pattern file \texttt{xor.pat}.
Under the option ``control'' we entered the number of cycles that
the network should be trained with the patternset. As training
function, we selected ``Std. Backpropagation''. Then we started
training the network.


\begin{picture}(460,160)(0,0)

% Ueberschriften:
\put(-10,140){\shortstack{2-2-1: \\full connected}}
\put(80,140){\shortstack{2-1-1: \\shortcut connected}}
\put(210,140){\shortstack{2-5-1: \\shortcut connected}}
\put(345,140){\shortstack{2-2-1: \\shortcut connected}}


% Figure 1 (A net, that didn't converge: 2-2-1 full connected.

\put(10,10){\circle{10}}
\put(40,10){\circle{10}}
\put(10,65){\circle{10}}
\put(40,65){\circle{10}}

\put(10,15){\line(0,1){45}} % linke gerade Connection nach oben
\put(40,15){\line(0,1){45}} % rechte gerade Connection nach oben
\put(10,15){\line(2,3){30}} % schraege Linie nach rechts oben
\put(40,15){\line(-2,3){30}} % schraege Line nach links oben

% Output - Neuron:
\put(25,120){\circle{10}}

% Connections fuer das Output Neuron:
\put(10,70){\line(1,3){15}} % schraege Linie nach rechts oben
\put(40,70){\line(-1,3){15}} % schraege Linie nach links oben

%--------------------------------------------------------------------
% Figure 2 (A net, that converges: 2-1-1 shortcut connected.

\put(100,10){\circle{10}}
\put(150,10){\circle{10}}
\put(125,68){\circle{10}}

\put(100,15){\line(1,2){24}} % schraege Linie nach rechts oben
\put(150,15){\line(-1,2){24}} % schraege Linie nach links oben

% Output - Neuron:
\put(125,130){\circle{10}}

% Connections fuer das Output Neuron:
\put(125,73){\line(0,1){52}} % gerade Connection nach oben
\put(100,15){\line(1,5){22.15}} % schraege Linie nach rechts oben
\put(150,15){\line(-1,5){22.15}} % schraege Linie nach links oben


%--------------------------------------------------------------------
% Figure 3 (A net, that converges: 2-5-1 shortcut connected.

% Input-Neuronen:
\put(245,10){\circle{10}}
\put(275,10){\circle{10}}

% Hidden Layer:
\put(200,65){\circle{10}}
\put(230,65){\circle{10}}
\put(260,65){\circle{10}}
\put(290,65){\circle{10}}
\put(320,65){\circle{10}}

% Connection vom LINKEN Input-NEURON zu Hidden-Layer:
\put(245,15){\line(-1,1){45}} % links aussen
\put(245,15){\line(-1,3){15}} % 2te von links
\put(245,15){\line(1,3){15}} % Mitte
\put(245,15){\line(1,1){45}} % 2te von rechts
\put(245,15){\line(3,2){71}} % rechts aussen


% Connection vom RECHTEN Input-NEURON zu Hidden-Layer:
\put(275,15){\line(-3,2){71}} %links aussen
\put(275,15){\line(-1,1){45}} % 2te von links
\put(275,15){\line(-1,3){15}} % Mitte
\put(275,15){\line(1,3){15}} % 2te von rechts
\put(275,15){\line(1,1){45}} % rechts aussen

% Output - Neuron:
\put(260,120){\circle{10}}

% Connections fuer das Output Neuron vom Hidden Layer!:
\put(200,70){\line(5,4){57.1}} % links aussen zu Output
\put(230,70){\line(2,3){30}} % 2te links aussen zu Output
\put(260,70){\line(0,1){45}} % Gerade in der Mitte zu Output
\put(290,70){\line(-2,3){30}} % 2te rechts aussen zu Output
\put(320,70){\line(-5,4){57.1}} % rechts aussen zu Output

% connections von input zu hidden layer!:
\put(245,25){\line(1,6){15}}
\put(275,25){\line(-1,6){15}}
\put(245,15){\line(0,1){10}}
\put(275,15){\line(0,1){10}}



%--------------------------------------------------------------------
% Figure 4 (A net, that converges: 2-2-1 shortcut connected.

\put(390,10){\circle{10}}
\put(420,10){\circle{10}}
\put(390,65){\circle{10}}
\put(420,65){\circle{10}}

\put(390,15){\line(0,1){45}} % linke gerade Connection nach oben
\put(420,15){\line(0,1){45}} % rechte gerade Connection nach oben
\put(390,15){\line(2,3){30}} % schraege Linie nach rechts oben
\put(420,15){\line(-2,3){30}} % schraege Line nach links oben

% Output - Neuron:
\put(405,120){\circle{10}}

% Connections fuer das Output Neuron:
\put(390,70){\line(1,3){15}} % schraege Linie nach rechts oben
\put(420,70){\line(-1,3){15}} % schraege Linie nach links oben

% Connections von input zu hidden layer: (weil "shortcut")
\put(390,25){\line(1,6){15}}
\put(420,25){\line(-1,6){15}}
\put(390,15){\line(0,1){10}}
\put(420,15){\line(0,1){10}}


\end{picture}



After experimenting for a while we had to admit that our net did
not converge after 10.000 cycles. \\
So, we tried another net structure,
consisting of two neurons in the input layer, one neuron in both
the hidden and the output layer and shortcut connections. This
structure converged after 2000 cycles.\\
As we could not get our first net structure learning, we tried
to add hidden neurons to it. With five hidden neurons and shortcut
connections, the error was under 0.10 after 7000 cycles.\\
Inspired that way, we tried our original 2-2-1 structure with
shortcut connections. To our surprise the error was under 0.05
after 7500 cycles.


\item Trying to find out why our first XOR net did not work

We realized that the our first 2-2-1 XOR net (with full connection) did not
converge to the fact of we were using the sigmoid function as transfer
function. So we repeated the experiment with the same net structure, but using
a heaviside function as transfer function this time.
The experiment worked and the network's error was near to zero after 2500
training cycles.

\end{enumerate}


\subsection{Calculating An XOR Net}

As we had big problems training a simple XOR net with two
input and two hidden neurons, one output neuron and full connection,
we tried to find out which weight values would be the best.
We assumed a heaviside function as activation function.\\

Our net (with description):


\begin{center}
\begin{picture}(100,140)(0,0)
% Figure 1 (A net, that didn't converge: 2-2-1 full connected.

\put(20,10){\circle{10}}
\put(50,10){\circle{10}}
\put(20,65){\circle{10}}
\put(50,65){\circle{10}}

\put(20,15){\line(0,1){45}} % linke gerade Connection nach oben
\put(50,15){\line(0,1){45}} % rechte gerade Connection nach oben
\put(20,15){\line(2,3){30}} % schraege Linie nach rechts oben
\put(50,15){\line(-2,3){30}} % schraege Line nach links oben

% Output - Neuron:
\put(35,120){\circle{10}}

% Connections fuer das Output Neuron:
\put(20,70){\line(1,3){15}} % schraege Linie nach rechts oben
\put(50,70){\line(-1,3){15}} % schraege Linie nach links oben


% Beschriftung:
% der Inuput Neuronen:
\put(17,8){$x$}
\put(47,8){$y$}

% die w_{i}s:
\put(6,40){$w_{1}$}
\put(22,55){$w_{2}$}
\put(34,57){$w_{3}$}
\put(53,40){$w_{4}$}
\put(11,85){$w_{5}$}
\put(45,85){$w_{6}$}

% die Thetas:
\put(0,60){$\theta_{1}$}
\put(60,60){$\theta_{2}$}
\put(42,115){$\theta_{3}$}


\end{picture}
\end{center}


Our equations were as follows: \\

For the first node in the hidden layer, the input function is
\begin{displaymath} % \label{eq:XORi1}
  I_{1} = x*w_{1} + y*w_{2} - \theta_{1}
\end{displaymath}
where $x$ and $y$ denote the input values represented by the two
input neurons.
Its activation function is the heaviside function specified as follows:
\begin{displaymath}
  A_{1} = \left\{ \begin{array}{ll}
                  0 &amp; \mbox{if $I_1 &gt; k_1$}\\
                  1 &amp; \mbox{if $I_1 \leq k_1$}
                \end{array}
        \right.
\end{displaymath}

Accordingly, these formulae are written for the second hidden
node:
\begin{displaymath}
  I_{2} = x*w_{3} + y*w_{4} - \theta_{2}
\end{displaymath}
\begin{displaymath}
  A_{2} = \left\{ \begin{array}{ll}
                  0 &amp; \mbox{if $I_2 &gt; k_2$}\\
                  1 &amp; \mbox{if $I_2 \leq k_2$}
                \end{array}
        \right.
\end{displaymath}

For the output node, the calculations look like this:
\begin{displaymath} \label{eq:XORi3}
  I_{3} = A_{1}*w_{5} + A_{2}*w_{6} - \theta_{3}
  % gibt's hier überhaupt ein THETA_3?
\end{displaymath}
\begin{displaymath}
  A_{3} = \left\{ \begin{array}{ll}
                  0 &amp; \mbox{if $I_3 &gt; k_3$}\\
                  1 &amp; \mbox{if $I_3 \leq k_3$}
                \end{array}
        \right.
\end{displaymath}
As $\theta_{i}$ can always be added on $k_{i}$, we perform the
following renaming:
\[ \theta_{1} + k_{1} = k_{1}^{*}, \ \ \
   \theta_{2} + k_{2} = k_{2}^{*}, \ \ \
   \theta_{3} + k_{3} = k_{3}^{*}. \]

We now distinguish among two cases of calculation:\\
The ``zero-cases'' where $x=y=0$ and $x=y=1$ and the net output
should be zero, and the ``one-cases'' where either $x=0$ and $y=1$
or $x=1$ and $y=0$ and the net output is expected to be one.

\begin{itemize}
  \item{The Zero-Cases:}\\
     In the first case, we set as a condition that $A_{1}=A_{2}=0$.
     Then we have $I_{1} &gt; k_{1}$ and $I_{2} &gt; k_{2}$.
     For the first case ($x=y=0$), we get 
     \[ 0 &gt; k_{1}^{*} \; , \ \ \ 0 &gt; k_{2}^{*} \; , \]
     and for the second case ($x=y=1$)
     \[ w_{1} + w_{2} &gt; k_{1}^{*} \; , \ \ \
        w_{3} + w_{4} &gt; k_{2}^{*} \; . \]
  \item{The One-Cases:}\\
     Here, the condition is that $A_{1} \neq A_{2}$.
     For the first case ($x=0$ and $y=1$) we suppose that $A_{1}=1$ 
     and $A_{2}=0$. This means that $I_{1} \leq k_{1}$ and $I_{2}
     &gt; k_{2}$, and thus
     \[ w_{2} \leq k_{1}^{*} \; , \ \ \
        w_{4} &gt; k_{2}^{*} \; . \]
     In the second case ($x=1$ and $y=0$), where $A_{1}=0$ and 
     $A_{2}=1$, we get
     \[ w_{1} &gt; k_{1}^{*} \; , \ \ \
        w_{3} \leq k_{2}^{*} \; . \]      
  \item{The output node weights:}\\
     The output node checks if $A_{1}=A_{2}$, or $A_{1} \neq A_{2}$.
     For the first case ($A_{3} = 0$, and $I_{3} &gt; k_{3}$) we get
     \[ 0 &gt; k_{3}^{*} \; , \]
     and for the cases that $A_{1}=1$ or $A_{2}=1$ ($A_{3}=1$ and
     $I_{3} \leq k_{3}$)
     \[ w_{5} \leq k_{3}^{*} \; , \ \ \
        w_{6} \leq k_{3}^{*} \; . \]
  \item{Conclusions:} \\
     Considering the formulae $w_{1} + w_{2} &gt; k_{1}^{*}$, $w_{2} 
     \leq k_{1}^{*}$, $w_{1} &gt; k_{1}^{*}$, and $k_{1}^{*} &lt; 0$, 
     it follows that
     \[ w_{1} &gt; 0 \; , \ \ \ w_{2} \leq k_{1}^{*} \; , \]
     and accordingly
     \[ w_{3} \leq k_{2}^{*} \; , \ \ \ w_{4} &gt; 0 \; . \]
 \end{itemize}

\chapter{Implementing a simple pruning method}

\section{Description}

For a description of this method, see \ref{ref:method1} on page~\pageref{ref:method1}.\\

The program in which we implemented this algorithm is called
\emph{netVis}\footnote{A program which displays the structure of a neural net.}.\\

We decided that connections which got pruned should not be displayed
in the visualisation of the neural net (after pruning), but we
implemented a list, which contains all pruned connections. With a
click on an item of the list the selected (already pruned) connection
is displayed as a bold, red line in the net structure.\\
The error sum is displayed before and after pruning (if possible).\\

A possible test case for this implementation is to use a net from
``ALGAE''\footnote{A program that simulates algae using neural nets.},
to prune it with \emph{netVis} and test it again in \emph{ALGAE}.

\section{The algorithm}

\begin{enumerate}
\item Search for the average (connection-) weights
      \[ \frac{1}{P} \sum_{p=1}^{P} c^{p} &lt; x \]
      and mark every connection found as ``TO ERASE''.
\item ``Feed'' the neural net with patterns and calculate the
      ``$\theta$-Output'' (``transmission value'') of the connections.
\item Add the $\theta$-Output to the bias of the node at which the
      connection ends.
\end{enumerate}


\section{Experimenting with XOR}
\ldots

\section{Experimenting with a neural net from ALGAE}
\ldots

\chapter{The ``Quadrantenproblem''}
\section{Description}
The ``Quadrantenproblem'' indicates the position (which quadrant) of
the point $(x,y)$ in a coordinate system which is divided into
four quadrants.\\

\section{Implementation}
As we tried to create a net in the SNNS, we recognized that it would
not be possible to train it, to the fact we had no pattern file.\\
So we decided to write our own pattern file. It consists of 144 patterns.\\

We had two main interpretations of the problem.
The first interpretation is as follows:
\begin{itemize}
    \item Output neuron 1 is the x coordinate
    \item Output neuron 2 is the y coordinate
    \item Output neuron 3 ``delivers'' the value the calculated quadrant has
\end{itemize}
So a output like (0,1,0) means: input lies in upper left quadrant and has the
                                value 0.\\
~\\
The second interpretation says that each of the four output nodes represents
one quadrant. ``The winner takes it all'' is the principle -- the output node
which shows the highest value indicates that the corresponding quadrant is
the solution.

\section{Hybrid methods}
We also searched on the web for some documents about hybrid methods,
but in all papers we found, ``hybrid'' meaned: grow net, then prune net
(without detailed description of e.g. when the best time to stop growing
and begin pruning is).




\appendix

\chapter{Protocols}

\section{2001/03/21}

\begin{enumerate}
  \item Responsible person for protocol: Annemarie Mayer
  \item Participating persons: Sebastian Gumpold, Annemarie Mayer
  \item Topics talked about:
    \begin{itemize}
      \item What is a neural network?
      \item What is sensitivity analysis?
      \item Which methods can be used to perform SA?
    \end{itemize}
  \item New assignments
    \begin{itemize}
      \item Introduction
    \end{itemize}
  \item Open questions
    \begin{itemize}
      \item Is there a possibility to analyse only single neurons?
      \item Under which aspects shall we order the different methods?
      \item Shall we develop our own methods?
      \item Do we have to treat this topic only theoretically?
      \item \ldots
    \end{itemize}
\end{enumerate}
\newpage

\section{2001/03/28}

\begin{enumerate}
  \item Responsible person for protocol: Sebastian Gumpold
  \item Participating persons: Annemarie Mayer, Sebastian Gumpold
  \item Topics talked about:
    \begin{itemize}
      \item Feedforward Neural Networks (FFNN)
      \item Abstract
      \item Introduction to ANNs
      \item Discussion of methods to determine a network's optimal size
    \end{itemize}
  \item New assignments
    \begin{itemize}
      \item 'Evolve' Introduction
    \end{itemize}
  \item Open questions
    \begin{itemize}
      \item Should we search for further methods (pruning, growing, evolution) on the web?
      \item Shall we find/'invent' algorithms for automating the NN design process?
      \item \ldots
    \end{itemize}
  \end{enumerate}
\newpage


\section{2001/03/31}

\begin{enumerate}
  \item Responsible person for protocol: Sebastian Gumpold
  \item Participating persons: Annemarie Mayer, Sebastian Gumpold
  \item Topics talked about:
    \begin{itemize}
      \item Examples with the SNNS (especially the XOR-Problem)
      \item Search for new methods/sources concerning our topic
      \item Changes in Text (as suggested by Prof. Schwaiger)
      \item Changes in Bibliography (adding URLs where possible)
    \end{itemize}
  \item New assignments
    \begin{itemize}
      \item Gurney's paper: URL
      \item Neurex (next time)
      \item Gorman und Sejnowski's paper (next time) \ldots
    \end{itemize}
  \item Open questions
    \begin{itemize}
      \item \ldots
    \end{itemize}
  \end{enumerate}
\newpage


\section{2001/04/06}

\begin{enumerate}
  \item Responsible person for protocol: Annemarie Mayer
  \item Participating persons: Mag. DI Roland Schwaiger, Sebastian Gumpold,
                               Annemarie Mayer, Clemens Bauer
  \item Topics talked about:
    \begin{itemize}
      \item Errors in the last version of the paper
      \item \ldots
    \end{itemize}
  \item New assignments
    \begin{itemize}
      \item Calculate if our first net would have worked with a heaviside function
      \item Implement the pruning method 1.b in netVis
      \item \ldots
    \end{itemize}
  \item Open questions
    \begin{itemize}
      \item \ldots
    \end{itemize}
\end{enumerate}
\newpage


\section{2001/04/26}

\begin{enumerate}
  \item Responsible person for protocol: Sebastian Gumpold
  \item Participating persons: Mag. DI Roland Schwaiger, Sebastian Gumpold,
                               Annemarie Mayer, Clemens Bauer
  \item Topics talked about:
    \begin{itemize}
      \item \ldots
    \end{itemize}
  \item New assignments
    \begin{itemize}
      \item Simulate the ``Quadrantenproblem'' in SNNS (also try pruning
            some nodes)
      \item Find documents about:
          \begin{itemize}
             \item Fusion of growing and pruning (documents which show hybrid beginnings)
             \item Are hybrid methods used in Robotics?
             \item Appliance of growing/pruning in ``real world problems''
          \end{itemize}
      \item Study new document (Burascano) as an addition to the Neurex paper
      \item \ldots
    \end{itemize}
  \item Open questions
    \begin{itemize}
      \item \ldots
    \end{itemize}
\end{enumerate}
\newpage


\bibliographystyle{ieeetr}
\addcontentsline{toc}{chapter}{Bibliography}
\bibliography{paper}

\end{document}
</pre></body></html>