\section{Introduction}

\subsection{Problem Description}

One of the current trends in the field of Artificial Neural Net (ANN) 
research is the use of activation functions other than the classics,
such as sigmoid or heaviside. Periodic activation functions, such as sine, 
that have long been thought of as unsuitable for ANNs, have proven to lead
to results better, or in the worst case equal to the ones achieved with
sigmoid activation functions\footnote{see \cite{sopena} and section \ref{rellit:sopena}}.
When trying to solve classic ANN benchmark 
problems (with an analytical approach) we found that the best results can 
be found when different activation functions are used.\\
Another topic that was of interest to our project was the evolvation of
ANNs. The Departement of Computer Science at the University of Salzburg 
runs a research program called ``siGis'' (which stands for ``Salzburg 
Interest Group on Integrated Systems''), that uses a system for evolving
the structure of ANNs (NetGEN)\footnote{for more detail see ``http://www.cosy.sbg.ac.at/project/sigis/sigis.html''}
.\\
When taking these two developments into account, it seems clear that
an evolvable activation function could lead to an improvement in ANN
performance.\\
The first approach of siGis was to use function tables, which completely 
changes the function when mutated. For this project, we chose an approach 
which makes a more subtle way of changing the activation function possible. We 
interpolate the activation function from an evolvable point set, using the
cubic spline interpolation method, which fulfills the criteria of an 
activation function in the SNNS\footnote{Stuttgart Neural Net Simulator}, being smooth in the first 
derivative, and continuous in the second derivative, both within the interval 
and at its boundaries.\\
%This paper deals with the process of implementing and testing this feature.


\subsection{What is the Activation Function?}
ANNs consist of two rough components: ``neurons'' (called units) and the 
connections between them. A single unit's input is the sum of all weighted
inputs. Mathematically expressed this means $inp = \sum_{i} x_{i} w_{i}$, $x_{i}$ 
being the input values and $w_{i}$ the respective weights.\\
The units output (on all outgoing connections) is $f(inp)$. This function $f$
is the ``activation function''. The fitness of a net can be measured in the squared
standard error (SSE), which is calculated as follows:\\
t \ldots target output\\
a \ldots actual output\\
n \ldots number of test patterns\\

$$ SSE = \sum_{i=1}^{n}(t_i-a_i)^2$$



\subsection{Fitness of Different Activation Functions}
\label{intr:XOR}
More fit activation functions lead to a drastic decrease of the ANN's 
complexity, which leads to much shorter learning times. Let us take a look
at the XOR-Problem to illustrate this:

\begin{table}[!h]
\parbox{4cm}{
\small
\centering
\begin{tabular}{ccc}
x & y & x XOR y\\[0.5ex]
0 & 0 & 0\\
0 & 1 & 1\\
1 & 0 & 1\\
1 & 1 & 0
\end{tabular}
\caption{\small XOR function table} 
}
\hfill
\parbox{8cm}{
The given set
is to be split into two classes by an ANN. By simply taking a look at the 
problem's graph it becomes evident that no linear function can find an 
adequate solution for this problem. 
}
\end{table}

\begin{figure}[!h]
\noindent 
\parbox{8cm}{
Furthermore, even ANNs with monotonic activation function without a hidden 
layer cannot solve it. We hence decided to use the sine function in the output 
layer. If the correct weights ($\frac{\pi}{2}$) are to be found by supervised 
learning, it is only possible with positive initial weights, otherwise the learning 
algorithm is trapped in local minima. If a customized cubic spline function is used, 
the standard squared error is below 0.01 after only 15 steps.
}
\hfill
\begin{minipage}{4cm}
\xymatrix{
 & & & \\
\ar@{}[rr]|<<<<<<<<<{1} & \bullet & \circ & \\
\ar[rrr]_>{x} & \circ \ar@{}[rr]_1& \bullet & \\
& \ar[uuu]^>{y}& & 
}
\caption{\small XOR function graph}
\end{minipage}
\end{figure}
