\section{Preliminary Literature}
\label{rellit:sopena}

For the readers convenience we include a summary of
	the article \textsl{Neural Networks with Periodic and 
	Monotonic Activation Function: a Comparative Study in Classification Problems} 
	\cite{sopena}.

\subsection{Introduction}

The article discusses a number of reasons why the use of sine activation functions can
lead to a significant improvement in performance of multilayer feed-forward networks.

A classification system assigns a paricular class to each set of values of variables
(the input space). A learning algorithm leads a system from an initial state to a minimum 
of error. Backpropagation is most frequently used.

To improve performance of neural networks several modifications of backpropagation have
been proposed, but none of these proposals have considered modifying the activation
function. Recent experiments with periodic activation functions (sine) have shown that
they are superior in learning speed, computational capacity and generalization.



\subsection{Characteristics of Periodic \\Activation Functions}

The use of a periodic function as a classifier divides the input space into an infinitive
number of regions, opposed to the two regions created by monotonic activation functions.

Consider the formula
$$ g(\vec{x}) = bias_o + \sum_{i=1}^{n_{hid}} \alpha_i sin(\vec{x}\vec{\omega}_i + bias_i)$$
as a representative sinusoid function.
The weights ($\vec{\omega}_i$) are called non-linear parameters. The weights of the connections
between layers ($\alpha_i$) are called linear parameters. The biases can be considered as weights 
with a simple transformation.


This allows approximation of more complex functions. Training a neural network of this
kind is basically constructing a Fourier series, the only difference being the possibility
of the network to adjust the values of the non-linear parameters.

Networks with sine functions have much more local minima in the error surface, in which
the error correction algorithm can be easily trapped before reaching the global minimum,
making sufficient variation from the initial position of the weights difficult.
This probability can be diminished by assigning an initial set of weights that is 
appropriate for the problem in question.

Non-linear parameters are difficult to adjust, thus they evolve much more slowly than linear 
parameters. To enhance the performance it proved to be convenient to assign different values
of learning raters for each layer.



\subsection{Experiments}

Learning speed is measured in the number of epochs required to learn the training set.
The learning was considered to be complete when the examples had been solved up to a given
tolerance. Partly another criterion has been applied, when learning all examples was 
impossible or not convenient for generalization purposes. The architecture used was
feed-forward with a layer of hidden units. The activation function was sinusoidal, the
initial weights were assigned randomly from a particular interval.\\
Momentum was set to 0. Without a bias, the network can only produce odd-like functions.
A possible approach to the conversion of the target function into an odd-like function is to
rescale the variables of the input space between 0 and 1.

\subsubsection{Two-Spirals Problem}
The points of two interlocking spirals are to be identified. The training set comprises 
194 points. \\
Standard BP with simple structure fails at this problem. The best algorithm (``Cascade
Correlation'') using standard BP with complex architecture required 1,700 epochs to learn
this problem, an algorithm, using sine as the activation function in the hidden layer 
and hyperbolic tangent in the output layer, only an average of 42.3 epochs. The range
of the initial weights is important. With small ranges learning is not possible.

\subsubsection{Separation Problem}
In this problem, like in the ``two spirals'' problem, the standard sigmoid function with
BP is trapped in local minima. Using sines instead of sigmoids not only prevents the
ANN from being trapped, but also allows it to learn around 180 times faster.

\subsubsection{Quadrants}
All points in quadrant one and three have target 1, the others have target 0.
This problem seems to be well suited for sigmoid activation functions, but a 2-2-1 
architecture with sine as activation function in the hidden layer and sigmoid in the
output layer learns faster than sigmoid function network.


\subsection{Conclusion}

Networks using sine functions are superior in performance to sigmoids (in terms of 
learning speed, computational capacity and generalization). Tasks impossible to learn 
for sigmoid functions were easily learned using sine functions. The main problem is
finding an initial set of weights in high-dimensional inputs. 
