boone.training

Class CrossEntropy

java.lang.Object

boone.training.TrainingSignalGenerator

boone.training.CrossEntropy

All Implemented Interfaces:

Storable, java.lang.Cloneable

public class CrossEntropy
extends TrainingSignalGenerator

The square error function is the default for training.

(BackpropTrainer, RpropTrainer). The error term that is used is

E = 1/2 (t - o)² .

However, this function is not needed here. Instead, the Backprop trainers expect the value of a specific part of the derivative of this function. Specifically, the weight update Δw for the above error function is:

Δw = - η ⋅ ∂E/∂w, thus
Δw = - η ⋅ (t - o) ⋅ - 1 ⋅ ∂o/∂w

The back-propagation trainers only expect

error signal = (t - o)

as the so-called "error signal". The rest of the term, "- η ⋅ - 1 ⋅ ∂o/∂w", does not need to be calculated.

So for example, if another error term

E = 1/2 (t - o²)²

was used, the weight change would be

Δw = - η ⋅ ∂E/∂w = - η (t-o²) * - 2 o * ∂o/∂w

and this function would need to return

error signal = (t - o²) ⋅ 2 o .

Constructor Summary

Constructors

Constructor and Description
CrossEntropy()

Method Summary

Modifier and Type	Method and Description
double	computeError(NeuralNet net, java.util.List<java.lang.Double> input, java.util.List<java.lang.Double> targets) Computes the cross entropy at the output layer for the given pattern.
void	computeErrorSignal(NeuralNet net, java.util.List<java.lang.Double> input, java.util.List<java.lang.Double> targets) Computes and sets the error signal for each output neuron to target / output.

Methods inherited from class boone.training.TrainingSignalGenerator

clone, fromXML, toXML

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CrossEntropy

public CrossEntropy()

Method Detail

computeErrorSignal

public void computeErrorSignal(NeuralNet net,
                               java.util.List<java.lang.Double> input,
                               java.util.List<java.lang.Double> targets)

Computes and sets the error signal for each output neuron to target / output. Note that the derivative of the neuron output is missing here, as it is later correctly multiplied to this signal (just for technical reasons). The problem of a zero output is handled by adding a small epsilon = 1.0E-15 to the output value.

Specified by:

computeErrorSignal in class TrainingSignalGenerator

Parameters:

net - the neural network

input - the input pattern

targets - the target pattern

computeError

public double computeError(NeuralNet net,
                           java.util.List<java.lang.Double> input,
                           java.util.List<java.lang.Double> targets)

Computes the cross entropy at the output layer for the given pattern. The target values resemble the probabilities of the correct distribution.

Specified by:

computeError in class TrainingSignalGenerator

Parameters:

net - the neural network

input - the input pattern

targets - the target pattern

Returns:

the cross entropy [nats]