boone.training

Class AdamTrainer

java.lang.Object

boone.Trainer

boone.training.BackpropTrainer

boone.training.AdamTrainer

All Implemented Interfaces:

Storable, java.lang.Cloneable

public class AdamTrainer
extends BackpropTrainer

The Adaptive Moment Estimation trainer. Instead of directly using the gradient information, a specific weight update value is calculated for each weight (and bias) based on a moving average of the gradient (first moment) and a moving average of the squared gradient (second moment). The bias corrected averages are then used as a regularization term for the weight update. In a sense the learning rate is adapted, but it may still be beneficial to optimize the basic learn rate to a specific pproblem. This method seems to work well with mini-batches, but has some problems in batch mode.

Since:

April 2021

Version:

$Id$

Author:

Helmut A. Mayer

Field Summary

Fields inherited from class boone.Trainer

currentEpoch, epochs, learnRate, net, shuffle, stepMode, testData, trainingData, trainingSignalGenerator

Constructor Summary

Constructors

Constructor and Description
AdamTrainer() Creates an Adam trainer with a learn rate of 0.001 and a batch size of 32.

Method Summary

Modifier and Type	Method and Description
protected double	calcAdaptation(BrainPart part) Calculates the Adam adaptation using estimated first and second moments of the gradient.
void	fromXML(org.jdom2.Element element) Reads attributes from XML.
double	getBeta1() Returns the value of beta1 used for the previous mean gradient.
double	getBeta2() Returns the value of beta2 used for the second moment of the gradient.
double	getEpsilon() Returns the fuzz factor used to avoid numerical problems.
protected void	reset() Resets bias correction for a fresh training procedure.
void	setBeta1(double beta1) Sets the value of the weight of the current gradient.
void	setBeta2(double beta2) Sets the value of the weight of the second moment of the gradient.
void	setEpsilon(double epsilon) Returns the fuzz factor used to avoid numerical problems.
org.jdom2.Element	toXML(org.jdom2.Element element) Writes specific elements to XML tree.

Methods inherited from class boone.training.BackpropTrainer

endPattern, endTrain, train, train, train

Methods inherited from class boone.Trainer

clone, endBatch, getBatchSize, getEpochs, getLearnRate, getNetwork, getRank, getTestData, getTrainingData, getTrainingSignalGenerator, getWinningNeuron, setBatchSize, setEpochs, setLearnRate, setNetwork, setShuffle, setStepMode, setTestData, setTrainingData, setTrainingSignalGenerator, test, test, toString, train, trainPattern, updateLinks, updateNeurons

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

AdamTrainer

public AdamTrainer()

Creates an Adam trainer with a learn rate of 0.001 and a batch size of 32.

Method Detail

reset

protected void reset()

Resets bias correction for a fresh training procedure.

Overrides:

reset in class BackpropTrainer

calcAdaptation

protected double calcAdaptation(BrainPart part)

Calculates the Adam adaptation using estimated first and second moments of the gradient.

Overrides:

calcAdaptation in class BackpropTrainer

Parameters:

part - the brain part to be adapted

Returns:

the Adam adaptation value

getBeta1

public double getBeta1()

Returns the value of beta1 used for the previous mean gradient. The weight of the current gradient is then (1 - beta1), so the value has to be in the unit interval.

Returns:

weight value of current gradient

setBeta1

public void setBeta1(double beta1)

Sets the value of the weight of the current gradient.

Parameters:

beta1 - weight value (in unit interval)

getBeta2

public double getBeta2()

Returns the value of beta2 used for the second moment of the gradient. The weight of the current gradient is then (1 - beta2), so the value has to be in the unit interval.

Returns:

weight value of current gradient

setBeta2

public void setBeta2(double beta2)

Sets the value of the weight of the second moment of the gradient.

Parameters:

beta2 - weight value (in unit interval)

getEpsilon

public double getEpsilon()

Returns the fuzz factor used to avoid numerical problems.

Returns:

the fuzz factor value

setEpsilon

public void setEpsilon(double epsilon)

Returns the fuzz factor used to avoid numerical problems.

Parameters:

epsilon - the fuzz factor value

toXML

public org.jdom2.Element toXML(org.jdom2.Element element)

Writes specific elements to XML tree.

Specified by:

toXML in interface Storable

Overrides:

toXML in class Trainer

Parameters:

element - the parent element

Returns:

the trainer element

fromXML

public void fromXML(org.jdom2.Element element)

Reads attributes from XML.

Specified by:

fromXML in interface Storable

Overrides:

fromXML in class Trainer

Parameters:

element - a JDOM element