## Online Local Gain Adaptation for Multi-Layer Perceptrons

N. N. Schraudolph. ** Online
Local Gain Adaptation for Multi-Layer Perceptrons**. Technical Report IDSIA-09-98,
Istituto Dalle Molle di Studi sull'Intelligenza Artificiale, 1998.

### Download

423.0kB | 121.8kB | 217.5kB |

### Abstract

We introduce a new method for adapting the step size of each individual weight in a multi-layer perceptron trained by stochastic gradient descent. Our technique derives from the K1 algorithm for linear systems (Sutton, 1992), which in turn is based on a diagonalized Kalman Filter. We expand upon Sutton's work in two regards: K1 is a) extended to nonlinear systems, and b) made more efficient by linearizing an exponentiation operation. The resulting ELK1 (extended, linearized K1) algorithm is computationally little more expensive than alternative proposals (Zimmermann, 1994; Almeida et al., 1999), and does not require an arbitrary smoothing parameter. On a first benchmark problem ELK1 clearly outperforms these alternatives, as well as stochastic gradient descent with momentum, even when the number of floating-point operations required per weight update is taken into account. Unlike the method of Almeida et al., ELK1 does not require statistical independence between successive training patterns, and handles large initial learning rates well.

### BibTeX Entry

@techreport{olga, author = {Nicol N. Schraudolph}, title = {\href{http://nic.schraudolph.org/pubs/olga.pdf}{ Online Local Gain Adaptation for Multi-Layer Perceptrons}}, number = {IDSIA-09-98}, institution = {Istituto Dalle Molle di Studi sull'Intelligenza Artificiale}, address = {Galleria 2, CH-6928 Manno, Switzerland}, year = 1998, b2h_type = {Other}, b2h_topic = {>Stochastic Meta-Descent}, abstract = { We introduce a new method for adapting the step size of each individual weight in a multi-layer perceptron trained by stochastic gradient descent. Our technique derives from the K1 algorithm for linear systems (Sutton, 1992), which in turn is based on a diagonalized Kalman Filter. We expand upon Sutton's work in two regards: K1 is a) extended to nonlinear systems, and b) made more efficient by linearizing an exponentiation operation. The resulting ELK1 (extended, linearized K1) algorithm is computationally little more expensive than alternative proposals (Zimmermann, 1994; Almeida et al., 1999), and does not require an arbitrary smoothing parameter. On a first benchmark problem ELK1 clearly outperforms these alternatives, as well as stochastic gradient descent with momentum, even when the number of floating-point operations required per weight update is taken into account. Unlike the method of Almeida et al., ELK1 does not require statistical independence between successive training patterns, and handles large initial learning rates well. }}