Recurrent neural networks

Per Unneberg

NBIS

2024-05-23

Table of contents

• Recap

Recap

Perceptron (single neuron)

Architecture

A single neuron has \(n\) inputs \(x_i\) and an output \(y\). To each input is associated a weight \(w_i\).

Activity rule

The activity rule is given by two steps:

\[a = \sum_{i} w_ix_i, \quad i=0,...,n\]

\[\begin{array}{ccc} \mathrm{activation} & & \mathrm{activity}\\ a & \rightarrow & y(a) \end{array}\]

(MacKay, 2003)

Beware of notation here. Points to make:

• \(w_0=1\) -> bias
• activation -> activity can be separated (next slide)

Alternative view of bias: an additional weight \(w_0\) with input permanently set to 1 (MacKay, 2003, p. 471)

(Alexander Amini, 2021, p. 5:43) point out inputs \(x_i\) represent one time point

Bibliography

Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

MacKay, D. J. C. (2003). Information Theory, Inference and Learning Algorithms (Illustrated edition). Cambridge University Press.