Jack Dermody

Recurrent neural network

GRU Recurrent Neural Networks

It's possible to get a lot done with simple recurrent neural networks (SRNN).

The job of our recurrent neural network is to be able to predict the set of possible next state transitions, after having observed any number of previous states.

Gated Recurrent Units (GRU) are a variation on Long Short Term Memory (LSTM) recurrent neural networks.

Introduction to Bright Wire

For example, a recurrent neural network node is implemented with a sub graph of nodes as follows:

A recurrent neural network expects an input column of type Matrix.

Teaching a Recurrent Neural Net Binary Addition

A better solution is to use recurrent neural networks that use a simple form of memory that allow them to learn from arbitrarily long sequences, and which use their memory to change their predictions based on what they've seen previously in the sequence.

For this example we create a simple recurrent neural network (RNN) with memory size 32 and RELU activation.

Simple recurrent neural networks can be a bit fiddly - they sometimes crash and burn while learning and other times they perform brilliantly.

Sequence to Sequence with LSTM

One way the output can be varied in length against the input is with something called a sequence to sequence (STS) recurrent neural network.

Long Short Term Memory (LSTM) networks are a recurrent neural network that can be used with STS neural networks.