Learning Long Short-Term Memory Networks

LSTM as a recurrent neural network

Long short-term memory (LSTM) is a recurrent neural network (RNN) architecture designed to address the vanishing gradient problem that makes traditional RNNs difficult to train on long sequences. By using a memory cell that can preserve information over many timesteps, LSTMs enable learning of long-term dependencies with relatively less sensitivity to the length of gaps in the input sequence. Although LSTMs mitigate vanishing gradients, they can still experience exploding gradients. An LSTM unit typically contains a cell state plus three interacting gates: an input gate, a forget gate, and an output gate. The forget gate decides what portion of the previous cell state to retain (values near 1 keep information; values near 0 discard it). The input gate controls which new candidate information is written into the cell state, and the output gate determines which parts of the updated cell state are exposed as the hidden/output state. In the “peephole” variant, gates can also directly access the cell state (via peephole connections), improving the gate’s ability to regulate information flow. LSTM variants can also incorporate convolution (peephole convolutional LSTM) for spatial/structured inputs. During training, LSTMs are commonly optimized with backpropagation through time in supervised settings, and their “error carousel” helps keep gradients flowing through the cell state. For sequence labeling tasks where alignment between inputs and outputs is unknown, LSTMs are often trained with connectionist temporal classification (CTC), which jointly learns recognition and alignment. LSTMs have been widely applied to tasks such as speech recognition, machine translation, time series analysis, and other sequence-based prediction and classification problems.

0/2 modes complete

Vanishing gradient problem in classic RNNs

0/2 modes complete