1 / 5
Text becomes a prediction task
The model reads a real sentence. Every word becomes an input; the following word becomes the target.
—
training sentences
—
token
—
words in the vocabulary
Show the real training text
Example
Every token predicts the one after it
Current input
—
→
Hidden state
updates
→
Training target
—
0 / 8
Read one word
Previous memory
zero
+
Current word
<start>
→
New memory
24 values
hₜ = tanh(Wₓ e(xₜ) + Wₕ hₜ₋₁ + b)
Hidden state hₜ — each bar is a value between −1 and +1
P(next)
Highest next-word probabilities
Watch training accumulate:
Backpropagation through time + AdaGrad in the browser
Training: 8,000 sentences
loss: —
Ready
Press “Generate text” to watch the model choose one word at a time.
Last choice
Model probabilities
Distance 0
Did the subject remain in memory?
Memory preserved
with subject
predicts: —
—
Memory reset after the subject
without subject
predicts: —
—
Real test
Probability of the correct pronoun as the subject recedes
Distance = the number of neutral words between the subject and its pronoun. The curve is recalculated from the RNN weights.
Explicit comparison
The trained model in the tour is a vanilla RNN; this is the LSTM cell designed to relieve its memory bottleneck.
Ready
Follow Cₜ₋₁ through one cell
Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ C̃ₜ
hₜ = oₜ ⊙ tanh(Cₜ)