Deep Learning Gated Recurrent Units

A Gated Recurrent Unit, or GRU, is a memory cell for sequence data such as sentences, sensor readings, and stock-style time series. It solves the same problem as the LSTM but uses a simpler design with fewer moving parts. Fewer parts mean faster training and lower memory use, while accuracy often stays close to an LSTM.

The Memory Problem in Sequences

A plain recurrent network forgets early information as a sequence grows longer. Gradients shrink while travelling back through many steps, so the network struggles to connect a word at the start of a paragraph with a word at the end. Gates fix this by deciding what to keep and what to discard at every step.

The Notebook Analogy

Imagine a student who takes notes during a long lecture. At every new sentence, the student asks two questions. Should the old notes change at all? Is any old note now irrelevant? A GRU asks the same questions with two gates.

Old notes (hidden state) + New sentence (input)
        |
   Two gates decide:
     Update gate: how much of the old notes to keep
     Reset gate:  how much of the old notes to consult
        |
   Updated notes (new hidden state)

The Two Gates

Update Gate

The update gate balances old memory against new information. A value near 1 keeps the old memory almost unchanged. A value near 0 replaces it with fresh content.

Reset Gate

The reset gate controls how much past information feeds into the new candidate memory. A value near 0 makes the cell ignore history and focus on the current input.

One Step Inside a GRU

Inputs: previous memory h(t-1), current input x(t)

1. Update gate  z = sigmoid(weights applied to x(t) and h(t-1))
2. Reset gate   r = sigmoid(weights applied to x(t) and h(t-1))
3. Candidate    c = tanh(weights applied to x(t) and r x h(t-1))
4. New memory   h(t) = (1 - z) x h(t-1) + z x c

Sigmoid produces values between 0 and 1, which act like dimmer switches. Tanh produces the candidate memory between -1 and 1.

Example: Reading a Sentence

Sentence: "The keys to the cabinet are on the table."

Word: keys     -> store plural subject in memory
Word: cabinet  -> keep the subject, ignore the new noun
Word: are      -> use stored plural to pick "are", not "is"

The update gate keeps the plural subject alive across several words. A simple recurrent network would often lose it.

GRU Versus LSTM

Feature	GRU	LSTM
Gates	2 (update, reset)	3 (input, forget, output)
Memory states	1 hidden state	Hidden state plus cell state
Parameters	Fewer	More
Training speed	Faster	Slower
Long sequences	Strong	Slightly stronger on very long tasks

Building a GRU

model = keras.Sequential([
    layers.Embedding(10000, 64),
    layers.GRU(64),
    layers.Dense(1, activation="sigmoid")
])

This model reads a sequence of word codes, summarizes it with a GRU layer, and outputs one probability, such as the chance that a review is positive.

When to Choose a GRU

• Small or medium datasets where fewer parameters reduce overfitting
• Projects with limited computing power
• Quick experiments that compare sequence architectures
• Time series forecasting and speech tasks with moderate length

Practical Tips

• Pad or trim sequences to equal length inside each batch
• Stack two GRU layers for harder tasks
• Add dropout between layers to control overfitting
• Try both GRU and LSTM and keep the winner on validation data

Key Terms

• GRU - a gated memory cell with two gates
• Hidden state - the memory passed from step to step
• Update gate - decides how much memory to keep
• Reset gate - decides how much history to use
• Sequence - ordered data such as words or time readings