TensorFlow GRU Layer

The Gated Recurrent Unit (GRU) is a streamlined alternative to LSTM. It combines the forget and input gates into a single update gate and merges the cell state and hidden state into one vector. GRU has fewer parameters than LSTM, trains faster, and achieves comparable accuracy on most tasks. It is the preferred choice when training speed matters or when the dataset is small enough that simpler models generalize better.

GRU vs LSTM Architecture

LSTM has three gates and two states:
  Gates: Forget gate, Input gate, Output gate
  States: Cell state (c_t) + Hidden state (h_t)
  Parameters per unit: ~4 × units²

GRU has two gates and one state:
  Gates: Reset gate (r_t), Update gate (z_t)
  State: Hidden state (h_t) only — no separate cell state
  Parameters per unit: ~3 × units²

For 128 units:
  LSTM: ~66,000 parameters in the recurrent layer
  GRU:  ~49,000 parameters — 26% fewer

GRU Mechanism

At each time step t:

UPDATE GATE — How much of the past to keep vs new info
  z_t = sigmoid(W_z × [h_{t-1}, x_t] + b_z)
  z_t near 1: keep mostly the old hidden state
  z_t near 0: mostly update with new information

RESET GATE — How much past to use when computing new candidate
  r_t = sigmoid(W_r × [h_{t-1}, x_t] + b_r)
  r_t near 0: ignore past hidden state for this step
  r_t near 1: include past hidden state fully

NEW CANDIDATE:
  h̃_t = tanh(W_h × [r_t × h_{t-1}, x_t] + b_h)

FINAL HIDDEN STATE UPDATE:
  h_t = (1 - z_t) × h_{t-1}  +  z_t × h̃_t
         ↑                         ↑
    (keep old)                (mix in new)

GRU in TensorFlow

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(input_dim=10000, output_dim=128, input_length=200),
    tf.keras.layers.GRU(units=256, dropout=0.2, recurrent_dropout=0.1),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()

Stacked Bidirectional GRU

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(10000, 128, input_length=200),

    tf.keras.layers.Bidirectional(
        tf.keras.layers.GRU(128, return_sequences=True, dropout=0.2)
    ),
    # Output: (batch, 200, 256)

    tf.keras.layers.Bidirectional(
        tf.keras.layers.GRU(64)
    ),
    # Output: (batch, 128)

    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(5, activation='softmax')
])

GRU for Time Series

# Predict next day's stock price from 30 days of history
import numpy as np
import tensorflow as tf

# Data shape: (samples, timesteps, features)
# e.g., 1000 windows of 30 days, each day has 5 features
x_train = np.random.random((1000, 30, 5)).astype('float32')
y_train = np.random.random((1000, 1)).astype('float32')   # next-day price

model = tf.keras.Sequential([
    tf.keras.layers.GRU(128, return_sequences=True, input_shape=(30, 5)),
    tf.keras.layers.GRU(64),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1)  # Linear output for regression
])

model.compile(optimizer='adam', loss='mse', metrics=['mae'])
model.fit(x_train, y_train, epochs=20, batch_size=32, validation_split=0.1)

When to Choose GRU Over LSTM

Choose GRU when:                     Choose LSTM when:
──────────────────────────────────────────────────────────
Training speed is a priority         Maximum accuracy is priority
Dataset is small (<10K samples)      Large datasets available
Sequence length is moderate          Very long sequences (>500 steps)
Real-time / mobile inference         Offline batch processing
Hyperparameter tuning budget is low  Fine-grained memory control needed
──────────────────────────────────────────────────────────

CuDNN-Optimized GRU

TensorFlow includes a GPU-optimized GRU implementation that runs 5–10× faster on NVIDIA GPUs. It activates automatically when you use the default GRU parameters and have a GPU available. Avoid setting recurrent_dropout if you want CuDNN acceleration — it disables the optimization.

# CuDNN-accelerated (fast on GPU):
tf.keras.layers.GRU(256)                        # Default params — uses CuDNN

# CuDNN disabled (slower on GPU):
tf.keras.layers.GRU(256, recurrent_dropout=0.1) # recurrent_dropout disables CuDNN

GRU Parameter Count Example

GRU(128, input_shape=(200, 64)):
  Parameters = 3 × units × (units + input_features + 1)
             = 3 × 128 × (128 + 64 + 1)
             = 3 × 128 × 193
             = 74,112 parameters

Equivalent LSTM(128):
             = 4 × 128 × (128 + 64 + 1)
             = 4 × 128 × 193
             = 98,816 parameters

GRU saves 24,704 parameters — 25% fewer — with minimal accuracy loss.

GRU is one of the two dominant recurrent architectures in modern deep learning. Its simplicity and speed make it the first choice for new sequence modeling projects, with LSTM as the fallback when maximum accuracy is required. The next topic covers text sequences — how to convert raw text into numerical tensors that RNN and LSTM layers can process.