TensorFlow Model Compilation
Model compilation is the step that configures your neural network for training. Before a model can learn anything, you must tell TensorFlow three things: what to measure (loss function), how to improve (optimizer), and what to report (metrics). The model.compile() call bundles all three together. Getting compilation right means the model converges efficiently. Getting it wrong means the model trains slowly, diverges, or produces meaningless results.
The Race Car Analogy
Building a neural network is like assembling a race car. Compilation is fitting the engine (optimizer), installing the fuel gauge (loss function), and attaching the speedometer and lap counter (metrics). The car cannot race without these components properly connected. model.compile() is the moment you connect everything and declare the car race-ready.
The compile() Method
import tensorflow as tf
model.compile(
optimizer='adam', # How to update weights
loss='sparse_categorical_crossentropy', # What to minimize
metrics=['accuracy'] # What to report
)
The Three Components in Detail
Component 1 — Loss Function
The loss function calculates a single number that represents how wrong the model's predictions are. The optimizer works to make this number as small as possible during training.
For Regression (predicting numbers)
# Mean Squared Error — penalizes large errors heavily (squares the error) model.compile(loss='mean_squared_error') model.compile(loss='mse') # shorthand # Mean Absolute Error — penalizes all errors equally (more robust to outliers) model.compile(loss='mean_absolute_error') model.compile(loss='mae') # Huber Loss — MSE for small errors, MAE for large errors (best of both) model.compile(loss=tf.keras.losses.Huber(delta=1.0))
For Binary Classification (yes/no)
# Binary Crossentropy — use when output layer has sigmoid activation model.compile(loss='binary_crossentropy')
For Multi-Class Classification (one of many categories)
# Sparse Categorical Crossentropy — labels are integers (0, 1, 2, ...) model.compile(loss='sparse_categorical_crossentropy') # Categorical Crossentropy — labels are one-hot encoded ([0,1,0], [1,0,0]) model.compile(loss='categorical_crossentropy')
Diagram — Which loss to use:
What does your output layer look like?
│
├── Dense(1, activation='sigmoid') ──► binary_crossentropy
│ (predicts 0 or 1)
│
├── Dense(N, activation='softmax') ──► sparse_categorical_crossentropy
│ (integer labels: 0, 1, 2...) OR categorical_crossentropy
│ (one-hot labels)
│
└── Dense(1) or Dense(N) ──► mse / mae / huber
(no activation — regression)
Component 2 — Optimizer
The optimizer uses the gradient (direction of steepest loss increase) to update model weights, moving them in the direction that reduces loss.
Adam — Adaptive Moment Estimation
# The most popular optimizer — adapts the learning rate per parameter model.compile(optimizer='adam') # Customize the learning rate model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005))
SGD — Stochastic Gradient Descent
# Simpler update rule — good when you understand your learning rate well
model.compile(optimizer=tf.keras.optimizers.SGD(
learning_rate=0.01,
momentum=0.9, # Carry momentum from previous steps
nesterov=True # Look-ahead momentum variant (often better)
))
RMSprop
# Adapts learning rates — works well for RNNs model.compile(optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.001))
Choosing an Optimizer
Task Recommended Optimizer ────────────────────────────────────────────────────── General purpose Adam (start here always) Image classification Adam or SGD with momentum RNN / LSTM / GRU Adam or RMSprop Fine-tuning (small updates) SGD with small lr (0.0001) Research/custom experiments AdamW or Lion ──────────────────────────────────────────────────────
Component 3 — Metrics
Metrics measure model performance but do not affect training. They provide human-readable feedback after each epoch.
# Accuracy — percentage of correct predictions
model.compile(metrics=['accuracy'])
# Multiple metrics at once
model.compile(metrics=['accuracy', 'mse'])
# For binary classification — precision and recall
model.compile(metrics=[
tf.keras.metrics.Precision(name='precision'),
tf.keras.metrics.Recall(name='recall'),
tf.keras.metrics.AUC(name='auc')
])
# For regression — mean absolute error
model.compile(metrics=['mae'])
Learning Rate: The Most Critical Hyperparameter
Learning Rate Too High: → Weights overshoot the optimal value → Loss oscillates or diverges (goes up instead of down) → May produce NaN values Loss curve: ╱╲╱╲╱╲╱ (unstable) Learning Rate Too Low: → Weights update in tiny steps → Training takes forever → Risk getting stuck in a poor local minimum Loss curve: ────────────────── (barely moves) Learning Rate Just Right: → Steady, consistent decrease in loss → Convergence within a reasonable number of epochs Loss curve: ╲──────────╲────── (smooth decrease)
# Common starting learning rates by optimizer: Adam: 0.001 (default) SGD: 0.01 RMSprop: 0.001 # How to search for the right learning rate # Start high (0.1) and decrease by 10× until loss stops diverging # The ideal LR is just below where divergence starts
Using Loss Objects vs. String Names
# String shorthand (convenient)
model.compile(loss='mse', optimizer='adam', metrics=['mae'])
# Loss objects (allow extra configuration)
model.compile(
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
metrics=[tf.keras.metrics.SparseCategoricalAccuracy()]
)
The from_logits=True flag tells TensorFlow that your output layer has no softmax activation — the raw logit values go directly to the loss function. This is numerically more stable than applying softmax first and is the recommended approach when using categorical crossentropy.
Compiling for Different Problem Types
# Binary classification (spam or not spam)
model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy', 'AUC']
)
# Multi-class classification (10 digit classes)
model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
# Regression (predict price)
model.compile(
optimizer=tf.keras.optimizers.Adam(0.001),
loss='mse',
metrics=['mae']
)
# Multi-label classification (a photo can have multiple tags)
model.compile(
optimizer='adam',
loss='binary_crossentropy', # Independent sigmoid per label
metrics=['accuracy']
)
Recompiling a Model
You can call compile() again at any time to change the optimizer or loss function. This is useful during transfer learning where you use a high learning rate for initial training then recompile with a lower rate for fine-tuning.
# Phase 1: Train with higher learning rate
model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(train_data, epochs=10)
# Phase 2: Fine-tune with lower learning rate
model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(train_data, epochs=5)
With the model compiled, you have everything in place to start training. The next topic covers model training in full detail — how to feed data, track progress, prevent overfitting, and interpret the results you see during each epoch.