TensorFlow Fine-Tuning Layers
Fine-tuning selectively unfreezes layers in a pre-trained model and trains them on your dataset at a very low learning rate. The earlier layers — which detect edges, colors, and basic textures — are generally universal and stay frozen. The later layers — which detect domain-specific features like specific flower petal shapes or car model details — benefit from updating their weights on your data. Fine-tuning bridges the gap between pure transfer learning and training from scratch.
Two-Phase Transfer Learning Strategy
Phase 1: Feature Extraction (frozen base) ─ All base model layers frozen (trainable=False) ─ Only your custom classifier head trains ─ Use standard learning rate: 0.001 ─ Train for 10–20 epochs ─ Goal: get the classifier head to a good starting point Phase 2: Fine-Tuning (partially unfrozen base) ─ Unfreeze top N layers of the base model ─ Train the whole model at a much lower learning rate: 0.00001 ─ Train for 5–15 more epochs ─ Goal: adapt domain-specific features to your data
Which Layers to Unfreeze
Pre-trained model layers (early → late): Layer 1–20: Edges, color gradients ← Always keep FROZEN Layer 21–50: Textures, simple patterns ← Usually keep FROZEN Layer 51–80: Object parts, complex features ← Fine-tune these Layer 81+: High-level object understanding ← Fine-tune these Rule of thumb: Unfreeze the top 20–40% of layers - The more different your data from ImageNet, the more layers to unfreeze - The smaller your dataset, the fewer layers to unfreeze
Complete Fine-Tuning Code
import tensorflow as tf
# ── Phase 1: Build model with frozen base ─────────────────────────
base = tf.keras.applications.MobileNetV2(
input_shape=(224, 224, 3),
include_top=False,
weights='imagenet'
)
base.trainable = False # Freeze everything
# Build the full model
inputs = tf.keras.Input(shape=(224, 224, 3))
x = tf.keras.applications.mobilenet_v2.preprocess_input(inputs)
x = base(x, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dense(256, activation='relu')(x)
x = tf.keras.layers.Dropout(0.3)(x)
outputs = tf.keras.layers.Dense(5, activation='softmax')(x)
model = tf.keras.Model(inputs, outputs)
# Compile Phase 1
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
print(f"Trainable layers: {len(model.trainable_variables)}")
# Train Phase 1
history1 = model.fit(train_ds, epochs=15, validation_data=val_ds)
# ── Phase 2: Unfreeze top layers and fine-tune ────────────────────
base.trainable = True
total_layers = len(base.layers)
fine_tune_from = total_layers - 30 # Unfreeze last 30 layers
print(f"Total base layers: {total_layers}")
print(f"Unfreezing from layer: {fine_tune_from}")
# Freeze the early layers, unfreeze the rest
for layer in base.layers[:fine_tune_from]:
layer.trainable = False
for layer in base.layers[fine_tune_from:]:
layer.trainable = True
# Recompile with a much lower learning rate
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=1e-5), # 100× smaller
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
print(f"Trainable layers after unfreezing: {len(model.trainable_variables)}")
# Train Phase 2
history2 = model.fit(
train_ds,
epochs=10,
validation_data=val_ds,
callbacks=[
tf.keras.callbacks.EarlyStopping(
monitor='val_accuracy', patience=4, restore_best_weights=True
)
]
)
Why the Learning Rate Must Be Low
High learning rate during fine-tuning: Pre-trained weights have carefully calibrated values. A large update destroys this structure in a few steps. Result: accuracy crashes back to random-guess level. Low learning rate (1e-5 or smaller): Each update is a tiny nudge to the weights. The model retains its ImageNet knowledge. It makes small, precise adjustments for your domain. Result: accuracy improves steadily above Phase 1 levels.
Diagnosing Fine-Tuning Results
Good fine-tuning:
Phase 1 best val_acc: 85%
Phase 2 best val_acc: 91% ← Improved by 6%
Bad fine-tuning (too high learning rate):
Phase 1 best val_acc: 85%
Phase 2 val_acc: 55% ← Dropped (weights destroyed)
Fix: reduce learning rate by 10×
Negligible improvement from fine-tuning:
Phase 1 best val_acc: 85%
Phase 2 best val_acc: 85.5% ← Barely improved
Causes: dataset too different from ImageNet,
not enough data to update reliably,
base features already optimal for your task
BatchNormalization Behavior During Fine-Tuning
# BatchNormalization layers behave differently during training vs inference # During fine-tuning, always pass training=False to the base model # This uses running statistics computed during ImageNet training # rather than small-batch statistics from your dataset x = base(preprocessed_inputs, training=False) # ← Important! # If you use training=True on a small dataset, the batch statistics # will be noisy and overwrite the carefully calibrated running stats.
Comparing Fine-Tuning Depths
Flowers Dataset (3,670 images, 5 classes): Strategy Val Accuracy ────────────────────────────────────────────────────── Train from scratch ~60% Feature extraction only ~82% Fine-tune last 10 layers ~86% Fine-tune last 30 layers ~91% Fine-tune last 50 layers ~90% ← slight overfit with small dataset Fine-tune entire model ~85% ← more overfitting, less improvement ────────────────────────────────────────────────────── Optimal: fine-tune a moderate number of late layers
Saving the Fine-Tuned Model
# Save the complete model (base + custom head, fine-tuned weights)
model.save('finetuned_mobilenetv2_flowers.keras')
# Load and use
loaded = tf.keras.models.load_model('finetuned_mobilenetv2_flowers.keras')
predictions = loaded.predict(test_ds)
Fine-tuning is the technique that makes transfer learning reach its full potential. Phase 1 trains your custom head to understand your classes using fixed features. Phase 2 refines those features so they are optimally suited for your data. Together, the two phases routinely achieve 5–10% accuracy improvements over feature extraction alone. The next topic applies fine-tuning to MobileNet in a practical end-to-end project.