TensorFlow Transfer Learning
Transfer learning lets you take a model that was already trained on a large dataset and reuse it for a new, related problem. Instead of training from scratch — which requires millions of images and weeks of compute time — you borrow knowledge from an existing model and adapt it to your specific task. This technique makes state-of-the-art accuracy achievable for most developers, even without large datasets or powerful hardware.
The Expert Analogy
Imagine you need a mobile app security review. You hire a cybersecurity expert who spent ten years mastering computer science, networking, and cryptography. You do not ask them to restart their education from scratch — you leverage the knowledge they already have and give them a few weeks to learn the specific quirks of mobile platforms. Transfer learning works exactly the same way.
Why Start From Scratch Is Hard
Training from Scratch: ──────────────────────────────────────────────────── Data needed: Millions of labeled images Training time: Days to weeks on expensive GPUs Cost: Hundreds to thousands of dollars Risk of failure: High without deep expertise ──────────────────────────────────────────────────── Transfer Learning: ──────────────────────────────────────────────────── Data needed: Hundreds to thousands of images Training time: Minutes to hours on a laptop Cost: Minimal or free (Google Colab) Risk of failure: Low — you build on proven work ────────────────────────────────────────────────────
What a Pre-trained Model Has Already Learned
Popular models like VGG16, ResNet50, and MobileNetV2 were trained on ImageNet — a dataset with 1.2 million images across 1,000 categories including animals, vehicles, food, and objects. Through this training, their early layers learned to detect universal visual features:
Layer Depth What It Detected (from ImageNet training) ────────────────────────────────────────────────────────── Early layers → Edges, color gradients, basic lines Middle layers → Corners, textures, repeating patterns Deep layers → Object parts (eyes, wheels, leaves) Final layers → Complete objects specific to ImageNet ──────────────────────────────────────────────────────────
The early and middle layers learn features that are useful for almost any image task — not just ImageNet categories. This is the knowledge you "transfer" to your new task.
The Two-Stage Transfer Learning Process
Stage 1: Feature Extraction
You take the pre-trained model, freeze all its layers (so their weights do not change), and add a few new layers at the end. Only the new layers train. The frozen layers act as a powerful feature extractor that transforms your input images into rich feature vectors.
Pre-trained Base (FROZEN)
┌───────────────────────────────┐
│ Conv Block 1 — edges │ ← Weights LOCKED
│ Conv Block 2 — textures │ ← Weights LOCKED
│ Conv Block 3 — shapes │ ← Weights LOCKED
│ Conv Block 4 — object parts │ ← Weights LOCKED
│ Global Average Pooling │ ← Weights LOCKED
└───────────────────────────────┘
│
▼
New Classifier Head (TRAINABLE)
┌───────────────────────────────┐
│ Dense(256, activation='relu') │ ← Learns from your data
│ Dense(N, activation='softmax')│ ← Outputs your N classes
└───────────────────────────────┘
Stage 2: Fine-Tuning (Optional)
After the classifier head has learned, you unfreeze some of the deeper layers of the base model and train the entire network at a very low learning rate. This lets the model make small adjustments to the pre-trained features to better fit your specific data.
Building a Transfer Learning Model in TensorFlow
import tensorflow as tf
# Step 1: Load pre-trained base model (without its top classifier)
base_model = tf.keras.applications.MobileNetV2(
input_shape=(224, 224, 3),
include_top=False, # Remove the final classification layer
weights='imagenet' # Use weights trained on ImageNet
)
# Step 2: Freeze the base model
base_model.trainable = False
# Step 3: Add your custom classifier head
inputs = tf.keras.Input(shape=(224, 224, 3))
x = base_model(inputs, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dense(256, activation='relu')(x)
x = tf.keras.layers.Dropout(0.3)(x)
outputs = tf.keras.layers.Dense(5, activation='softmax')(x) # 5 custom classes
model = tf.keras.Model(inputs, outputs)
# Step 4: Compile and train
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
What include_top=False Does
Every pre-trained model ends with a Dense layer that outputs scores for 1,000 ImageNet categories. You remove this layer because your task has a different number of classes. Setting include_top=False gives you the base model without that final layer, so you can attach your own classifier designed for your specific categories.
GlobalAveragePooling2D vs. Flatten
After the base model, you need to convert the 3D feature maps into a 1D vector before passing them to Dense layers. Two options exist:
GlobalAveragePooling2D Flatten
───────────────────────────────────────────────────
Output size Small (e.g., 1280) Large (e.g., 100,352)
Parameters Fewer Many more
Overfitting risk Lower Higher
Speed Faster Slower
Recommendation Preferred for transfer Avoid for large inputs
GlobalAveragePooling2D averages all spatial locations in each feature map, reducing a 7×7×1280 feature map to a single vector of 1280 values. This removes most spatial information but keeps the essence of what each filter detected anywhere in the image.
Fine-Tuning: Going Deeper
import tensorflow as tf
# After initial training of the classifier head...
# Unfreeze the top 30 layers of the base model
base_model.trainable = True
for layer in base_model.layers[:-30]:
layer.trainable = False
# Recompile with a much lower learning rate
model.compile(
optimizer=tf.keras.optimizers.Adam(learning_rate=1e-5), # 100x smaller
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
# Continue training
model.fit(train_dataset, epochs=10, validation_data=val_dataset)
A low learning rate during fine-tuning is critical. The pre-trained weights are already good — you want to make small, careful adjustments, not large ones that destroy the learned knowledge.
Popular Pre-trained Models Available in Keras
- MobileNetV2 — fast and lightweight; ideal for mobile and embedded devices
- ResNet50 — deep residual network; excellent accuracy for medium-sized datasets
- EfficientNetB0–B7 — highly efficient; best accuracy per parameter
- VGG16 / VGG19 — older but simple architecture; easy to understand and modify
- InceptionV3 — Google's architecture; very good for fine-grained classification
- Xception — improves on Inception with depthwise separable convolutions
Practical Example: Classifying Dog Breeds
You want to classify 10 dog breeds using 500 photos per breed (5,000 total). Training a CNN from scratch with 5,000 images almost certainly leads to overfitting — the model memorizes the training photos but fails on new ones. Transfer learning solves this:
- Load MobileNetV2 trained on ImageNet (which includes dog photos)
- Freeze the base model — it already knows what dogs look like
- Add a Dense layer with 10 outputs (one per breed)
- Train only the Dense layer on your 5,000 photos
- Optionally fine-tune the top layers of MobileNetV2 for breed-specific features
With this approach, you achieve 85–90% accuracy on just 5,000 images — a result that would typically require 500,000 images when training from scratch.
When to Use Transfer Learning
- Your dataset has fewer than 10,000 images per class
- Your task involves images similar to everyday objects (photos of real-world scenes)
- You need high accuracy quickly without large computing resources
- You are building a proof-of-concept before investing in a custom architecture
Transfer learning democratizes machine learning by putting high-quality models within reach of anyone with a laptop and a few hundred images. The next topics show you how to use specific pre-trained models, control which layers get frozen, and push accuracy higher through systematic fine-tuning.