TensorFlow Model Evaluation
Model evaluation measures how well a trained model performs on data it has never seen before. A model that scores 99% accuracy on training data but 60% on test data has memorized the training examples rather than learned general patterns. Proper evaluation tells you whether your model is ready for real-world use and which weaknesses need fixing before deployment.
The Student Test Analogy
Training a model is like a student studying from a textbook. Model evaluation is the final exam using different questions the student has never practiced. High training accuracy with low test accuracy means the student memorized the textbook answers rather than understanding the subject. The exam reveals the truth. TensorFlow evaluation reveals the same truth about your model.
model.evaluate()
After training, call model.evaluate() on your test set to get the loss and all metrics configured during compilation. The model never updates its weights during evaluation.
import tensorflow as tf
# Evaluate on the test set
results = model.evaluate(x_test, y_test, batch_size=32, verbose=1)
# results is a list: [loss, metric1, metric2, ...]
print(f"Test Loss: {results[0]:.4f}")
print(f"Test Accuracy: {results[1]:.4f}")
The Three Data Splits
Full Dataset
│
├── Training Set (70–80%)
│ → Model learns (weights updated) on this
│ → Used in model.fit()
│
├── Validation Set (10–15%)
│ → Monitors overfitting during training
│ → Never used to update weights
│ → Used as validation_data in model.fit()
│
└── Test Set (10–15%)
→ Final, one-time evaluation
→ Simulate real-world data the model has never seen
→ Used in model.evaluate() only after training is complete
→ Never used to make any training decisions
Cross-Validation for Small Datasets
When your dataset is small (fewer than 5,000 samples), a single train/test split might produce unreliable results depending on which samples land in each set. K-fold cross-validation solves this by training and evaluating K times on different data splits.
Diagram — 5-Fold Cross-Validation: Original Data: [Block 1][Block 2][Block 3][Block 4][Block 5] Fold 1: Train=[2,3,4,5] Test=[1] Fold 2: Train=[1,3,4,5] Test=[2] Fold 3: Train=[1,2,4,5] Test=[3] Fold 4: Train=[1,2,3,5] Test=[4] Fold 5: Train=[1,2,3,4] Test=[5] Final accuracy = average of all 5 test results This gives a much more reliable estimate than a single split.
from sklearn.model_selection import KFold
import numpy as np
kfold = KFold(n_splits=5, shuffle=True, random_state=42)
accuracies = []
for train_idx, test_idx in kfold.split(X, y):
x_train_fold, x_test_fold = X[train_idx], X[test_idx]
y_train_fold, y_test_fold = y[train_idx], y[test_idx]
model = build_model() # fresh model each fold
model.fit(x_train_fold, y_train_fold, epochs=20, verbose=0)
_, acc = model.evaluate(x_test_fold, y_test_fold, verbose=0)
accuracies.append(acc)
print(f"Mean accuracy: {np.mean(accuracies):.3f} ± {np.std(accuracies):.3f}")
Evaluation Metrics for Different Problem Types
Classification Metrics
# Confusion matrix (requires sklearn or manual calculation) from sklearn.metrics import confusion_matrix, classification_report y_pred_probs = model.predict(x_test) y_pred = np.argmax(y_pred_probs, axis=1) cm = confusion_matrix(y_test, y_pred) print(cm) # For 3-class problem: # Predicted # Cat Dog Bird # Actual Cat [45 3 2] # Dog [ 2 48 0] # Bird [ 1 0 49]
print(classification_report(y_test, y_pred,
target_names=['Cat', 'Dog', 'Bird']))
# precision recall f1-score support
# Cat 0.94 0.90 0.92 50
# Dog 0.94 0.96 0.95 50
# Bird 0.96 0.98 0.97 50
# accuracy 0.95 150
Precision, Recall, and F1 Score
Precision = TP / (TP + FP) → Of all samples predicted positive, how many actually were? → High precision = few false alarms Recall = TP / (TP + FN) → Of all actual positives, how many did the model catch? → High recall = few missed positives F1 Score = 2 × (Precision × Recall) / (Precision + Recall) → Balanced combination of precision and recall → Best when classes are imbalanced When to prioritize recall: cancer screening (missing cancer is worse than a false alarm) When to prioritize precision: spam filter (blocking a real email is worse than missing spam)
ROC-AUC for Binary Classification
from sklearn.metrics import roc_auc_score
y_pred_probs = model.predict(x_test).flatten()
auc = roc_auc_score(y_test, y_pred_probs)
print(f"AUC: {auc:.4f}")
# AUC = 0.5 → model is no better than random guessing
# AUC = 1.0 → perfect classifier
# AUC > 0.9 → excellent
# AUC 0.7–0.9 → good
Regression Metrics
import numpy as np
y_pred = model.predict(x_test).flatten()
y_true = y_test
# Mean Absolute Error — average prediction error in original units
mae = np.mean(np.abs(y_true - y_pred))
# Root Mean Squared Error — penalizes large errors more
rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
# R² (coefficient of determination) — 1.0 = perfect, 0.0 = no better than mean
ss_res = np.sum((y_true - y_pred) ** 2)
ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
r2 = 1 - (ss_res / ss_tot)
print(f"MAE: {mae:.2f}")
print(f"RMSE: {rmse:.2f}")
print(f"R²: {r2:.4f}")
Diagnosing Model Problems from Evaluation Results
Scenario 1: Underfitting Training accuracy: 65% Validation accuracy: 63% Both low and close together Fix: More layers, more neurons, more epochs, more complex model Scenario 2: Overfitting Training accuracy: 98% Validation accuracy: 72% Large gap between train and val Fix: Dropout, regularization, more data, data augmentation Scenario 3: Well-fit Training accuracy: 92% Validation accuracy: 89% Small gap, both high → Model is ready Scenario 4: Data leakage Training accuracy: 99.9% Validation accuracy: 99.8% Both suspiciously perfect → Check if test data accidentally leaked into training
Evaluating on a tf.data.Dataset
# model.evaluate() works directly with tf.data.Dataset test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32) results = model.evaluate(test_dataset)
Generating Predictions
# Get raw probability scores for all classes y_pred_probs = model.predict(x_test) # shape: (n_samples, n_classes) # Convert to class indices (for multi-class) y_pred_classes = np.argmax(y_pred_probs, axis=1) # Convert to binary 0/1 (for binary classification) y_pred_binary = (y_pred_probs.flatten() > 0.5).astype(int)
Thorough evaluation catches problems before they reach production users. Running only accuracy on a balanced dataset can hide severe failures on specific classes or edge cases. Always check confusion matrices, per-class metrics, and behavior on difficult examples before declaring a model production-ready. The next topic covers saving and loading models — how to preserve a trained model and reuse it without retraining.