CV Model Deployment

A model that works in a notebook helps nobody until real users can reach it. Deployment moves a trained model into an app, a server, a camera, or a phone. The target device shapes every decision, from model size to file format.

The Deployment Journey

Train  ->  Export  ->  Optimize  ->  Package  ->  Serve  ->  Monitor
 |           |            |            |           |           |
 PyTorch    ONNX or      shrink and   container   API or      watch
 weights    TFLite       speed up     or app      on-device   accuracy

Where Models Run

Target        Strength                        Challenge
------------  ------------------------------  ----------------------
Cloud server  powerful hardware, easy updates network delay, running cost
Edge device   fast response, data stays local limited power and memory
Phone         works offline, personal data    battery, app size
Browser       no install needed               smaller, slower models

A factory camera that must react in milliseconds suits an edge device. A photo-editing website with heavy models suits a cloud server.

Exporting to a Portable Format

Training frameworks save models in their own formats. Portable formats let other tools run the model without the original framework.

  • ONNX: a widely supported exchange format for many runtimes
  • TensorFlow Lite: a compact format for phones and small boards
  • Core ML: the format for Apple devices
  • TensorRT: a runtime that speeds up models on NVIDIA hardware
import torch

model.eval()
dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, dummy, "model.onnx",
                  input_names=["image"], output_names=["scores"],
                  dynamic_axes={"image": {0: "batch"}})

Running the Exported Model

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("model.onnx")
batch = np.random.rand(1, 3, 224, 224).astype(np.float32)
scores = session.run(None, {"image": batch})[0]
print(scores.argmax())

The runtime needs no training code, so the deployment package stays small.

Making Models Smaller and Faster

Quantization

Quantization stores weights as 8-bit integers in place of 32-bit decimals. The file shrinks to about a quarter of its size, and speed often improves a lot. Accuracy usually drops by a small amount that testing can confirm.

32-bit weight:  0.73618492...   (4 bytes)
8-bit weight:   187             (1 byte)

Pruning

Pruning removes weights that contribute little. Think of trimming dead branches from a tree.

Knowledge Distillation

A large teacher model trains a small student model to imitate its outputs. The student keeps much of the teacher accuracy at a fraction of the size.

Measuring What Matters

  • Latency: time to process one image, measured in milliseconds
  • Throughput: images processed per second
  • Memory: how much RAM the model needs during use
  • Size: the file the user downloads
  • Accuracy: results on a held-out test set after every optimization
import time

start = time.perf_counter()
for _ in range(100):
    session.run(None, {"image": batch})
ms = (time.perf_counter() - start) * 10
print("Average latency:", round(ms, 1), "ms")

Run a few warm-up passes before timing, because the first calls load caches and run slower.

Serving Through an API

A web service accepts an image and returns predictions as text. Frameworks such as FastAPI wrap the model in a few lines. Package the service in a container so it runs identically on every server. Add input checks that reject oversized files and unsupported formats.

Monitoring After Launch

Real-world images drift away from training images over time. A new camera model, a change in lighting, or a seasonal shift can lower accuracy quietly. Track confidence scores, log a sample of inputs, and review wrong predictions regularly. Retrain when performance slips below an agreed limit.

Privacy and Safety in Production

  • Process images on the device whenever the task allows
  • Encrypt images in transit and in storage
  • Delete uploads once processing ends unless users agree to keep them
  • Provide a fallback or human review path when confidence is low

Deployment Checklist

[ ] Target device and latency goal defined
[ ] Model exported and verified against original outputs
[ ] Quantized model tested on the full test set
[ ] Input validation and error handling in place
[ ] Monitoring and retraining plan written

Practice Exercise

Export a pretrained ResNet-18 to ONNX and run it with ONNX Runtime. Compare the predictions with the original PyTorch model on ten images. Time both versions and record the speed difference.

Choosing the Right Target

Begin with the user experience. Ask how fast answers must arrive, whether photos may leave the device, and how often the model will update. Cloud hosting simplifies updates and supports large models. Local processing protects privacy and works without a network connection. Many products combine both: a small model on the device filters easy cases, and a larger cloud model handles hard ones.

Camera -> Small on-device model -> confident? -> answer now
                                  |
                                  not sure -> send to cloud model
Previous lesson
Back to course