Computer Vision Object Tracking

Object tracking follows an object from one video frame to another and keeps its identity. Detection answers the question of what appears in a single image. Tracking answers the question of where each item goes over time. Picture a name tag stuck on every person in a crowd. The tag stays with the person from the first frame to the last.

Detection Versus Tracking

Detection only:           Tracking:
Frame 1: [car] [car]      Frame 1: [car #1] [car #2]
Frame 2: [car] [car]      Frame 2: [car #1] [car #2]
Frame 3: [car] [car]      Frame 3: [car #1] [car #2]
(no names, no paths)      (stable IDs, full paths)

IDs unlock useful numbers: how long a person waited, which direction a car drove, and how many unique visitors crossed a line.

Two Main Approaches

Single Object Trackers

The user draws a box around one target in the first frame. The tracker learns the target appearance and searches for it in each new frame. Classic choices include KCF and CSRT. These trackers run fast and suit simple scenes with one subject.

Tracking by Detection

A detector finds all objects in every frame. A tracking algorithm links new boxes to existing IDs. Popular methods include SORT, DeepSORT, and ByteTrack. This approach handles many objects and new arrivals.

The Predict, Match, Update Loop

        +-----------------------------------+
        v                                   |
[ Predict where each track moves ]          |
        |                                   |
        v                                   |
[ Match predictions to new detections ]     |
        |                                   |
        v                                   |
[ Update tracks, start new, retire old ] ---+

Predict

A Kalman filter estimates the upcoming position from the speed and direction seen so far. A car moving right at ten pixels per frame will likely appear ten pixels further right in the following frame.

Match

The algorithm measures overlap between each predicted box and each detected box using IoU, short for intersection over union. The Hungarian algorithm pairs boxes so the total overlap becomes as large as possible.

              Det A   Det B   Det C
Track 1       0.82    0.05    0.00
Track 2       0.03    0.74    0.10
Track 3       0.00    0.08    0.00   <- no match: track may be lost

Update

Matched tracks absorb the new box and refine their speed estimate. Unmatched detections spawn new tracks. Tracks without matches for several frames get retired.

Simple Tracker in Code

import cv2

cap = cv2.VideoCapture("traffic.mp4")
ok, frame = cap.read()

box = cv2.selectROI("Pick target", frame)
tracker = cv2.TrackerCSRT_create()   # older builds need opencv-contrib-python
tracker.init(frame, box)

while True:
    ok, frame = cap.read()
    if not ok:
        break
    found, box = tracker.update(frame)
    if found:
        x, y, w, h = map(int, box)
        cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
    cv2.imshow("Tracking", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

Common Challenges

• Occlusion: a bus hides a car for several frames
• ID switches: two people cross paths and swap tags
• Appearance change: lighting, rotation, or scale shifts confuse the model
• Speed: real-time video demands about 25 frames per second

Appearance Features Reduce Confusion

DeepSORT adds a small neural network that creates a fingerprint of each object from its pixels. When two boxes overlap heavily, the fingerprint tells the tracker which box carries which identity.

Measuring Tracker Quality

• MOTA combines misses, false alarms, and ID switches into one score
• IDF1 measures how consistently the tracker keeps identities
• FPS shows whether the system keeps up with live video

Applications

Sports analysts follow players to study movement patterns. Retail teams measure queue lengths. Wildlife researchers count animals passing a trail camera. Robots follow a moving target while avoiding obstacles. Each use pairs a reliable detector with a tracker tuned for the scene.

Practice Exercise

Record a short clip of a ball rolling across a table. Select the ball with selectROI, run the CSRT tracker, and draw the path by storing the box center in every frame. Repeat the experiment with a second ball that crosses the first. Watch whether the single tracker keeps following the right ball. The test shows why multi-object methods rely on motion prediction and appearance features.

Choosing a Method

Situation                          Suggested Method
---------------------------------  --------------------
One target, simple scene           CSRT or KCF
Many objects, speed matters        SORT or ByteTrack
Crowded scene, crossing paths      DeepSORT

Tracking Tips for Beginners

• Start with a clear video where objects stay visible and lighting stays steady
• Keep the detector confidence threshold moderate so weak boxes do not spawn false tracks
• Let a track survive a few missed frames to bridge short occlusions
• Draw each ID and a short trail behind every object to inspect behavior by eye