Computer Vision Evaluation Metrics

A model that looks great in a demo can fail in the real world. Metrics turn quality into numbers, so teams can compare models, spot weaknesses, and track progress. Each vision task needs its own set of measurements.

The Confusion Matrix

Think of a smoke alarm. The alarm can ring when a fire exists, stay silent when no fire exists, ring without a fire, or stay silent during a fire. Four outcomes cover every case.

                     Reality: Cat      Reality: No Cat
Model says Cat       TP (true hit)     FP (false alarm)
Model says No Cat    FN (missed cat)   TN (correct pass)

A Worked Example

A cat detector checks 100 photos. Twenty photos contain a cat. The model finds 15 of them and raises 10 false alarms.

TP = 15    FN = 5    FP = 10    TN = 70

Accuracy, Precision, Recall, and F1

Accuracy  = (TP + TN) / total        = 85 / 100 = 0.85
Precision = TP / (TP + FP)           = 15 / 25  = 0.60
Recall    = TP / (TP + FN)           = 15 / 20  = 0.75
F1        = 2 x P x R / (P + R)      = 0.67
  • Accuracy shows the share of all correct answers
  • Precision answers: when the model says cat, how often is it right?
  • Recall answers: of all real cats, how many did the model find?
  • F1 blends precision and recall into one balanced score

Why Accuracy Can Mislead

Suppose only 1 photo in 100 contains a defect. A lazy model that always answers "no defect" scores 99 percent accuracy and catches nothing. Precision and recall expose the problem right away, since recall drops to zero.

The Precision and Recall Trade-Off

Models output confidence scores. A strict threshold produces fewer detections that are mostly right, which raises precision and lowers recall. A loose threshold catches more targets and adds false alarms, which raises recall and lowers precision.

Threshold   Precision   Recall
0.9         0.95        0.40
0.7         0.85        0.65
0.5         0.70        0.82
0.3         0.50        0.93

Teams pick the threshold that matches the cost of mistakes. A safety inspection favors high recall. A system that triggers costly manual reviews favors high precision.

Intersection over Union

Object detectors draw boxes, so a correct label is not enough. The box must land on the object. IoU measures the overlap between the predicted box and the true box.

 +----------+
 |  Truth   |
 |     +----+-----+
 |     |////|     |
 +-----+----+     |
       | Predicted|
       +----------+

IoU = overlap area / combined area
Example: overlap 50, combined 150 -> IoU = 0.33

Many benchmarks count a detection as correct when IoU reaches 0.5. Stricter tests demand 0.75 or higher.

Average Precision and mAP

Average Precision, or AP, measures the area under the precision-recall curve for one class. Mean Average Precision, or mAP, averages AP across all classes.

Precision
 1.0 |*
     |  *
     |    * *
     |        *
     |           *
 0.0 +--------------- Recall
     0               1     AP = area under curve
  • mAP@0.5 uses an IoU threshold of 0.5
  • mAP@0.5:0.95 averages results across ten IoU thresholds and rewards precise boxes

Segmentation Metrics

Segmentation labels every pixel, so metrics compare predicted masks with true masks.

Dice = 2 x overlap / (size of mask A + size of mask B)
mIoU = average IoU across all classes

Matching Tasks to Metrics

Task                  Main Metrics
--------------------  ------------------------------
Classification        Accuracy, precision, recall, F1
Object detection      IoU, AP, mAP
Segmentation          mIoU, Dice
Tracking              MOTA, IDF1
Text recognition      Character error rate

Good Habits

  • Test on images the model never saw during training
  • Report several metrics rather than a single number
  • Review a sample of mistakes by eye for patterns
  • Check results across lighting, skin tones, ages, and camera types to confirm fair performance

Practice Exercise

Build a tiny test set of 20 photos with 8 containing a bicycle. Run any pretrained classifier and record the four outcomes by hand. Compute precision and recall from your counts. Lower the confidence threshold and repeat the calculation. Plot the two numbers and observe how one rises while the other falls.

Questions to Ask Yourself

  • Which mistake costs more in this project: a false alarm or a missed target?
  • Does the test set resemble the photos the system will meet in daily use?
  • Do several small groups of images, such as night shots, show weaker results?
Previous lesson
Back to course
Next lesson