Computer Vision Feature Matching

Feature matching finds the same physical point in two different pictures. Think of two tourists photographing one cathedral from different spots. A person recognizes the clock tower and the arched door in both photos instantly. Feature matching teaches software to make the same links automatically.

The Matching Pipeline

Image A ---> Keypoints + Descriptors ---+
                                        +--> Match --> Filter --> Use result
Image B ---> Keypoints + Descriptors ---+
  1. Detect keypoints, which are distinctive spots such as corners
  2. Describe each keypoint with a list of numbers called a descriptor
  3. Compare descriptors between images to propose matches
  4. Filter out weak or wrong matches
  5. Use the surviving matches for stitching, alignment, or recognition

Visualizing Matches

 Image A                    Image B
+-----------+              +-----------+
|  o1       |--------------|    o1     |
|       o2  |--------------|  o2       |
|   o3      |---------\    |       o3  |
|           |          \---|   x       |   <- wrong match
+-----------+              +-----------+

Most lines connect true partners. A few lines connect unrelated points, and filtering exists to remove them.

Matching Descriptors

The brute-force matcher compares one descriptor against all descriptors in the other image and keeps the closest. The distance measure depends on descriptor type.

Descriptor type          Distance measure
-----------------------  -----------------
ORB, BRIEF, BRISK        Hamming (bit differences)
SIFT, SURF-like floats   L2 (straight-line distance)

import cv2

a = cv2.imread("scene1.jpg", cv2.IMREAD_GRAYSCALE)
b = cv2.imread("scene2.jpg", cv2.IMREAD_GRAYSCALE)

orb = cv2.ORB_create(nfeatures=1500)
kp1, des1 = orb.detectAndCompute(a, None)
kp2, des2 = orb.detectAndCompute(b, None)

bf = cv2.BFMatcher(cv2.NORM_HAMMING)
pairs = bf.knnMatch(des1, des2, k=2)

The Ratio Test

A good match stands far ahead of the runner-up. A bad match has a second-best candidate almost as close as the first. David Lowe proposed comparing the two distances and keeping a match only when the best one is clearly smaller.

keep match if   best_distance < 0.75 x second_best_distance

Match 1: best 20, second 70  -> 20 < 52  keep
Match 2: best 48, second 55  -> 48 > 41  drop

good = []
for m, n in pairs:
    if m.distance < 0.75 * n.distance:
        good.append(m)

Removing Wrong Matches With RANSAC

Some bad matches survive the ratio test. RANSAC removes them by guessing. The algorithm picks four random matches, builds a transformation between the images, and counts how many other matches agree. The process repeats many times, and the best-agreeing transformation wins. Matches that disagree are outliers.

import numpy as np

src = np.float32([kp1[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
dst = np.float32([kp2[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)

H, mask = cv2.findHomography(src, dst, cv2.RANSAC, 5.0)
print("Inliers:", int(mask.sum()))

The matrix H, called a homography, describes how one image maps onto the other. Warping image A with H aligns it to image B.

Faster Search With FLANN

Brute force slows down with thousands of keypoints. FLANN builds a smart index of descriptors and finds near neighbors quickly, trading a tiny loss in precision for large speed gains.

Where Matching Shines

  • Panorama stitching from overlapping photos
  • Finding a book cover inside a cluttered shelf photo
  • Aligning medical scans taken at different times
  • Tracking camera movement in robotics

Troubleshooting

  • Few matches often mean low texture, such as plain white walls
  • Many wrong matches suggest a ratio threshold that is too loose
  • Large viewpoint changes need stronger descriptors like SIFT
  • Blurry images yield weak keypoints, so sharpen or reshoot

Real-World Walkthrough: Stitching a Panorama

A traveler takes three overlapping photos of a mountain range. The software detects keypoints in each image and matches them between neighbors. RANSAC removes bad pairs and produces a homography for each photo. The program warps the side photos into the plane of the center photo and blends the overlapping edges. The result looks like one wide photograph.

Photo 1 \
Photo 2 -- Match -- RANSAC homography -- Warp -- Blend -- Panorama
Photo 3 /

Choosing a Descriptor

  • ORB runs fast and is free to use, which suits phones and live video
  • SIFT handles scale and rotation changes with higher accuracy and runs slower
  • AKAZE offers a balanced middle option for many projects

Practice Exercise

Photograph a book cover on a table and photograph the same table with the book among other objects. Match the cover photo against the busy scene and draw the outline of the book using the homography. Rotate the book and test how far the match survives.

Previous lesson
Back to course
Next lesson