CV Vision Transformers

Transformers first conquered language tasks, where they read sentences as sequences of words. In 2020, researchers showed that the same machinery can read images when the image is cut into a sequence of small pieces. The resulting models, called Vision Transformers or ViT, now rival and often beat convolutional networks.

The Jigsaw Puzzle Idea

Imagine a photo cut into square puzzle pieces. A convolutional network scans nearby pieces to build up understanding slowly. A Vision Transformer lets every piece talk to every other piece at once. A piece showing a wheel can immediately learn that a distant piece shows a car roof.

Step by Step Pipeline

Image 224x224
     |
     v
Cut into 16x16 patches  ->  14 x 14 = 196 patches
     |
     v
Flatten each patch + linear projection  ->  196 vectors
     |
     v
Add [CLS] token + position embeddings
     |
     v
Transformer encoder layers (self-attention + MLP) x 12
     |
     v
[CLS] output -> classifier -> "golden retriever"

Patches as Words

Each 16x16 patch holds 768 pixel values for a color image. A linear layer converts those values into a vector. Language models treat words the same way, so the patch acts as the word of an image.

Position Embeddings

Attention ignores order by itself. Position embeddings tag each patch with its location, so the network knows the top-left patch from the bottom-right patch.

The Class Token

A special extra vector, called the class token, travels through every layer and gathers information from all patches. The final classifier reads only this token.

Self-Attention in Plain Words

Every patch asks a question and every patch offers an answer. A patch compares its question against all others and scores how relevant each one is. High scores pull more information in. The weights form an attention map, which shows what the model focuses on.

Patch "wheel" attends to:
  car body   ########  high
  road       ###       medium
  sky        #         low

Convolution Versus Attention

Feature              CNN                    Vision Transformer
-------------------  ---------------------  --------------------
View of the image    Local, grows with depth Global from layer one
Built-in assumptions Strong (nearby pixels)  Weak
Data needed          Moderate               Large
Computing cost       Grows with image area  Grows with patch count squared

CNNs bake in the idea that nearby pixels matter. That assumption helps when data is limited. Transformers learn such rules from data, so they need big training sets, but they reach higher accuracy once enough data exists.

Popular Variants

  • DeiT trains ViT efficiently on smaller datasets through a teacher model
  • Swin Transformer limits attention to shifted windows and handles large images and detection tasks
  • DINO learns strong visual features without labels
  • CLIP links images and text, so a model can match photos with written descriptions

Using a Pretrained ViT

from torchvision import models

weights = models.ViT_B_16_Weights.DEFAULT
model = models.vit_b_16(weights=weights)
model.eval()

preprocess = weights.transforms()

The name vit_b_16 means the base-size model with 16x16 patches.

When to Choose a Vision Transformer

  • Large datasets or strong pretrained weights are available
  • Global context matters, such as scene understanding
  • Fine-tuning a ready-made model fits the budget

Small datasets and tight hardware limits often favor a CNN such as ResNet or MobileNet. Hybrid designs that mix convolution and attention offer a middle path.

Key Takeaways

  • Patches turn an image into a sequence of tokens
  • Self-attention connects distant regions directly
  • Pretraining on large data drives the best results

Practice Exercise

Load a pretrained ViT and classify five photos. Print the top three guesses for each one. Crop a photo so the main object sits in a corner and repeat the test. Global attention often keeps the prediction steady, while a small crop that removes context can shift it. Notes from this experiment build intuition about what the model needs to see.

Common Questions

Why 16x16 patches?

Smaller patches give more tokens and finer detail, but attention cost grows with the square of the token count. Patches of 16x16 pixels balance detail and speed for 224x224 images. Larger images often use bigger patches or windowed attention.

Do Vision Transformers replace CNNs?

Both families thrive. CNNs remain popular on phones and small devices because they run efficiently. Transformers lead many benchmarks that offer large training sets and strong computing budgets.

Previous lesson
Back to course
Next lesson