Deep Learning Self-Supervised Learning

Labelled data is expensive. Someone must tag every photo, transcribe every recording, and review every document. The internet holds billions of unlabelled examples that nobody tagged. Self-supervised learning lets a network teach itself from this raw material by inventing its own practice questions from the data.

Three Learning Styles Compared

Supervised:       data + human labels  -> learn the labels
Unsupervised:     data only            -> find patterns
Self-supervised:  data only            -> create labels from the data itself
                                          then learn from those labels

The Fill-in-the-Blank Idea

Imagine a student who covers a word in a sentence and tries to guess it. The correct answer sits right there in the original sentence, so no teacher is needed. The student practises millions of times and absorbs language along the way. Self-supervised learning follows the same trick.

Original:  "The cat sat on the mat."
Practice:  "The cat sat on the [HIDDEN]."
Answer:    "mat"   (taken from the original text)

This practice task is called a pretext task. The pretext task itself has little value. The knowledge the network gains while solving it is the real prize.

Popular Pretext Tasks

Masked Prediction

The network hides parts of the input and predicts them. BERT masks words in sentences. Masked image models hide patches of a picture and rebuild them.

Next-Item Prediction

The network predicts what comes next in a sequence. Large language models train this way, using the next word as the label.

Rotation Prediction

The network rotates an image by 0, 90, 180, or 270 degrees and guesses the rotation. To succeed, it must learn what objects look like upright.

Image rotated 90 degrees -> network answers "90"
Image rotated 0 degrees  -> network answers "0"

Contrastive Learning

The network compares pairs. Two different edits of the same photo form a matching pair, and two different photos form a mismatched pair. The model pulls matching pairs close together and pushes others apart.

Photo A, cropped      \
                        -> pull together
Photo A, colour-shifted /

Photo A  ...........  Photo B -> push apart

The Two-Stage Recipe

Stage 1 (pretraining):
  Huge unlabelled dataset -> pretext task -> general knowledge

Stage 2 (fine-tuning):
  Small labelled dataset -> specific task -> accurate model

Stage 1 may use millions of examples without labels. Stage 2 may need only a few hundred labelled examples, because the network already understands the basics.

Comparison with Transfer Learning

Transfer learning reuses a model trained on a labelled dataset. Self-supervised learning builds the starting model without labels. Both feed into the same fine-tuning step, and modern pretrained models usually combine the two ideas.

Benefits

• Reduces the cost of manual labelling
• Uses the huge supply of unlabelled data
• Produces general features that transfer to many tasks
• Improves results when labelled data stays scarce

Applications

• Language models that learn from raw text
• Speech recognition trained on unlabelled audio
• Medical imaging where expert labels are rare
• Satellite photo analysis

Limitations

• Pretraining needs large computing resources
• Poor pretext tasks teach unhelpful features
• Biases in raw data pass into the model

Key Terms

• Self-supervised learning - learning from labels created out of the data
• Pretext task - the practice question invented for pretraining
• Contrastive learning - learning by comparing similar and different pairs
• Pretraining - the first stage on unlabelled data
• Fine-tuning - the second stage on labelled data