Machine Learning Regularization L1 and L2

Regularization adds a penalty to the model's cost function to prevent it from growing too complex. It discourages the model from assigning very large values to feature weights (coefficients), which is a hallmark of overfitting. The two most common forms are L1 (Lasso) and L2 (Ridge) regularization.

Why Large Coefficients Cause Overfitting

In Linear Regression, the model learns a formula like:
  Price = 1.2×Size + 0.8×Rooms + 150×Noise_Feature + 5000

If "Noise_Feature" is random data, the model assigns it a
large coefficient (150) because it accidentally helps on training data.
On test data, this large coefficient causes wild, wrong predictions.

Regularization penalizes large coefficients → they stay small
→ less sensitivity to noise → better generalization.

L2 Regularization (Ridge)

Cost Function with L2:
  Total Cost = Original Loss + λ × Sum of (all coefficients²)

Effect:
  All coefficients get smaller, but none reach exactly zero.
  Every feature still contributes, just with reduced influence.

λ (lambda) = regularization strength:
  λ = 0  → No regularization (original model)
  λ small → Light penalty (slight shrinkage)
  λ large → Heavy penalty (all coefficients near zero)

Example:
  Without Ridge: coefficients = [150, 0.8, 1.2, 0.5, 200]
  With Ridge (λ=1): coefficients ≈ [12, 0.75, 1.1, 0.45, 18]

The noise feature dropped from 150 to 12 — still in the model
but much less influential.

Best for: All features are somewhat relevant.
          Want to keep all features but reduce their impact.

L1 Regularization (Lasso)

Cost Function with L1:
  Total Cost = Original Loss + λ × Sum of |all coefficients|

Effect:
  Some coefficients are pushed to EXACTLY zero.
  Those features are completely removed from the model.
  L1 performs automatic feature selection.

Example:
  Without Lasso: coefficients = [150, 0.8, 1.2, 0.5, 200]
  With Lasso (λ=1): coefficients ≈ [0, 0.6, 1.0, 0, 0]

Features 1, 4, and 5 were eliminated (set to zero).
Only features 2 and 3 remain in the model.

Best for: Many features, only a few are truly relevant.
          Want a sparse, interpretable model.

L1 vs L2 Comparison

┌────────────────────────┬─────────────────────┬─────────────────────┐
│ Feature                │ L1 (Lasso)          │ L2 (Ridge)          │
├────────────────────────┼─────────────────────┼─────────────────────┤
│ Penalty term           │ |coefficient|       │ coefficient²        │
│ Coefficients → zero?   │ Yes (exactly zero)  │ No (near zero)      │
│ Feature selection?     │ Yes (automatic)     │ No (keeps all)      │
│ Best when              │ Many irrelevant      │ All features matter │
│                        │ features exist       │ roughly equally     │
│ Sparse model?          │ Yes                 │ No                  │
│ Algorithm name         │ Lasso Regression    │ Ridge Regression    │
└────────────────────────┴─────────────────────┴─────────────────────┘

Elastic Net: Combining L1 and L2

Elastic Net uses both penalties together:
  Total Cost = Loss + λ1 × |coeff| + λ2 × coeff²

Benefits:
  - Selects important features (from L1)
  - Handles correlated features well (from L2)
  - More stable than pure L1 when features are correlated

Use Elastic Net when:
  - Many features with some being irrelevant
  - Several correlated features exist in the dataset

Choosing Lambda

Lambda is a hyperparameter — it must be tuned.

Method: Cross Validation
  Try many values: λ = 0.001, 0.01, 0.1, 1, 10, 100
  For each λ, measure cross-validation accuracy
  Choose the λ that gives the best validation performance

Too small λ → Almost no regularization → May overfit
Too large λ → Too much shrinkage → May underfit

Regularization Beyond Linear Models

Regularization applies to many model types:

┌─────────────────────────────────┬────────────────────────────────┐
│ Model                           │ Regularization Technique       │
├─────────────────────────────────┼────────────────────────────────┤
│ Linear / Logistic Regression    │ L1 (Lasso), L2 (Ridge),        │
│                                 │ Elastic Net                    │
│ Neural Networks                 │ L2 weight decay, Dropout,      │
│                                 │ Early Stopping                 │
│ Decision Trees / Random Forest  │ max_depth, min_samples_leaf    │
│ SVM                             │ C parameter (inverse of λ)     │
└─────────────────────────────────┴────────────────────────────────┘