PyTorch Linear Layers

A linear layer (also called a fully connected layer or dense layer) is the most fundamental building block in a neural network. It takes a set of input values, multiplies them by learned weights, adds a bias, and produces output values. Every classification head, regression output, and embedding projection uses linear layers.

The Math Behind a Linear Layer

A linear layer computes this formula for each output: output = input × weight + bias. In matrix form: Y = XW^T + b. The weights W and bias b are the learnable parameters — the values that change during training.

Input:   [x1, x2, x3]   ← 3 features
Weights: W (learned 3×2 matrix)
Bias:    b (learned 2-element vector)

Output = input × W^T + b   → [y1, y2]  ← 2 output values

Diagram:
  x1 ─┬─w11─→ y1
  x2 ─┤ w21
  x3 ─┘ w31

  x1 ─┬─w12─→ y2
  x2 ─┤ w22
  x3 ─┘ w32

Creating a Linear Layer

import torch
import torch.nn as nn

layer = nn.Linear(in_features=3, out_features=2)

print(layer.weight.shape)   # torch.Size([2, 3])
print(layer.bias.shape)     # torch.Size([2])

Notice the weight matrix shape is [out_features, in_features] — PyTorch stores it transposed relative to the math formula, and handles the transposition internally.

Forward Pass Through a Linear Layer

import torch
import torch.nn as nn

layer = nn.Linear(3, 2)

x = torch.tensor([[1.0, 2.0, 3.0],   # sample 1
                  [4.0, 5.0, 6.0]])  # sample 2

output = layer(x)
print(output.shape)   # torch.Size([2, 2])
print(output)         # 2 samples, each with 2 outputs

Stacking Linear Layers

Real networks stack multiple linear layers with activation functions between them. Without activations, stacking linear layers is mathematically identical to having a single linear layer — the model can't learn non-linear patterns.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),   # 784 inputs → 256 hidden units
    nn.ReLU(),
    nn.Linear(256, 128),   # 256 → 128
    nn.ReLU(),
    nn.Linear(128, 10)     # 128 → 10 classes
)

print(model)

Layer Size Diagram

Input: 784 pixel values (28×28 image flattened)
  │
  ▼ nn.Linear(784 → 256)
  256 values
  │
  ▼ ReLU
  256 values (negatives zeroed out)
  │
  ▼ nn.Linear(256 → 128)
  128 values
  │
  ▼ ReLU
  128 values
  │
  ▼ nn.Linear(128 → 10)
  10 values (one score per class)

Counting Parameters

Each linear layer has in_features × out_features weight parameters plus out_features bias parameters.

nn.Linear(784, 256):
  weights: 784 × 256 = 200,704
  biases:  256
  total:   200,960 parameters

nn.Linear(256, 128):
  weights: 256 × 128 = 32,768
  biases:  128
  total:   32,896 parameters

Linear Layer Without Bias

You can disable the bias term when a bias is not needed — for example, when batch normalization follows the linear layer (batch norm already contains a learnable shift parameter).

layer = nn.Linear(10, 5, bias=False)
print(layer.bias)   # None

Custom Weight Initialization

PyTorch initializes linear layer weights using Kaiming uniform by default. You can override this for specific use cases.

import torch.nn as nn
import torch.nn.init as init

layer = nn.Linear(10, 5)

# Xavier uniform initialization
init.xavier_uniform_(layer.weight)
init.zeros_(layer.bias)

Summary

A linear layer multiplies input by learned weights and adds a bias. It takes in_features inputs and produces out_features outputs. The weight matrix has shape [out_features, in_features]. Stack linear layers with activation functions between them to build deep networks. Each layer adds in_features × out_features + out_features learnable parameters. Set bias=False when following the layer with batch normalization. Linear layers form the backbone of every classification, regression, and language model.