TensorFlow Activation Functions
Activation functions introduce non-linearity into neural networks. Without them, stacking 100 Dense layers produces exactly the same result as a single linear equation — all that depth accomplishes nothing. An activation function takes the output of a neuron's linear calculation and transforms it before passing it forward. This non-linear transformation lets deep networks learn curves, decision boundaries, and complex patterns that no straight line could fit.
Why Non-Linearity Matters
The world's data is rarely linear. House prices do not increase proportionally with every extra square meter — small apartments are expensive per square foot, medium ones are cheaper, and enormous mansions are expensive again. Cancer detection requires recognizing complex pixel combinations, not just "more brightness = more likely cancer." Activation functions give neural networks the ability to model these real-world complexities.
Without activation (all linear):
Layer 1: output = W1 × input + b1
Layer 2: output = W2 × (W1 × input + b1) + b2
= (W2×W1) × input + (W2×b1 + b2)
= W_combined × input + b_combined
→ Just one linear equation, no matter how deep
With activation (non-linear):
Layer 1: output = ReLU(W1 × input + b1)
Layer 2: output = ReLU(W2 × ReLU(...) + b2)
→ Cannot be collapsed into one equation
→ Can approximate any function
ReLU — Rectified Linear Unit
ReLU is the most widely used activation function in hidden layers. It outputs the input if positive and zero if negative. Its simplicity makes it fast to compute and its gradient is either 0 or 1, which prevents the vanishing gradient problem that plagued earlier activations.
Formula: f(x) = max(0, x)
Input: -3 -1 0 1 3 5
Output: 0 0 0 1 3 5
Graph:
| /
| /
─────|──────/─────── x
| /
| 0
import tensorflow as tf x = tf.constant([-3.0, -1.0, 0.0, 1.0, 3.0]) print(tf.nn.relu(x)) # [0.0, 0.0, 0.0, 1.0, 3.0] # In a layer: tf.keras.layers.Dense(64, activation='relu')
The Dying ReLU Problem
If many inputs are always negative, the neurons always output zero and their gradients are zero — the neuron "dies" and stops learning. Solutions: use He initialization, reduce the learning rate, or switch to Leaky ReLU.
Leaky ReLU
Leaky ReLU allows a small, non-zero gradient for negative inputs, preventing neurons from dying permanently.
Formula: f(x) = max(0.1x, x) Input: -3 -1 0 1 3 Output: -0.3 -0.1 0 1 3
# As a layer
tf.keras.layers.LeakyReLU(alpha=0.1)
# As a separate layer in Sequential
model = tf.keras.Sequential([
tf.keras.layers.Dense(64),
tf.keras.layers.LeakyReLU(alpha=0.2)
])
Sigmoid
Sigmoid squashes any input to a value between 0 and 1. This makes it perfect for binary classification output layers where the output represents a probability.
Formula: f(x) = 1 / (1 + e^(-x))
Input: -5 -2 0 2 5
Output: 0.007 0.119 0.5 0.880 0.993
Graph:
1 |──────────────────────────╮
| ╭──╯
0.5 |────────────────────╮──
| ╭───╯
0 |────────────────╯──────── x
x = tf.constant([-5.0, -2.0, 0.0, 2.0, 5.0]) print(tf.nn.sigmoid(x)) # [0.007, 0.119, 0.500, 0.880, 0.993] # Used in binary classification output tf.keras.layers.Dense(1, activation='sigmoid')
Sigmoid Drawback
For very large or very small inputs, the sigmoid gradient approaches zero. This creates the vanishing gradient problem in deep networks. Avoid sigmoid in hidden layers — use it only in binary classification output layers.
Tanh — Hyperbolic Tangent
Tanh is a shifted and scaled version of sigmoid. It squashes inputs to between -1 and 1, centering outputs around zero. Zero-centered outputs help gradient flow in RNNs.
Formula: f(x) = (e^x - e^(-x)) / (e^x + e^(-x)) Input: -3 -1 0 1 3 Output: -0.995 -0.762 0 0.762 0.995 # In a layer: tf.keras.layers.Dense(64, activation='tanh') # Common use: LSTM and GRU hidden states
Softmax
Softmax converts a vector of raw scores into a probability distribution. Every output value falls between 0 and 1, and all outputs sum to exactly 1. Use softmax in the output layer for multi-class classification where the model picks one class from many.
Formula: softmax(x_i) = e^(x_i) / Σ e^(x_j) Example — 3-class classification: Raw scores (logits): [2.0, 1.0, 0.1] After softmax: [0.659, 0.242, 0.099] Sum: 1.000 The class with the highest probability (0.659, index 0) is the prediction.
x = tf.constant([[2.0, 1.0, 0.1]]) print(tf.nn.softmax(x)) # [[0.659, 0.242, 0.099]] # Final layer for multi-class classification tf.keras.layers.Dense(num_classes, activation='softmax')
ELU — Exponential Linear Unit
ELU allows negative outputs for negative inputs (unlike ReLU) but does so smoothly. This reduces the bias shift problem that occurs when ReLU neurons always output zero or positive values.
Formula: f(x) = x if x > 0 else α(e^x - 1) where α=1.0 tf.keras.layers.Dense(64, activation='elu')
SELU — Scaled ELU
SELU is a self-normalizing activation: if you use it throughout a deep network with LeCun initialization, the network outputs tend to normalize automatically (mean near 0, variance near 1). This removes the need for BatchNormalization in many architectures.
tf.keras.layers.Dense(64, activation='selu',
kernel_initializer='lecun_normal')
Linear (No Activation)
Specifying no activation leaves the neuron outputs as raw numbers with no transformation. Use this in regression output layers where you want to predict an unbounded numerical value.
# Regression — predict house price (any positive number) tf.keras.layers.Dense(1) # No activation tf.keras.layers.Dense(1, activation='linear') # Same thing explicitly
Activation Function Quick Reference
Function Output Range Best Use ──────────────────────────────────────────────────────────────── ReLU [0, +∞) Hidden layers (most common) Leaky ReLU (-∞, +∞) Hidden layers (avoids dying neurons) Sigmoid (0, 1) Binary classification output Tanh (-1, 1) RNN hidden states Softmax (0, 1), sum=1 Multi-class classification output ELU (-α, +∞) Hidden layers (smooth negatives) SELU (-λα, +∞) Deep fully-connected (self-normalizing) Linear (-∞, +∞) Regression output ────────────────────────────────────────────────────────────────
Applying Activation as a Separate Layer
# Inline approach (most common)
tf.keras.layers.Dense(64, activation='relu')
# Separate layer approach (useful for inspecting intermediate values)
model = tf.keras.Sequential([
tf.keras.layers.Dense(64),
tf.keras.layers.Activation('relu'),
tf.keras.layers.Dense(10),
tf.keras.layers.Activation('softmax')
])
Custom Activation Functions
import tensorflow as tf
# Define a custom activation: swish = x × sigmoid(x)
def swish(x):
return x * tf.nn.sigmoid(x)
# Use it in a layer
layer = tf.keras.layers.Dense(64, activation=swish)
# Or register it so it can be referenced by name
@tf.keras.utils.register_keras_serializable()
def my_activation(x):
return tf.nn.relu(x) ** 2 # Squared ReLU
layer = tf.keras.layers.Dense(64, activation='my_activation')
Choosing the right activation function is as important as choosing the right architecture. ReLU handles most hidden layers well. Sigmoid and softmax handle outputs. Tanh handles sequential data. The next topic explains model compilation — where you wire together the loss function, optimizer, and metrics that guide the learning process.