TensorFlow Constants and Variables
TensorFlow stores data in two distinct types of tensor containers: constants and variables. Choosing the right type matters because one is immutable (fixed forever) and the other is mutable (can be updated during training). Every weight inside a neural network is a variable. Every piece of fixed configuration — like the number of classes or a decay rate — is a constant.
The Whiteboard Analogy
A constant is like text printed in a book — permanent and unchanged no matter how many times you read it. A variable is like writing on a whiteboard — you can erase and rewrite it as many times as you need. During training, TensorFlow erases and rewrites every variable (weight) after each batch, moving it slightly closer to the correct value.
tf.constant — Immutable Tensors
A constant holds a fixed value that cannot change after creation. TensorFlow stores constants in the computation graph as literal values.
import tensorflow as tf
# Scalar constant
learning_rate = tf.constant(0.001, dtype=tf.float32)
# Vector constant
class_names = tf.constant(['cat', 'dog', 'bird'])
# Matrix constant
kernel = tf.constant([[1, 0, -1],
[1, 0, -1],
[1, 0, -1]], dtype=tf.float32)
print(learning_rate) # tf.Tensor(0.001, shape=(), dtype=float32)
print(class_names) # tf.Tensor([b'cat' b'dog' b'bird'], ...)
print(kernel.shape) # (3, 3)
Key Properties of Constants
- Cannot be modified after creation
- Stored directly in the graph — no memory allocation needed at runtime
- Ideal for fixed configuration values, lookup tables, and filter initializers
- Attempting to assign a new value raises an error
# This raises an error — constants are immutable c = tf.constant(5.0) c.assign(10.0) # AttributeError: 'EagerTensor' has no attribute 'assign'
tf.Variable — Mutable Tensors
A variable holds a value that TensorFlow can update during training. Every weight matrix and bias vector inside a neural network layer is stored as a tf.Variable. The optimizer calls assign or apply_gradients to update variable values after each training step.
# Create a variable
weight = tf.Variable([[0.5, -0.3],
[0.1, 0.8]], dtype=tf.float32)
print(weight)
#
# Read the current value
print(weight.numpy())
# [[ 0.5 -0.3]
# [ 0.1 0.8]]
# Update the variable in place
weight.assign([[1.0, 2.0],
[3.0, 4.0]])
# Add to the current value
weight.assign_add([[0.1, 0.1],
[0.1, 0.1]])
# Subtract from the current value
weight.assign_sub([[0.05, 0.05],
[0.05, 0.05]])
Constants vs. Variables: Side-by-Side Comparison
Feature tf.constant tf.Variable ──────────────────────────────────────────────────────────── Mutable No Yes Tracked by GradientTape No (by default) Yes (automatic) Use in training Fixed hyperparams Model weights Memory allocation At graph build At runtime Typical use Config, labels Weights, biases assign() method Not available Available ────────────────────────────────────────────────────────────
How Variables Get Updated During Training
Diagram — Variable Update Loop:
Initial weight: tf.Variable(0.5)
│
▼
[Forward Pass]
prediction = weight * input
│
▼
[Loss Calculation]
loss = (prediction - true_value)²
│
▼
[GradientTape computes gradient]
gradient = d(loss) / d(weight)
│
▼
[Optimizer updates weight]
weight = weight - learning_rate × gradient
weight.assign(new_value)
│
▼
Updated weight: tf.Variable(0.48) ← slightly closer to correct value
Creating Variables With Different Initializers
Neural network weights need smart initialization. Starting all weights at zero means every neuron computes the same output and learns the same thing — the network never specializes. TensorFlow provides several initializers:
# Glorot uniform (default for Dense layers — balances signal strength) w1 = tf.Variable(tf.keras.initializers.GlorotUniform()(shape=(128, 64))) # He normal (recommended for ReLU activations) w2 = tf.Variable(tf.keras.initializers.HeNormal()(shape=(64, 32))) # Random normal with custom mean and stddev w3 = tf.Variable(tf.random.normal(shape=(32, 10), mean=0.0, stddev=0.05)) # All zeros (only appropriate for bias vectors) b1 = tf.Variable(tf.zeros(shape=(64,))) # All ones b2 = tf.Variable(tf.ones(shape=(10,)))
Trainable vs. Non-Trainable Variables
Variables have a trainable flag. Trainable variables get updated during training. Non-trainable variables exist in the model but the optimizer ignores them. BatchNormalization layers use non-trainable variables to store running mean and variance statistics.
# Non-trainable variable — the optimizer skips this
running_mean = tf.Variable(0.0, trainable=False)
# Check which variables are trainable in a model
for var in model.trainable_variables:
print(var.name, var.shape)
for var in model.non_trainable_variables:
print(var.name, var.shape)
Variable Scope and Naming
Every tf.Variable has an internal name that helps you identify it in model summaries and saved checkpoints.
w = tf.Variable(tf.zeros([3, 3]), name='conv_kernel') print(w.name) # conv_kernel:0 # The ":0" suffix is TensorFlow's way of saying # "this is the first output tensor of the 'conv_kernel' variable operation"
Constants and Variables Inside a Custom Layer
import tensorflow as tf
class SimpleLinear(tf.keras.layers.Layer):
def __init__(self, units):
super().__init__()
self.units = units
def build(self, input_shape):
# Variables created in build() are trainable by default
self.w = self.add_weight(
name='kernel',
shape=(input_shape[-1], self.units),
initializer='glorot_uniform',
trainable=True
)
self.b = self.add_weight(
name='bias',
shape=(self.units,),
initializer='zeros',
trainable=True
)
# Constant scale factor — never changes
self.scale = tf.constant(2.0)
def call(self, inputs):
return tf.matmul(inputs, self.w) + self.b
Checking Memory Usage of Variables
# Count total trainable parameters in a model
total_params = sum([tf.size(v).numpy() for v in model.trainable_variables])
print(f"Total parameters: {total_params:,}")
# Each float32 parameter uses 4 bytes
memory_mb = (total_params * 4) / (1024 * 1024)
print(f"Approximate model size: {memory_mb:.1f} MB")
Practical Rules for Using Each Type
- Use tf.constant for learning rates, number of classes, fixed filter weights, and any value that never changes during training
- Use tf.Variable for all model weights, biases, embedding matrices, and any value the optimizer needs to update
- Use trainable=False for variables that track running statistics (like BatchNormalization) or that you deliberately freeze during transfer learning
- Prefer layer.add_weight() over raw tf.Variable when defining weights inside custom Keras layers, because Keras then tracks, saves, and loads those weights automatically
Understanding the distinction between constants and variables explains why freezing layers in transfer learning works: setting layer.trainable = False marks all that layer's variables as non-trainable, so the optimizer skips them entirely. The next topic covers the mathematical operations you can perform on tensors — the building blocks of every layer computation.