TensorFlow Operations
Operations (ops) are the mathematical computations that transform tensors into new tensors. Every layer in a neural network, every loss calculation, and every gradient update runs through a series of operations. TensorFlow provides hundreds of built-in ops covering arithmetic, linear algebra, statistics, and more. Knowing the most important ones lets you understand what happens inside your model and write efficient custom computations.
Operations as Assembly Line Steps
Think of a tensor as a box of raw materials moving through a factory. Each operation is one workstation on the assembly line. The input box arrives, the workstation transforms the contents, and a new box exits. The original box is unchanged — TensorFlow operations produce new tensors rather than modifying existing ones (unless explicitly told to do otherwise).
Input Tensor A ──► [Addition Op] ──► Output Tensor C Input Tensor B ──► [Addition Op] ──► (A + B stored in C) A and B are unchanged. C is a brand-new tensor.
Basic Arithmetic Operations
import tensorflow as tf a = tf.constant([10, 20, 30], dtype=tf.float32) b = tf.constant([1, 2, 3], dtype=tf.float32) # Addition print(tf.add(a, b)) # [11, 22, 33] print(a + b) # Same — Python operators map to tf ops # Subtraction print(tf.subtract(a, b)) # [9, 18, 27] print(a - b) # Multiplication (element-wise) print(tf.multiply(a, b)) # [10, 40, 90] print(a * b) # Division print(tf.divide(a, b)) # [10.0, 10.0, 10.0] print(a / b) # Power print(tf.pow(a, 2)) # [100, 400, 900] print(a ** 2) # Square root print(tf.sqrt(a)) # [3.16, 4.47, 5.47]
Reduction Operations
Reduction ops collapse one or more dimensions of a tensor into a single summary value. They are used constantly in loss functions, normalization, and pooling layers.
x = tf.constant([[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0]])
# Sum all elements
print(tf.reduce_sum(x)) # 21.0
# Sum along axis 0 (collapse rows)
print(tf.reduce_sum(x, axis=0)) # [5.0, 7.0, 9.0]
# Sum along axis 1 (collapse columns)
print(tf.reduce_sum(x, axis=1)) # [6.0, 15.0]
# Mean
print(tf.reduce_mean(x)) # 3.5
# Maximum value
print(tf.reduce_max(x)) # 6.0
print(tf.reduce_max(x, axis=1)) # [3.0, 6.0]
# Minimum value
print(tf.reduce_min(x)) # 1.0
# Index of maximum value
print(tf.argmax(x, axis=1)) # [2, 2] — column 2 is largest in each row
print(tf.argmin(x, axis=1)) # [0, 0]
Matrix Operations
Linear algebra operations form the mathematical core of neural networks. Every Dense layer performs a matrix multiplication.
A = tf.constant([[1.0, 2.0],
[3.0, 4.0]])
B = tf.constant([[5.0, 6.0],
[7.0, 8.0]])
# Matrix multiplication (dot product)
C = tf.matmul(A, B)
# [[1×5+2×7, 1×6+2×8], = [[19, 22],
# [3×5+4×7, 3×6+4×8]] [43, 50]]
# Transpose (flip rows and columns)
print(tf.transpose(A))
# [[1.0, 3.0],
# [2.0, 4.0]]
# Inverse (requires square matrix)
print(tf.linalg.inv(A))
# Determinant
print(tf.linalg.det(A)) # 1×4 - 2×3 = -2.0
Comparison Operations
a = tf.constant([1, 5, 3, 7, 2]) b = tf.constant([2, 5, 1, 6, 9]) print(tf.equal(a, b)) # [False, True, False, False, False] print(tf.not_equal(a, b)) # [True, False, True, True, True] print(tf.greater(a, b)) # [False, False, True, True, False] print(tf.less(a, b)) # [True, False, False, False, True] print(tf.greater_equal(a, b)) # [False, True, True, True, False]
String and Type Operations
# Cast between types x = tf.constant([1, 2, 3], dtype=tf.int32) y = tf.cast(x, dtype=tf.float32) # Check data type print(x.dtype) # tf.int32 print(y.dtype) # tf.float32 # Concatenate tensors along an axis a = tf.constant([[1, 2], [3, 4]]) b = tf.constant([[5, 6]]) print(tf.concat([a, b], axis=0)) # [[1, 2], # [3, 4], # [5, 6]] # Stack tensors (adds a new dimension) t1 = tf.constant([1, 2, 3]) t2 = tf.constant([4, 5, 6]) print(tf.stack([t1, t2], axis=0)) # [[1, 2, 3], # [4, 5, 6]]
Shape Manipulation Operations
x = tf.constant([1, 2, 3, 4, 5, 6]) # Reshape — change dimensions without changing data print(tf.reshape(x, [2, 3])) # [[1, 2, 3], # [4, 5, 6]] print(tf.reshape(x, [3, 2])) # [[1, 2], # [3, 4], # [5, 6]] # Use -1 to let TensorFlow infer one dimension automatically print(tf.reshape(x, [2, -1])) # TensorFlow infers columns = 3 # Expand dimensions (insert a new axis) y = tf.constant([1.0, 2.0, 3.0]) # shape: (3,) print(tf.expand_dims(y, axis=0)) # shape: (1, 3) print(tf.expand_dims(y, axis=1)) # shape: (3, 1) # Squeeze (remove dimensions of size 1) z = tf.constant([[[1, 2, 3]]]) # shape: (1, 1, 3) print(tf.squeeze(z)) # shape: (3,)
Conditional Operations
# tf.where — select from two tensors based on a condition condition = tf.constant([True, False, True, False]) x = tf.constant([1.0, 2.0, 3.0, 4.0]) y = tf.constant([10.0, 20.0, 30.0, 40.0]) result = tf.where(condition, x, y) print(result) # [1.0, 20.0, 3.0, 40.0] # Where condition=True → take from x # Where condition=False → take from y # Clipping — constrain values to a range raw = tf.constant([-2.0, 0.5, 3.0, 10.0]) clipped = tf.clip_by_value(raw, clip_value_min=0.0, clip_value_max=1.0) print(clipped) # [0.0, 0.5, 1.0, 1.0]
Statistical Operations
data = tf.constant([2.0, 4.0, 4.0, 4.0, 5.0, 5.0, 7.0, 9.0]) print(tf.reduce_mean(data)) # Mean: 5.0 print(tf.math.reduce_std(data)) # Standard deviation: 2.0 print(tf.math.reduce_variance(data)) # Variance: 4.0 # Cumulative sum print(tf.cumsum(data)) # [2, 6, 10, 14, 19, 24, 31, 40]
Operations Inside a Neural Network Layer
This diagram shows exactly which operations run when data passes through a single Dense layer:
Input x (shape: batch_size × input_features)
│
├── [tf.matmul(x, W)] ← Matrix multiplication with weight matrix W
│ (shape: batch_size × output_units)
│
├── [tf.add(result, b)] ← Add bias vector b
│ (shape: output_units)
│
└── [activation(result)] ← Apply activation function
e.g. tf.nn.relu(), tf.nn.sigmoid()
# Manual Dense layer forward pass using ops
W = tf.Variable(tf.random.normal([20, 64]))
b = tf.Variable(tf.zeros([64]))
def dense_forward(x):
z = tf.matmul(x, W) + b # Linear transformation
return tf.nn.relu(z) # Activation function
The @tf.function Decorator
By default, TensorFlow executes operations eagerly — one by one as you write them. The @tf.function decorator compiles a Python function into an optimized TensorFlow graph that runs significantly faster, especially for loops and training steps.
@tf.function
def fast_computation(x, y):
return tf.matmul(x, y) + tf.reduce_sum(x)
# First call traces the function (slightly slower)
result = fast_computation(tf.ones([100, 100]), tf.ones([100, 100]))
# Subsequent calls run the compiled graph (much faster)
result = fast_computation(tf.ones([100, 100]), tf.ones([100, 100]))
Operations are the vocabulary of TensorFlow. Every concept from here forward — activation functions, loss functions, gradients, layer outputs — is built from these fundamental ops. The next topic explains TensorFlow's data type system in detail, showing you why choosing the right dtype speeds up training and prevents numerical errors.