TensorFlow Data Types
Every tensor in TensorFlow stores a specific type of number called its dtype (data type). Choosing the right dtype affects training speed, memory consumption, numerical accuracy, and hardware compatibility. Using float32 instead of float64 can cut memory usage in half and double training speed on GPUs. Using the wrong dtype silently produces incorrect results. This topic explains every dtype TensorFlow supports and tells you exactly when to use each one.
The Storage Container Analogy
Think of dtypes like different sizes of storage containers. A tiny container (int8) holds numbers from -128 to 127 — very small but uses minimal space. A large container (float64) holds extremely precise decimal numbers but consumes eight times more space than a small container (int8). Neural network training rarely needs that extra precision. The right container size is the smallest one that holds your data accurately.
Floating-Point Types (Decimal Numbers)
tf.float32 — The Default Standard
Float32 stores decimal numbers with 32 bits (4 bytes). It provides enough precision for virtually all machine learning tasks and runs efficiently on both CPUs and GPUs. Almost every neural network weight, bias, and activation value uses float32.
x = tf.constant(3.14159, dtype=tf.float32) print(x) # tf.Tensor(3.14159, shape=(), dtype=float32) # Range: approximately ±3.4 × 10^38 # Precision: ~7 decimal digits
tf.float16 — Half Precision
Float16 uses only 16 bits (2 bytes) — half the memory of float32. Modern NVIDIA GPUs (V100, A100) run float16 operations twice as fast as float32. Many training pipelines use mixed precision: float16 for most operations, float32 for the final accumulation step to preserve accuracy.
x = tf.constant(3.14, dtype=tf.float16) # Range: approximately ±65,504 # Precision: ~3 decimal digits # Warning: values outside the range produce Inf or NaN
tf.bfloat16 — Brain Float
Bfloat16 also uses 16 bits but distributes them differently from float16. It has the same exponent range as float32 but less decimal precision. Google TPUs use bfloat16 natively. It is less prone to overflow than float16 during training.
x = tf.constant(3.14, dtype=tf.bfloat16) # Same range as float32, less precision # Preferred over float16 on TPUs and for unstable training
tf.float64 — Double Precision
Float64 uses 64 bits (8 bytes) and provides 15 decimal digits of precision. Scientific computing and physics simulations sometimes need this precision, but machine learning almost never does. Avoid float64 in neural networks — it doubles memory usage and halves GPU speed with no practical benefit for most models.
x = tf.constant(3.14159265358979, dtype=tf.float64) # High precision but expensive — use only when scientifically necessary
Integer Types (Whole Numbers)
tf.int32 — Standard Integer
The default integer type. Holds whole numbers from -2,147,483,648 to 2,147,483,647. Use for class labels, indices, and counts.
labels = tf.constant([0, 1, 2, 1, 0], dtype=tf.int32)
tf.int64 — Large Integer
Holds very large whole numbers (up to about 9.2 × 10^18). Use for dataset sizes or file byte offsets that exceed the int32 range.
dataset_size = tf.constant(5_000_000_000, dtype=tf.int64)
tf.int8 and tf.uint8
Int8 holds values from -128 to 127. Uint8 holds values from 0 to 255. Image pixel values fit perfectly in uint8, making it the standard format for image loading. Quantized models use int8 for weights and activations to reduce model size by 4× compared to float32.
# Raw image pixels — uint8 is natural and memory-efficient pixel_values = tf.constant([255, 128, 0, 64], dtype=tf.uint8) # After normalization for model input, convert to float32 normalized = tf.cast(pixel_values, tf.float32) / 255.0 print(normalized) # [1.0, 0.502, 0.0, 0.251]
tf.int16 and tf.uint16
Used for audio waveforms (16-bit PCM audio is standard) and medical imaging formats where pixel values exceed 255.
Boolean Type
tf.bool
Holds True or False values. Appears in masks, conditional operations, and attention masks in transformer models.
mask = tf.constant([True, False, True, True]) # Boolean indexing with tf.boolean_mask values = tf.constant([10, 20, 30, 40]) filtered = tf.boolean_mask(values, mask) print(filtered) # [10, 30, 40]
String Type
tf.string
Stores byte strings. Used primarily in NLP pipelines for storing raw text before tokenization. TensorFlow's text processing layers accept tf.string tensors.
sentences = tf.constant(['Hello world', 'TensorFlow is great']) print(sentences.dtype) # tf.string # String operations lengths = tf.strings.length(sentences) print(lengths) # [11, 22]
Data Type Conversion with tf.cast
Convert any tensor to a different dtype using tf.cast. This is one of the most frequently used operations in data preprocessing pipelines.
# Integer labels → float32 for loss calculation labels_int = tf.constant([0, 1, 2, 0]) labels_float = tf.cast(labels_int, tf.float32) # uint8 images → float32 normalized inputs image_uint8 = tf.constant([[[100, 150, 200]]], dtype=tf.uint8) image_float = tf.cast(image_uint8, tf.float32) / 255.0 # float32 predictions → bool (thresholding) probs = tf.constant([0.8, 0.3, 0.9, 0.45]) predicted_positives = tf.cast(probs > 0.5, tf.bool) print(predicted_positives) # [True, False, True, False]
Mixed Precision Training
Mixed precision training uses float16 for forward and backward passes (fast) while keeping a float32 copy of the weights for the optimizer update step (accurate). This technique speeds up training by 2–3× on compatible GPUs.
import tensorflow as tf
# Enable mixed precision globally
tf.keras.mixed_precision.set_global_policy('mixed_float16')
# Build your model normally — Keras handles the dtype internally
model = tf.keras.Sequential([
tf.keras.layers.Dense(512, activation='relu', input_shape=(784,)),
tf.keras.layers.Dense(256, activation='relu'),
# Output layer uses float32 for numerical stability
tf.keras.layers.Dense(10, activation='softmax', dtype='float32')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
Dtype Mismatch Errors
Dtype mismatches are a common source of errors. TensorFlow does not automatically convert dtypes between tensors — you must cast explicitly.
a = tf.constant([1.0], dtype=tf.float32) b = tf.constant([2.0], dtype=tf.float64) # This raises an error: # c = a + b → InvalidArgumentError: cannot compute Add with dtypes float32 and float64 # Fix: cast one to match the other c = a + tf.cast(b, tf.float32) # Works
Dtype Quick Reference
Dtype Bits Typical Use ────────────────────────────────────────────────────────────────── tf.float32 32 Model weights, activations (default) tf.float16 16 Fast GPU training (mixed precision) tf.bfloat16 16 TPU training, overflow-resistant tf.float64 64 Scientific computing (rarely in ML) tf.int32 32 Labels, indices, integer counts tf.int64 64 Very large indices or counts tf.int8 8 Quantized model weights tf.uint8 8 Raw image pixels (0–255) tf.uint16 16 16-bit audio, medical images tf.bool 1 Masks, conditions, flags tf.string var Raw text data ──────────────────────────────────────────────────────────────────
Choosing the right dtype makes your models faster, smaller, and more numerically stable. The next topic covers tensor shapes and ranks — how to read, manipulate, and verify the dimensional structure of tensors, which is the key skill for debugging shape mismatch errors in complex models.