TensorFlow Loading CSV Data

CSV (Comma-Separated Values) files are among the most common data formats in machine learning. Sales records, medical data, weather readings, and financial datasets all arrive as CSV files. TensorFlow provides multiple ways to load CSV data efficiently, from simple pandas-based approaches for small files to streaming tf.data pipelines for large datasets that do not fit in memory.

Method 1 — Pandas Into NumPy (Small Datasets)

For CSV files under a few hundred megabytes, load with pandas and convert to NumPy arrays. This is the fastest approach to get started.

import pandas as pd
import numpy as np
import tensorflow as tf

# Load CSV
df = pd.read_csv('titanic.csv')
print(df.head())
print(df.shape)   # (rows, columns)

# Separate features and label
target_column = 'survived'
y = df[target_column].values
X = df.drop(columns=[target_column]).values.astype('float32')

# Split
split = int(0.8 * len(X))
x_train, x_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]

# Build and train model normally
model.fit(x_train, y_train, epochs=20)

Handling Missing Values and Categorical Columns

Real CSV files almost always have missing values and non-numeric columns. Handle these before feeding data to TensorFlow.

# Fill missing numeric values with column median
df['age'].fillna(df['age'].median(), inplace=True)

# Fill missing categorical values with mode (most common)
df['cabin'].fillna('Unknown', inplace=True)

# Encode categorical columns as integers
df['sex'] = df['sex'].map({'male': 0, 'female': 1})

# One-hot encode columns with many categories
df = pd.get_dummies(df, columns=['embarked'], drop_first=True)

# Check for remaining NaN values
print(df.isnull().sum())

Method 2 — tf.data CSV Pipeline (Large Datasets)

For large CSV files, stream data from disk using tf.data. The file never fully loads into memory — TensorFlow reads and processes batches on demand.

import tensorflow as tf

# Column names and their types
COLUMN_NAMES = ['age', 'income', 'education', 'hours_per_week', 'income_bracket']
DEFAULTS = [0.0, 0.0, 0.0, 0.0, 0]

def parse_csv_row(row):
    # Decode one CSV line into a dictionary
    fields = tf.io.decode_csv(row, record_defaults=DEFAULTS)
    features = dict(zip(COLUMN_NAMES[:-1], fields[:-1]))
    label = fields[-1]
    return features, label

dataset = (
    tf.data.TextLineDataset('large_data.csv')
    .skip(1)                          # Skip the header row
    .map(parse_csv_row, num_parallel_calls=tf.data.AUTOTUNE)
    .shuffle(10000)
    .batch(64)
    .prefetch(tf.data.AUTOTUNE)
)

Method 3 — make_csv_dataset (Easiest Streaming Approach)

dataset = tf.data.experimental.make_csv_dataset(
    file_pattern='data/*.csv',   # Supports multiple files with wildcard
    batch_size=32,
    column_names=COLUMN_NAMES,
    label_name='income_bracket',
    num_epochs=1,
    ignore_errors=True
)

# Batches return a dict of feature tensors plus a label tensor
for feature_batch, label_batch in dataset.take(1):
    print("Feature keys:", list(feature_batch.keys()))
    print("Batch size:", label_batch.shape)

Normalization: Scaling Feature Values

Neural networks train much faster when input features have similar numerical scales. A feature ranging from 0–100 and another ranging from 0–1,000,000 cause the optimizer to take wildly different gradient steps for each feature.

import tensorflow as tf

# Compute normalization statistics from training data
normalizer = tf.keras.layers.Normalization()
normalizer.adapt(x_train)   # Learn mean and variance from training set

# Add normalization as the first layer in the model
model = tf.keras.Sequential([
    normalizer,
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

# Now x_train feeds in with original values — normalization happens inside the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, epochs=20)

Diagram — CSV to Model Flow

data.csv (on disk)
     │
     ▼
[Read file / pd.read_csv / TextLineDataset]
     │
     ▼
[Parse and clean]
  - Handle missing values
  - Encode categorical columns
  - Split features and label
     │
     ▼
[Normalize features]
  - Zero mean, unit variance
     │
     ▼
[Create tf.data.Dataset]
  - shuffle → batch → prefetch
     │
     ▼
[model.fit(dataset)]

Reading Multiple CSV Files

# Load all CSV files from a folder at once
import glob

file_paths = glob.glob('data/monthly_records/*.csv')
all_dfs = [pd.read_csv(f) for f in file_paths]
combined_df = pd.concat(all_dfs, ignore_index=True)
combined_df = combined_df.sample(frac=1).reset_index(drop=True)  # shuffle rows

Saving Preprocessing to Model for Deployment

When you embed the Normalization layer inside the model, the normalization statistics are saved with the model. At inference time, users pass raw un-normalized values and the model handles scaling internally.

# At inference time — no need to normalize manually
raw_input = [[42, 80000, 14, 40]]   # age, income, education, hours
prediction = model.predict(raw_input)

Loading CSV data cleanly and efficiently sets the stage for all tabular machine learning tasks. The next topic covers image data loading, where TensorFlow provides specialized utilities for reading, decoding, and feeding image files to convolutional neural networks.