DE Batch Processing

The majority of data engineering pipelines in production today use batch processing. It is the backbone of most data warehouses, reporting systems, and scheduled analytics workflows. Understanding batch processing — how it works, why it exists, and when it fits — gives data engineers a foundation for designing reliable data systems.

What Is Batch Processing

Batch processing collects data over a defined period and processes it all together at a scheduled time. Instead of handling each record the moment it arrives, the system waits, accumulates records, and then works through them as a group — a "batch." The schedule might be hourly, daily, weekly, or monthly depending on the business need.

The Post Office Analogy

A post office does not deliver each letter the second it arrives at the sorting facility. Mail accumulates throughout the day. At the end of the shift, sorters process the entire pile together — grouping by destination, sorting by street, loading delivery vans. The next morning, delivery happens for everything accumulated during the previous day. Batch processing works exactly this way: collect, then process the collected set together.

How Batch Pipelines Work

Trigger: Schedule (e.g., every day at 2:00 AM)

Step 1: Extract
  Collect all new records created since the last batch run
  Example: SELECT * FROM orders WHERE created_date = '2024-05-15'

Step 2: Process
  Apply transformation logic to the full batch
  Filter invalid records
  Join with reference tables
  Calculate derived fields

Step 3: Load
  Write processed records to the destination in bulk

Step 4: Log & Alert
  Record how many records processed
  Send alert if any step failed

Common Batch Processing Tools

Tool              | Strengths
------------------|---------------------------------------------
Apache Spark      | Distributed; handles terabytes efficiently
Python + Pandas   | Simple; good for smaller datasets
dbt               | SQL-based transformations in the warehouse
AWS Glue          | Serverless Spark on AWS; managed service
Google Dataflow   | Serverless; unified batch and stream API
Hadoop MapReduce  | Older; still used in legacy systems

Batch Processing in the Real World

Payroll Processing

A company with 10,000 employees calculates paychecks once per month. The system collects attendance records, sales commissions, leaves, and deductions accumulated over the month, then processes all 10,000 records together to generate payslips. Running this calculation in real time every second would serve no purpose — employees only need the result on payday.

Daily Sales Reports

A retail chain runs a batch job at midnight. It extracts all sales transactions from the previous day, aggregates revenue by store, product category, and region, and loads the results into the reporting warehouse. Executives find an updated report every morning.

Data Warehouse Loading

Most data warehouse pipelines run in batch. Source system snapshots extract nightly, transform through dbt or Spark jobs, and load into warehouse tables ready for analyst queries each business day morning.

Advantages of Batch Processing

Simplicity

Batch pipelines are significantly easier to build, test, and debug than streaming pipelines. The process runs at a known time on a known dataset. When something fails, the engineer reruns the job for the same date range and gets the same result.

Efficiency

Processing records in bulk allows systems to optimize I/O, use bulk insert operations, and apply vectorized computations. Reading and writing millions of records at once is far more efficient than handling them one at a time.

Lower Cost

Batch jobs run at off-peak hours when compute resources cost less. A warehouse query that runs at 2 AM uses available capacity efficiently without competing with interactive analyst queries during business hours.

Limitations of Batch Processing

Data Latency

Batch processing always introduces a delay between when data is created and when it appears in the destination. A nightly batch means data is always at least several hours old. For most business reporting, this is acceptable. For fraud detection or real-time recommendations, it is not.

Large Failure Impact

If a batch job fails, the entire batch of unprocessed records waits until the engineer fixes the problem and reruns the job. A streaming system processes each record independently, so a failure affects only one record, not an entire day's worth of data.

Windowing in Batch Processing

Batch jobs define a time window for the data they process. A daily batch processes records from midnight to midnight. A weekly batch processes the last seven days. Choosing the right window matters — a window too narrow might miss records that arrived late from a slow source system. Engineers add a small overlap called a "lookback window" to catch late-arriving records.

Normal daily window:
  Process records where created_at >= '2024-05-15 00:00'
                    AND created_at <  '2024-05-16 00:00'

With lookback (catches late arrivals):
  Process records where created_at >= '2024-05-14 22:00'
                    AND created_at <  '2024-05-16 00:00'
  Then deduplicate on order_id to remove records already processed

Scheduling Batch Jobs

Orchestration tools manage batch job schedules and dependencies. If a downstream job depends on an upstream job completing successfully, the orchestrator waits before triggering the next step. Apache Airflow expresses this as a DAG — a Directed Acyclic Graph — where each node is a task and each arrow defines a dependency.

Summary

Batch processing accumulates data over time and processes it in scheduled bulk runs. It is simpler, more efficient, and less expensive than streaming at the cost of data latency. Most data warehouse pipelines, reporting jobs, and ETL workflows use batch processing. Knowing how to design reliable, idempotent batch jobs is one of the most practical skills in data engineering.