DE What Is Big Data
The term "big data" gets used loosely, but in data engineering it refers to something specific: datasets so large, fast-moving, or varied in format that traditional tools like a single relational database or a Python script cannot process them efficiently. Big data requires a fundamentally different architectural approach.
The Classic Definition: The Five V's
Data engineers traditionally describe big data using five characteristics called the Five V's.
Volume
The sheer amount of data. A traditional database handles gigabytes or a few terabytes. Big data systems handle terabytes to petabytes — a petabyte is one million gigabytes. A single large social media platform generates several petabytes of data every day.
Velocity
The speed at which data arrives. Stock exchange platforms process millions of trades per second. IoT sensor networks stream millions of readings per minute. Traditional batch systems cannot keep up — the data piles up faster than it can be processed.
Variety
The diversity of formats. Big data environments simultaneously handle structured tables, semi-structured JSON and XML, unstructured text, images, audio, video, and log files. A single relational database cannot store or process all of these natively.
Veracity
The reliability and accuracy of the data. Large datasets collected from many heterogeneous sources contain noise, errors, and inconsistencies. Big data systems must handle uncertain and incomplete data gracefully while still producing trustworthy results.
Value
The useful insight extracted from the data. Raw big data by itself has no inherent value. The value comes from processing, analyzing, and applying it to decisions — better recommendations, faster fraud detection, more efficient logistics, personalized medicine.
A Concrete Scale Example
Scale Comparison: Small Data: A shop with 1,000 transactions/day --> SQLite or MySQL handles it fine Medium Data: A retailer with 1M transactions/day --> PostgreSQL with good indexing Large Data: An e-commerce giant with 100M transactions/day --> Needs distributed systems Big Data: A payment network with 10B transactions/day --> Needs Spark, Kafka, NoSQL
Why Traditional Tools Break
A traditional database runs on one machine. Disk storage, RAM, and CPU all have hard limits. When data outgrows one machine's capacity, the only options are to buy a bigger machine (vertical scaling) or to redesign the system to work across many machines (horizontal scaling). Vertical scaling hits physical limits. Big data tools are built from the ground up for horizontal scaling — adding more machines to a cluster to share the load.
The Warehouse Distribution Analogy
A single warehouse worker can pick and pack a certain number of orders per day. When orders triple, one worker cannot triple their speed. The solution is to hire more workers and divide the work among them. Big data distributed computing does exactly this: divide the dataset across many machines (workers), process each portion in parallel, and combine the results. Ten machines working in parallel finish the job in one-tenth the time.
Where Big Data Comes From
Source | Data Type | Volume Example ---------------------|--------------------|----------------------- Social media | Text, images, video| Facebook: 4PB/day E-commerce clicks | Event logs | Amazon: billions/day IoT sensors | Time-series | Smart factory: TB/hour Mobile apps | Usage events | Google Play: PB/day Financial trading | Transactions | NYSE: 5B messages/day Genomics research | DNA sequences | Single genome: 200GB Satellite imagery | Images | Planet: 3M images/day
The Big Data Technology Stack
Specific technologies handle the different dimensions of big data. Together they form what engineers call the "big data stack."
Distributed Storage
HDFS (Hadoop Distributed File System) and cloud object storage (Amazon S3, Google Cloud Storage) store massive datasets across many servers. Files split into blocks distributed across the cluster. No single disk holds a complete file — the distributed system manages where each piece lives.
Distributed Processing
Apache Spark processes large datasets across a cluster of machines in parallel. What a single machine would take hours to compute, a 100-node Spark cluster completes in minutes by dividing the work.
Real-Time Ingestion
Apache Kafka handles high-velocity data streams, ingesting millions of events per second and making them available to downstream processing systems immediately.
Distributed Query Engines
Presto (now Trino) and Apache Hive run SQL queries across distributed storage, making big data accessible to analysts who know SQL without writing specialized code.
Is Your Data Actually "Big"?
Not every data problem is a big data problem. Many companies apply big data technology to datasets that a well-tuned PostgreSQL database would handle perfectly. Big data tools add complexity, cost, and operational burden. Apply them only when the data volume, velocity, or variety genuinely exceeds what traditional tools can handle.
You need big data tools when: - A single database server cannot store all the data - Queries take hours even with full optimization - Data arrives faster than one machine can ingest - Data format variety prevents a single tool from handling it You do NOT need big data tools when: - Total data fits on one server (< 1TB for most cases) - Daily batch processing completes in acceptable time - A managed cloud warehouse handles the query load
Summary
Big data describes datasets characterized by high volume, velocity, variety, veracity, and value — dimensions that exceed traditional single-machine tool capabilities. Distributed storage and processing systems like HDFS, Apache Spark, and Kafka handle these workloads by spreading them across clusters of machines working in parallel. Understanding what big data is — and when a problem genuinely requires big data tools — prevents over-engineering and keeps data systems appropriately sized for their actual workload.