DE Distributed Computing Concepts
Distributed computing divides a large computational problem across multiple machines working together as a single system. Every modern big data tool — Spark, Kafka, Flink, Cassandra — builds on distributed computing principles. Understanding these concepts explains why distributed systems behave the way they do and why certain trade-offs are unavoidable.
Why Distribution Is Necessary
A single machine has fixed limits: a maximum amount of RAM, a fixed number of CPU cores, and a disk with a maximum read speed. When a problem exceeds these limits — a dataset too large for one machine's RAM, a computation too slow for one CPU — the only scalable solution is to spread the work across multiple machines.
The Restaurant Kitchen Analogy
A single chef can prepare 50 meals per hour. When 500 diners arrive, one chef cannot scale up. The restaurant hires 10 chefs, divides the menu among them, and each chef prepares their portion simultaneously. But coordination becomes critical — chefs share one stove, one pantry, and must communicate to avoid making the same dish twice. Distributed computing faces identical coordination challenges: work division, resource sharing, and communication overhead.
Key Concepts in Distributed Systems
Horizontal vs Vertical Scaling
Vertical Scaling (Scale Up): Add more power to ONE machine - More RAM, more CPU cores, bigger disk - Limits: physical hardware ceiling - Example: Upgrade server from 64GB to 512GB RAM Horizontal Scaling (Scale Out): Add MORE machines to share the load - Each machine handles a portion of the data/requests - Limits: coordination complexity grows with node count - Example: Add 10 more servers to a Spark cluster
Distributed systems favor horizontal scaling because it removes the physical ceiling of a single machine and allows incremental capacity growth by adding nodes.
Partitioning
Partitioning divides data into chunks that distribute across nodes. Each node owns and processes only its partitions. Good partitioning balances the work evenly — if one node gets 90% of the data (a "hot partition"), it becomes a bottleneck while others sit idle.
Dataset: 1 billion order records Hash Partitioning (by customer_id): Node 1: customers with hash(id) % 4 = 0 --> 250M records Node 2: customers with hash(id) % 4 = 1 --> 250M records Node 3: customers with hash(id) % 4 = 2 --> 250M records Node 4: customers with hash(id) % 4 = 3 --> 250M records Range Partitioning (by date): Node 1: Jan-Mar 2024 Node 2: Apr-Jun 2024 Node 3: Jul-Sep 2024 Node 4: Oct-Dec 2024
Replication
Replication stores copies of data on multiple nodes. If one node fails, the data remains accessible from its replica. Kafka, HDFS, and Cassandra all replicate data across nodes by default. Replication trades storage space for availability and fault tolerance.
Fault Tolerance
Machines in a distributed cluster fail regularly. A 1,000-node cluster statistically has several hardware failures per week. Fault-tolerant systems detect failures and continue operating without human intervention. Spark recomputes lost partitions using lineage information. Kafka replays events from a replicated log. HDFS reads data from a replica when the primary node is unavailable.
The CAP Theorem
The CAP theorem is one of the most important concepts in distributed systems. It states that a distributed system can guarantee only two of the following three properties simultaneously — never all three.
Consistency (C)
Every read receives the most recent write or an error. All nodes see the same data at the same time. No node serves stale data.
Availability (A)
Every request receives a response — not an error — even if the data may not be the most recent version. The system always responds.
Partition Tolerance (P)
The system continues operating even when network communication between some nodes fails (a network partition). In real distributed systems, network partitions happen. Partition tolerance is non-negotiable — every distributed system must handle it.
CAP Trade-off (P is always required in practice): CP Systems (Consistency + Partition Tolerance): - Sacrifice availability during partition - Examples: HBase, ZooKeeper - Use when: Financial transactions, inventory counts AP Systems (Availability + Partition Tolerance): - Sacrifice consistency; may serve stale data - Examples: Cassandra, DynamoDB, CouchDB - Use when: Social feeds, shopping carts, DNS
Consistency Models
Strong Consistency
After a write completes, all subsequent reads return the new value from any node. This is what traditional relational databases with ACID transactions guarantee. It requires coordination between nodes, which adds latency.
Eventual Consistency
After a write, replicas update over a short period. During the propagation window, different nodes may return different values. Eventually all replicas converge to the same state. Most NoSQL databases and distributed caches use eventual consistency for speed.
Shuffling in Distributed Processing
Shuffling is the most expensive operation in distributed data processing. A shuffle redistributes data across all nodes — necessary for operations like groupBy, join, and sort where related records must land on the same node for processing.
Before Shuffle (data partitioned by order_id): Node 1: Orders from customers C001, C005, C009 Node 2: Orders from customers C002, C006, C010 Node 3: Orders from customers C003, C007, C011 Operation: GROUP BY customer_id (all orders for one customer must be on one node) Shuffle (data moves across the network): Node 1: All orders for C001, C002, C003 Node 2: All orders for C004, C005, C006 Node 3: All orders for C007, C008, C009
Network transfer during a shuffle is orders of magnitude slower than in-memory processing. Data engineers minimize shuffles by filtering data before grouping operations and choosing partition keys that align with frequent join and group keys.
Leader Election
Distributed systems often need one node to act as coordinator — the master, the primary, the controller. If that node fails, the cluster must automatically elect a new leader without human intervention. ZooKeeper, etcd, and Raft-based consensus algorithms solve this problem. Kafka uses a controller node elected from the cluster brokers to manage partition leadership.
Idempotency in Distributed Systems
In a distributed system, a message or operation might be delivered or executed more than once due to retries after partial failures. An idempotent operation produces the same result whether executed once or many times. Data engineers design pipelines to be idempotent — writing to a destination in a way that duplicate writes do not produce duplicate records.
Summary
Distributed computing spreads data and computation across multiple machines to overcome single-machine limitations. Core concepts include horizontal scaling, partitioning, replication, and fault tolerance. The CAP theorem describes the fundamental trade-off between consistency and availability when network partitions occur. Shuffling is the key performance bottleneck in distributed processing. Data engineers who understand these principles design systems that scale reliably and behave predictably under failure.