System Design Fault Tolerance

Fault tolerance is the ability of a system to keep working when parts of it fail. Hardware breaks, networks drop, and software crashes. A fault tolerant design expects these events and hides them from users.

Faults, Errors, and Failures

A fault is the root problem, such as a broken disk. An error is the wrong behavior that the fault triggers. A failure happens when users see the damage. Good design stops a fault before it grows into a failure.

Layman example

A car carries a spare tire. A flat tire is a fault. The driver swaps in the spare and reaches the destination. The trip never fails.

Redundancy

Redundancy means keeping extra copies of important parts. Extra servers, extra databases, and extra network links all count. When one copy dies, another copy takes over.

            +----------------+
 Users ---> | Load Balancer  |
            +----------------+
              /     |      \
        [Server1] [Server2] [Server3]
           OK       FAILED     OK
 Traffic flows only to healthy servers.

Failover

Failover switches work from a broken component to a healthy one. Two styles exist.

• Active-passive: one server works while a standby waits. The standby wakes up after a failure.
• Active-active: all servers share the work. Survivors absorb the load of any failed server.

Health Checks

A health check is a small test that asks "Are you alive?" every few seconds. A server that misses several checks leaves the rotation. The load balancer stops sending it traffic until it recovers.

Timeouts and Retries

A slow service can freeze every caller that waits for it. A timeout sets a time limit on each call. A retry tries again after a failure. Retries must follow careful rules.

Exponential Backoff

Backoff doubles the wait time after each failed try. The waits might run 1 second, 2 seconds, 4 seconds, and 8 seconds. Short waits pile pressure on a struggling service. Growing waits give it room to recover. Random delay, called jitter, prevents all clients from retrying at the same moment.

Circuit Breaker

A circuit breaker works like the fuse in a home. When a service keeps failing, the breaker opens and blocks further calls for a short period. The caller gets a quick answer instead of a long wait.

 CLOSED  --(many failures)-->  OPEN
    ^                           |
    |                   (wait period ends)
    |                           v
    +--(test call works)--  HALF-OPEN
        (test call fails: back to OPEN)

• Closed: calls flow normally.
• Open: calls stop and return an error or fallback at once.
• Half-open: a few test calls check whether the service recovered.

Graceful Degradation

A system under stress can drop non-essential features and protect the core. A video app might lower picture quality instead of stopping playback. A shopping site might hide personalized suggestions and keep checkout alive.

Bulkheads

Ships use sealed compartments called bulkheads. Water in one compartment cannot sink the whole ship. Software borrows the idea. Each service gets its own pool of threads or connections. A failing feature drains only its own pool.

Data Protection

• Replicate data across several servers.
• Store copies in different data centers or regions.
• Take regular backups and test the restore process.
• Write operations so they can run twice without harm. Engineers call this idempotency.

Single Points of Failure

A single point of failure is any part whose loss stops the whole system. Designers hunt for these parts. A lone database, a single load balancer, or one power source all qualify. Each one needs a backup.

Chaos Testing

Some companies break their own systems on purpose during working hours. Engineers switch off servers and watch how the system reacts. Weak spots appear before a real outage exposes them.

Summary Table

Technique	Purpose
Redundancy	Provide spare parts
Failover	Switch to healthy parts
Health checks	Detect broken parts
Retry with backoff	Recover from short glitches
Circuit breaker	Stop repeated failing calls
Graceful degradation	Protect core features