SRE Load Testing
A service that works for ten users may fail for ten thousand. Load testing answers a simple question before real customers ask it: how much demand can the system handle? Teams that test early avoid surprises on launch day.
What Is Load Testing?
Load testing sends a controlled amount of artificial traffic to a service and measures how it behaves. Picture a bridge inspector who drives heavier and heavier trucks across a new bridge to confirm that it holds. The inspector learns the safe limit before the public uses the bridge. Engineers apply the same idea to servers.
Types of Performance Tests
| Test Type | Goal | Pattern |
|---|---|---|
| Load test | Confirm normal and expected peak demand works | Climb to the expected peak and hold |
| Stress test | Find the breaking point | Keep increasing until failure |
| Spike test | Check sudden jumps in demand | Jump quickly from low to high |
| Soak test | Find slow leaks such as memory growth | Hold moderate load for many hours |
Shapes of Traffic
Load Test Stress Test Spike Test Soak Test users users users users │ ______ │ / │ /\ │ __________ │ / │ / │ / \ │/ │ / │ / │___/ \___ │ └────────── time └────────── time └────────── time └────────── time
Planning a Test
- Define the goal: such as handling 2,000 requests per second with p95 latency under 500 milliseconds.
- Choose real journeys: mimic what users do, such as browse, search, and pay.
- Prepare data: create realistic test products and accounts.
- Pick a safe environment: use a copy of production or a quiet window with approval.
- Set stop rules: halt the test automatically when errors pass a safe limit.
What to Watch During a Test
Combine the golden signals with resource charts. Watch latency percentiles, error rate, requests per second, CPU, memory, disk, database connections, and queue length. The first chart that bends usually points to the bottleneck.
Example: Festival Sale
An online store expects 5,000 shoppers at once during a festival sale. The team builds a test with a 30-minute ramp from 500 to 5,000 virtual shoppers. At 3,200 shoppers, p95 latency climbs from 400 milliseconds to 3 seconds. The database CPU sits at 95 percent. The team adds a read replica, caches product pages, and reruns the test. This time the system holds 6,000 shoppers with p95 latency at 450 milliseconds. The sale launches with confidence.
Reading Results
Find the Knee
Every system has a knee, the point where latency starts rising sharply. Operate well below it. A safe target keeps normal peaks at about 60 to 70 percent of the knee.
Compare Against Targets
Match results with the service level objective. A test that meets the objective at expected peak passes. A test that fails shows how much capacity to add.
Repeat After Changes
Run the same test after major releases. A steady baseline reveals performance regressions early.
Common Tools
Popular open-source options include k6, Locust, and Apache JMeter. All three generate virtual users, send scripted requests, and report latency and error numbers. Choose the one that matches your team's language skills.
Safe Practices
- Tell dependent teams and vendors before a large test.
- Never send test emails or payments to real people.
- Start with small load and double it in steps.
- Keep a person ready to stop the test instantly.
Common Mistakes
- Testing only the home page.
- Using a tiny test database that behaves faster than the real one.
- Running the load generator on a weak machine that becomes the bottleneck.
- Skipping the soak test and missing memory leaks.
Key Takeaways
Load testing replaces guessing with evidence. Define a clear goal, use realistic journeys, watch the signals, and repeat the test after every important change.