The systems that fail at scale almost never fail because of a single bad decision. They fail because of a dozen reasonable decisions made in sequence, none of which seemed problematic at the time, that compound into fragility once load increases by 10×.
This is an article about designing systems with the right load assumptions and failure tolerance from the start — without over-engineering prematurely.
The Core Mental Model: Design for the Failure, Not the Happy Path
Amateur system design focuses on what should happen. Professional system design focuses on what will go wrong.
For every component in your system, ask:
- What happens when this is slow?
- What happens when this is unavailable?
- What happens when this returns incorrect data?
- What happens when this is called 100× more than expected?
If the answer to any of these is "the whole system breaks," you have a fragility to address.
The Five Scalability Killers
1. Synchronous coupling between services
When service A calls service B synchronously and waits for a response, A's latency is now at least as high as B's. More critically: if B is slow or down, A degrades or fails.
The fix is not always async messaging — sometimes synchronous calls are correct. But every synchronous dependency is a failure coupling. Map them explicitly and decide consciously which ones are acceptable.
For write-heavy operations that don't need an immediate response — user event tracking, notification sending, audit logging — async queues (SQS, Kafka, RabbitMQ) decouple the systems. The caller fires and continues; the downstream processes when it can.
2. The database as a bottleneck
At small scale, a single relational database handles everything comfortably. At scale, three things happen:
- Read volume overwhelms the primary
- Write contention on hot rows becomes a bottleneck
- Schema changes become operational events
The standard progression: read replicas for read-heavy workloads → connection pooling (PgBouncer, RDS Proxy) → query optimisation and indexing → sharding for the truly large (rare, and expensive to implement)
For most applications, read replicas and a well-tuned query layer handle 10×–50× growth. Sharding is a last resort.
The mistake: adding caching (Redis) without fixing the underlying query problem. Cache is a band-aid on a slow query, not a substitute for fixing it.
3. State in the wrong place
Application servers that hold state (user sessions, in-memory caches, ongoing computations) cannot be horizontally scaled without session affinity hacks. When a deployment happens or a pod restarts, state is lost.
Design application servers to be stateless. Sessions in Redis. Caches in a distributed cache layer. Any state that must survive restarts lives outside the application process.
This is not just a scalability decision — it's a reliability decision. Stateless services can be replaced, restarted, and scaled without losing user data.
4. Missing circuit breakers
A circuit breaker is a pattern where a service stops attempting to call a failing downstream dependency after a threshold of failures, returns a fallback instead, and periodically tests if the dependency has recovered.
Without circuit breakers, a slow downstream service causes request queues to build up, threads (or async workers) to be consumed waiting, and eventually the calling service to degrade under the backpressure.
With circuit breakers, the calling service detects the failure fast, returns a degraded but functional response, and recovers cleanly when the downstream is healthy again.
Libraries like Resilience4j (Java), Polly (.NET), and cockatiel (Node.js) implement this pattern. Building it yourself is not complex — the pattern is the insight.
5. Logging and observability as an afterthought
You cannot debug a production incident in a system you cannot observe. The cost of adding observability after the fact is 5× the cost of building it in from the start.
Before your first production deployment:
- Structured logs with request IDs that trace across services
- Metrics: request rate, error rate, latency percentiles (p50, p95, p99), queue depths
- Distributed tracing (OpenTelemetry + Jaeger or Tempo)
- Alerting on error rate spikes and latency degradation, not just uptime
A Practical Scaling Checklist
Before you scale a system, run through these:
| Question | If No: |
|---|---|
| Are application servers stateless? | Move session/state to Redis |
| Are all synchronous dependencies necessary? | Move non-critical writes to async queues |
| Do you have read replicas for read-heavy tables? | Add replicas, route reads |
| Do you have circuit breakers on external calls? | Add with a library |
| Do you have p95/p99 latency tracking? | Instrument before you need it |
| Do you know which database queries run on every request? | Add query logging, review |
When Not to Over-Engineer
Everything above has a cost. Most early-stage systems do not need all of it.
The rule I use: design for 10× current load, not 100×. Build the architecture that handles an order of magnitude more traffic with no code changes (stateless servers, read replicas, basic caching). The infrastructure for 100× comes later, when you have the problem and can justify the engineering investment.
The failure mode to avoid is building a distributed system with Kafka, service mesh, and sharded databases for a product with 50 users. That engineering time has high opportunity cost and adds operational complexity that slows the product down.
If you're designing a system and want a second opinion on the architecture before you build, reach out. Getting the foundation right is significantly cheaper than refactoring under load.