Your checkout calls a payment API, the payment API slows down, and within minutes every PHP-FPM worker is stuck waiting, Nginx starts returning 504s and the whole ERP is down, even though only one dependency failed. That is the core challenge of distributed system design: once your application talks to other services over a network (databases, Redis, queues, third-party APIs, microservices), partial failure is normal. This guide explains why one slow service takes everything down and shows the practical patterns, with Laravel, Node.js, Nginx, Redis and Docker examples, for building fault tolerant systems that degrade gracefully instead of collapsing.
The Problem: One Failure Brings Down Everything
A single server either works or it does not. A distributed architecture has many more states: a service can be slow, half-available, returning errors only for some requests, or reachable from one node but not another. Code written as if every call succeeds instantly turns these partial failures into full outages.
Symptoms of a Fragile Distributed Architecture
# Workers exhausted while waiting on a slow upstream
WARNING: [pool www] server reached pm.max_children setting (40), consider raising it
# Nginx gives up first
upstream timed out (110: Connection timed out) while reading response header from upstream
# Laravel / Guzzle
cURL error 28: Operation timed out after 30001 milliseconds
# Node.js
Error: connect ETIMEDOUT 10.0.0.42:443
Error: socket hang up
# Duplicate side effects after retries
SQLSTATE[23505]: Unique violation: duplicate key value violates unique constraint "payments_reference_unique"- A slow third-party API causes errors on pages that do not even use it.
- Customers are charged twice or receive two emails after a timeout and retry.
- When a dependency recovers, a wave of retries knocks it down again.
- Restarting one container fixes things temporarily, then the problem returns.
Root Cause: Why Distributed Systems Fail
Network calls are not function calls
Every remote call can hang, fail, or succeed without you hearing back. Many HTTP clients have very long or unlimited default timeouts (Guzzle defaults to no timeout, for example), so one hung call holds a worker for as long as the upstream holds the connection.
Shared resource exhaustion
Worker pools, database connections and thread pools are finite. If slow calls to one dependency hold them, healthy requests have nothing left. This is cascading failure.
Unsafe retries
Retrying immediately and indefinitely multiplies load on a struggling service (a retry storm). Retrying a non-idempotent operation, such as "charge card", creates duplicates when the first attempt actually succeeded.
Dual writes
Writing to the database and then publishing to a queue in two separate steps means a crash in between leaves the systems inconsistent.

Step-by-Step Solution: Designing Fault Tolerant Systems
Step 1: Put a timeout on every network call
Set connect and total timeouts shorter than the timeout of whatever calls you, so your service fails fast instead of piling up waiting requests.
// Laravel HTTP client
$response = Http::connectTimeout(2)
->timeout(5)
->get('https://payments.example.com/v1/status');
// Node.js 18+ fetch
const res = await fetch(url, { signal: AbortSignal.timeout(5000) });# PostgreSQL: cap runaway queries for the app role
ALTER ROLE app_user SET statement_timeout = '15s';
# Nginx: fail before clients give up
proxy_connect_timeout 3s;
proxy_read_timeout 30s;If Nginx is already timing out, see how to fix Nginx 504 Gateway Timeout.
Step 2: Retry with exponential backoff and jitter
Retry only transient errors (timeouts, 502, 503, 429), cap the attempts, and add random jitter so clients do not retry in sync.
// Laravel: 3 attempts, backoff in ms, only on connection errors or 5xx
$response = Http::timeout(5)
->retry([200, 600, 1500], function ($exception, $request) {
return $exception instanceof \Illuminate\Http\Client\ConnectionException
|| optional($exception->response)->serverError();
})
->post($url, $payload);
// Node.js
async function withRetry(fn, attempts = 3) {
for (let i = 0; i < attempts; i++) {
try { return await fn(); }
catch (err) {
if (i === attempts - 1) throw err;
const base = 200 * 2 ** i;
await new Promise(r => setTimeout(r, base + Math.random() * base));
}
}
}Step 3: Make operations idempotent
Retries and queue redelivery mean the same request can arrive twice. Give each operation a unique key and let the database reject duplicates.
-- PostgreSQL
CREATE TABLE processed_requests (
idempotency_key text PRIMARY KEY,
response jsonb,
created_at timestamptz DEFAULT now()
);
INSERT INTO processed_requests (idempotency_key)
VALUES ('order-9812-charge')
ON CONFLICT (idempotency_key) DO NOTHING; -- 0 rows = already handled// Laravel queued job: skip if this order was already charged
public function handle(): void
{
$inserted = DB::table('processed_requests')
->insertOrIgnore(['idempotency_key' => "order-{$this->orderId}-charge"]);
if ($inserted === 0) return;
// ... call payment provider with the same key as Idempotency-Key header
}Step 4: Add a circuit breaker
After repeated failures, stop calling the dependency for a cool-down period and return a fallback immediately. This protects your workers and gives the dependency room to recover.
// Simple Redis-backed circuit breaker in Laravel
function callPayments(callable $fn) {
if (Cache::get('cb:payments:open')) {
throw new RuntimeException('Payments temporarily unavailable');
}
try {
$result = $fn();
Cache::forget('cb:payments:failures');
return $result;
} catch (Throwable $e) {
$fails = Cache::increment('cb:payments:failures');
if ($fails >= 5) {
Cache::put('cb:payments:open', true, now()->addSeconds(30));
Cache::forget('cb:payments:failures');
}
throw $e;
}
}In Node.js, the opossum library provides the same pattern.
Step 5: Move slow work to queues with an outbox
Do not make users wait on emails, PDFs or external syncs. Write the business change and an outbox row in one transaction, then a worker publishes it. Our guide on Laravel queues with Redis covers worker setup.
DB::transaction(function () use ($order) {
$order->update(['status' => 'paid']);
DB::table('outbox')->insert([
'event' => 'order.paid',
'payload' => json_encode(['id' => $order->id]),
]);
});
// a scheduled worker reads unsent outbox rows, dispatches jobs, marks them sentStep 6: Health checks and redundancy
Run at least two instances of each stateless service and let the orchestrator restart or remove unhealthy ones.
# docker-compose.yml
services:
api:
image: myapp/api:latest
deploy:
replicas: 2
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
interval: 10s
timeout: 3s
retries: 3Keep /health cheap. A liveness check should not call every dependency, or one slow database makes all instances look dead at once. For capacity planning, read scalable system design for millions of users.
Step 7: Degrade gracefully
Decide in advance what users see when a dependency is down: cached prices, a "payment pending" status, or a disabled recommendations widget, rather than a 500 for the whole page.
Code and Commands Quick Reference
docker compose ps # health status per container
docker inspect --format '{{json .State.Health}}' api-1
redis-cli get cb:payments:open # is the circuit open?
psql -c "SELECT count(*) FROM outbox WHERE sent_at IS NULL;" # backlog
php artisan queue:failed # failed jobs to inspect
tc qdisc add dev eth0 root netem delay 3000ms # test env only: inject latencyVerification
- Inject latency or stop a dependency in staging; the rest of the app should keep serving pages.
- Confirm requests to the failing service return within your timeout, not after 30+ seconds.
- Send the same request twice with one idempotency key; only one side effect should happen.
- Watch the circuit open after repeated failures and close again once the dependency recovers.
- Kill one container; the health check should mark it unhealthy and traffic should shift to the other replica.
Prevention
- Review every new integration for timeout, retry, idempotency and fallback behavior.
- Use separate worker pools or queues for slow dependencies so they cannot starve everything else.
- Alert on error rate, p95 latency, queue backlog and open circuits.
- Run small failure drills regularly, starting in staging.
- Document what each service does when its dependencies are unavailable.
FAQs
What is distributed system design?
It is designing software that runs across multiple processes or machines communicating over a network, with explicit handling for latency, partial failure and data consistency.
What makes fault tolerant systems different?
They assume components will fail and contain the damage with timeouts, retries, idempotency, circuit breakers, redundancy and graceful degradation.
Do I need microservices for a distributed architecture?
No. A Laravel monolith that uses PostgreSQL, Redis, a queue and external APIs is already distributed and needs the same patterns.
Should I retry every failed request?
No. Retry only transient failures, cap attempts, use backoff with jitter, and only retry operations that are idempotent.
What is the difference between a timeout and a circuit breaker?
A timeout limits how long one call waits. A circuit breaker stops making calls at all for a while after repeated failures.




Comments
Join the discussion. Replies load on demand, up to 5 levels.
0 threads
No comments yet
Be the first to start the conversation.