Your order service calls a payment API, the payment API slows down, and within minutes every PHP-FPM worker or Node.js request is stuck waiting on it. The order service stops responding, then the API gateway in front of it starts timing out, and one slow dependency has become a full outage. The circuit breaker pattern stops this chain reaction: after repeated failures it stops calling the broken dependency for a while, fails fast with a fallback, and probes carefully before resuming. This guide shows how to implement it in Laravel and Node.js, with the timeouts, metrics and tests that make it work in production.
The Problem: One Slow Service Brings Down Everything
In a microservices or service-oriented backend, each service depends on others: payments, inventory, email, search, third-party APIs. When one of them becomes slow rather than fully down, callers keep sending requests and waiting. Each waiting request holds a worker, a database connection or memory. Eventually the caller runs out of capacity and fails too. This is a cascading failure, and retries without limits make it worse because they multiply load on a service that is already struggling. If you are still deciding how to split services, see monolith vs modular monolith vs microservices; the more network hops you add, the more you need this pattern.
Symptoms of Missing Microservices Resilience
- Response times of a healthy service jump whenever one downstream dependency is slow.
- All PHP-FPM children are busy and Nginx returns
502or504errors. - Node.js event loop stays responsive but open sockets and memory keep climbing.
- Logs fill with the same timeout error thousands of times per minute.
- The failing dependency cannot recover because retries keep hammering it.
Typical log lines:
cURL error 28: Operation timed out after 30001 milliseconds with 0 bytes received
upstream timed out (110: Connection timed out) while reading response header from upstream
WARNING: [pool www] server reached pm.max_children setting (20), consider raising it
Error: connect ETIMEDOUT 10.0.0.45:443
FetchError: request to https://payments.internal/charge failed, reason: socket hang upRoot Cause: Unbounded Waiting and Blind Retries
Cascading failures usually come from a few design gaps:
- No timeouts or very long ones. Guzzle and many HTTP clients wait indefinitely or for 30 seconds or more by default unless configured.
- Retries without backoff or limits. Every failure turns into three or more extra calls.
- No failure memory. Each request rediscovers that the dependency is broken by waiting for a timeout.
- No fallback. The caller has nothing useful to return, so it errors or hangs.
- Shared resource pools. One slow dependency consumes the same workers that serve unrelated endpoints.
A circuit breaker adds failure memory. It wraps calls to a dependency and moves between three states:
- Closed: calls pass through; failures are counted.
- Open: after a failure threshold is reached, calls fail immediately without touching the dependency, for a cooldown period.
- Half-open: after the cooldown, a limited number of trial calls go through. If they succeed, the circuit closes; if they fail, it opens again.

Step-by-Step Solution: How to Implement the Circuit Breaker Pattern
Step 1: Set strict timeouts on every outbound call
A circuit breaker cannot help if a single call can hang for a minute. Set a connect timeout and a total timeout that fit your latency budget, often 1 to 3 seconds for internal services. In Laravel's HTTP client:
$response = Http::connectTimeout(2)
->timeout(3)
->retry(2, 200, throw: false) // max 2 attempts, 200 ms between
->get('https://payments.internal/status');Step 2: Choose thresholds for fault tolerance
Decide per dependency: how many consecutive failures (or what failure rate in a time window) opens the circuit, how long it stays open, and how many trial calls are allowed in half-open. A reasonable start is 5 failures, a 30 second cooldown and 1 trial call. Count timeouts, connection errors and 5xx responses as failures; do not count 4xx validation errors, because those are caller mistakes, not dependency failures.
Step 3: Store circuit state where all workers can see it
PHP-FPM workers do not share memory, so a Laravel circuit breaker must keep state in Redis or another shared cache. Otherwise each worker learns about the outage separately. Node.js keeps state per process, which is usually acceptable, but with many pods you may also want shared state.
Step 4: Wrap the dependency call and add a fallback
The fallback is what makes the pattern useful to users: a cached value, a default, a queued job for later, or a clear "temporarily unavailable" message. For writes, pushing work into a queue is often the best fallback; see how to build a reliable message queue system.
Step 5: Expose state and alert on it
Log every state change and export it as a metric. An open circuit is a signal that a dependency is unhealthy even if users see only degraded results.
Code/Commands: Circuit Breaker in Laravel and Node.js
Laravel circuit breaker backed by Redis
<?php
namespace App\Support;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Log;
use Throwable;
class CircuitBreaker
{
public function __construct(
private string $name,
private int $failureThreshold = 5,
private int $openSeconds = 30,
) {}
public function call(callable $action, callable $fallback)
{
$openUntil = Cache::get("cb:{$this->name}:open_until");
if ($openUntil && now()->timestamp < $openUntil) {
return $fallback(); // OPEN: fail fast
}
if ($openUntil) { // HALF-OPEN: one trial only
if (! Cache::add("cb:{$this->name}:trial", 1, 10)) {
return $fallback();
}
}
try {
$result = $action();
$this->reset(); // success closes the circuit
return $result;
} catch (Throwable $e) {
$this->recordFailure($e);
return $fallback();
}
}
private function recordFailure(Throwable $e): void
{
$key = "cb:{$this->name}:failures";
$failures = Cache::increment($key);
Cache::put($key, $failures, 60);
if ($failures >= $this->failureThreshold) {
Cache::put("cb:{$this->name}:open_until", now()->timestamp + $this->openSeconds, 3600);
Cache::forget("cb:{$this->name}:trial");
Log::warning("Circuit {$this->name} OPEN", ['error' => $e->getMessage()]);
}
}
private function reset(): void
{
if (Cache::get("cb:{$this->name}:open_until")) {
Log::info("Circuit {$this->name} CLOSED");
}
Cache::forget("cb:{$this->name}:failures");
Cache::forget("cb:{$this->name}:open_until");
Cache::forget("cb:{$this->name}:trial");
}
}Use it around an HTTP call, throwing on server errors so they count as failures:
$breaker = new CircuitBreaker('payments', failureThreshold: 5, openSeconds: 30);
$status = $breaker->call(
fn () => Http::connectTimeout(2)->timeout(3)
->get('https://payments.internal/status')
->throwIfServerError()
->json(),
fn () => Cache::get('payments:last_status', ['status' => 'unknown']),
);Node.js circuit breaker with opossum
For Node.js and Next.js API routes, the opossum library implements the full state machine:
npm install opossumconst CircuitBreaker = require('opossum');
async function getInventory(sku) {
const res = await fetch(`https://inventory.internal/items/${sku}`, {
signal: AbortSignal.timeout(3000),
});
if (res.status >= 500) throw new Error(`inventory ${res.status}`);
return res.json();
}
const breaker = new CircuitBreaker(getInventory, {
timeout: 3000, // treat slow calls as failures
errorThresholdPercentage: 50, // open when half of recent calls fail
volumeThreshold: 10, // need at least 10 calls before judging
resetTimeout: 30000, // try half-open after 30 s
});
breaker.fallback((sku) => ({ sku, available: null, stale: true }));
breaker.on('open', () => console.warn('inventory circuit OPEN'));
breaker.on('halfOpen', () => console.info('inventory circuit HALF-OPEN'));
breaker.on('close', () => console.info('inventory circuit CLOSED'));
module.exports = (sku) => breaker.fire(sku);Nginx as a coarse breaker at the edge
Nginx passive health checks behave like a simple breaker for upstream servers: after max_fails failures, a server is skipped for fail_timeout.
upstream payments {
server 10.0.0.45:8080 max_fails=3 fail_timeout=30s;
server 10.0.0.46:8080 max_fails=3 fail_timeout=30s;
}
location /payments/ {
proxy_pass http://payments;
proxy_connect_timeout 2s;
proxy_read_timeout 5s;
proxy_next_upstream error timeout http_502 http_503;
}Verification: Confirm the Breaker Opens and Recovers
Test in staging by making the dependency fail on purpose. With Docker, stop or pause the container:
# Simulate an outage
docker compose stop inventory
# or simulate a hang (process frozen, connections time out)
docker compose pause inventory
# Send traffic and watch latency drop once the circuit opens
for i in $(seq 1 30); do
curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://localhost:8000/api/products/42
done
# Check Laravel breaker state in Redis
redis-cli KEYS 'cb:*'
redis-cli GET laravel_cache_cb:payments:failures
# Restore and confirm the circuit closes after the cooldown
docker compose unpause inventory
docker compose start inventoryThe first few requests should take about the timeout length, then responses should become fast fallbacks once the circuit opens. After the cooldown, one trial request reaches the dependency and, if it succeeds, normal responses return. Check the logs for the OPEN and CLOSED messages. The exact Redis key prefix depends on your CACHE_PREFIX setting.
Prevention: Building Fault Tolerance Into Every Service
- Set timeouts on every HTTP client, database connection and queue consumer; review defaults in Guzzle,
fetchand ORM configs. - Combine breakers with retries that use exponential backoff and jitter, and cap the attempts. More on this in how to design a distributed system that handles failures.
- Use one breaker per dependency (and sometimes per endpoint), not one global breaker.
- Isolate resources with bulkheads: separate queues, worker pools or connection pools for critical and non-critical dependencies.
- Design fallbacks with the product team so degraded mode is acceptable to users.
- Alert on open circuits and track how often each one trips.
- Run failure drills regularly, not only after an incident.
FAQs
What is the circuit breaker pattern in microservices?
It is a resilience pattern that wraps calls to a remote service, tracks failures, and stops calling the service for a period once failures pass a threshold, returning a fallback instead. It then tests recovery with limited trial calls.
What is the difference between a retry and a circuit breaker?
A retry repeats a failed call hoping a temporary error clears. A circuit breaker stops calls when failures are persistent. Use retries for brief glitches and the breaker to stop retries from overloading a service that is truly down.
How does a circuit breaker improve microservices resilience?
It prevents cascading failures by freeing workers quickly, gives the failing dependency room to recover, and keeps the rest of the system responsive with degraded results instead of errors.
Should circuit breaker state be shared across servers?
For PHP-FPM it should be, usually in Redis, because workers do not share memory. For Node.js, per-process state is often fine, though shared state trips faster in large deployments.
Does a service mesh replace an application circuit breaker?
Meshes such as Istio or Linkerd can provide outlier detection and connection limits at the network layer. Application-level breakers are still useful because they can return business-aware fallbacks.




Comments
Join the discussion. Replies load on demand, up to 5 levels.
0 threads
No comments yet
Be the first to start the conversation.