A single Node.js process can only hold so many open sockets before file descriptors, memory, and event-loop latency become the bottleneck. Real websocket scaling means running multiple app instances behind a load balancer and making sure messages still reach every client in a room—even when those clients land on different servers. This guide covers sticky sessions, a distributed WebSocket architecture with Redis pub sub, Socket.IO adapters, Nginx upstream settings, connection limits, heartbeats, and the close codes you will see when affinity breaks.
Problem: Why WebSocket Scaling Breaks on One Server
WebSockets are long-lived TCP connections. Unlike short HTTP requests, you cannot casually spray them across random backends and expect shared in-memory room maps to work. One process stores roomId → Set<socket> in RAM. The moment you add a second process for capacity or high availability, that map is no longer global.
Teams often discover this after they already shipped a chat or live ERP notification feature (see the companion websocket chat application guide). Horizontal growth is the right next step—covered conceptually in horizontal vs vertical scaling—but WebSockets need an explicit fan-out bus, not only a bigger VM.
Symptoms of Broken Multi-Server WebSockets
- Two users in the same room never see each other's messages when they hit different pods.
- Browser close code
1006(abnormal closure) after deploys, LB health flips, or idle proxy timeouts. - Reconnect lands on another instance; UI looks connected but room state is empty until a hard refresh.
- Socket.IO falls back to long-polling and then fails handshake mid-upgrade because the next request hit a different server without sticky sessions.
- Nginx or kernel logs:
worker_connections are not enough,Too many open files, or upstreamConnection reset by peer. - CPU climbs while message latency grows: one hot instance holds most sticky clients; others sit idle.
Root Cause: Local Sockets, Sticky Misses, and Missing Fan-Out
- Process-local state. Rooms, presence, and pending acks live in one Node heap. Other instances cannot see those sockets.
- Sticky sessions alone are not enough.
ip_hashor cookie affinity keeps a client on one server for reconnects, but peers in the same room still land elsewhere. Without a shared bus, they never meet. - No Redis (or equivalent) pub/sub. Publish on instance A never reaches local sockets on instance B.
- Proxy limits. Default Nginx
proxy_read_timeout, missingUpgrade/Connectionheaders, or lowulimit -ndrop quiet connections with1006. - Uneven affinity. Sticky by IP collapses behind corporate NATs; one app node becomes the bottleneck.
For when Redis pub/sub is the right bus versus durable Streams, see Redis Pub/Sub vs Streams. Broader multi-node resilience patterns are in high availability architecture.
Step-by-Step Solution: Scale WebSocket Connections Across Servers
Step 1: Decide sticky vs bus-first design
Use sticky sessions to keep a single client's reconnects and Socket.IO polling upgrades on one process. Still add Redis pub/sub (or the Socket.IO Redis adapter) so room broadcasts cross instances. Treat sticky as an optimization for reconnect locality—not as your messaging fabric.
Step 2: Put multiple Node WebSocket processes behind Nginx
Run at least two app instances (for example ports 8081–8083). Terminate TLS at Nginx or a cloud LB, then proxy /ws (or your Socket.IO path) with WebSocket upgrade headers, long read timeouts, and either ip_hash or a sticky cookie.
Step 3: Raise connection limits early
Each socket consumes a file descriptor. Raise ulimit -n (systemd LimitNOFILE), Nginx worker_connections, and OS somaxconn before load tests. Monitor open FDs per process; hitting the ceiling looks like random 1006 failures under load.
Step 4: Add Redis pub/sub fan-out (distributed WebSocket architecture)
When a process accepts a chat (or notification) message, PUBLISH it to a channel such as ws:room:{id}. Every app instance SUBSCRIBEs (or PSUBSCRIBE ws:room:*) and forwards only to its local sockets in that room. Persist history in PostgreSQL; Redis here is the live bus.
Step 5: Prefer the Socket.IO Redis adapter when you already use Socket.IO
@socket.io/redis-adapter wires the same idea for you: io.to(room).emit(…) fans out across nodes. You still want sticky sessions so the Engine.IO handshake and polling transport stay coherent.
Step 6: Heartbeat, rooms, and graceful draining
Application pings every ~25s; close silent sockets. On deploy, stop accepting new upgrades, wait for active sockets to drain (or force reconnect with backoff), then exit. Clients should rejoin rooms after onopen—patterns in websocket reconnect.
Code/Commands for WebSocket Scaling
Nginx upstream with WebSocket upgrade and sticky affinity
# /etc/nginx/nginx.conf (excerpt)
events {
worker_connections 8192;
}
upstream ws_backend {
# sticky by client IP; swap for sticky cookie module if clients share NATs
ip_hash;
server 127.0.0.1:8081;
server 127.0.0.1:8082;
server 127.0.0.1:8083;
keepalive 64;
}
server {
listen 443 ssl http2;
server_name realtime.example.com;
location /ws {
proxy_pass http://ws_backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
}Node.js ws + Redis pub/sub across instances
npm i ws redis dotenv
# REDIS_URL=redis://127.0.0.1:6379 PORT=8081// scale-server.mjs
import http from 'node:http';
import { WebSocketServer } from 'ws';
import { createClient } from 'redis';
import 'dotenv/config';
const port = Number(process.env.PORT || 8081);
const server = http.createServer();
const wss = new WebSocketServer({ server, path: '/ws' });
const pub = createClient({ url: process.env.REDIS_URL });
const sub = pub.duplicate();
await Promise.all([pub.connect(), sub.connect()]);
const rooms = new Map(); // roomId -> Set<WebSocket>
function localBroadcast(roomId, payload) {
const set = rooms.get(roomId);
if (!set) return;
const data = JSON.stringify(payload);
for (const ws of set) {
if (ws.readyState === 1) ws.send(data);
}
}
await sub.pSubscribe('ws:room:*', (message, channel) => {
const roomId = channel.slice('ws:room:'.length);
localBroadcast(roomId, JSON.parse(message));
});
wss.on('connection', (ws) => {
ws.isAlive = true;
ws.rooms = new Set();
ws.on('pong', () => { ws.isAlive = true; });
ws.on('message', async (buf) => {
let msg; try { msg = JSON.parse(buf.toString()); } catch { return; }
if (msg.type === 'join' && msg.roomId) {
const roomId = String(msg.roomId);
if (!rooms.has(roomId)) rooms.set(roomId, new Set());
rooms.get(roomId).add(ws);
ws.rooms.add(roomId);
return;
}
if (msg.type === 'chat' && msg.roomId && msg.text) {
const payload = {
type: 'chat',
roomId: String(msg.roomId),
text: String(msg.text).slice(0, 2000),
ts: Date.now(),
via: process.env.PORT,
};
// Fan out to every instance; subscribers call localBroadcast
await pub.publish(`ws:room:${payload.roomId}`, JSON.stringify(payload));
}
});
ws.on('close', () => {
for (const roomId of ws.rooms) {
rooms.get(roomId)?.delete(ws);
}
});
});
setInterval(() => {
for (const ws of wss.clients) {
if (!ws.isAlive) { ws.terminate(); continue; }
ws.isAlive = false;
ws.ping();
}
}, 25000);
server.listen(port, () => console.log('ws listening', port));Socket.IO Redis adapter (alternative)
import { Server } from 'socket.io';
import { createAdapter } from '@socket.io/redis-adapter';
import { createClient } from 'redis';
const pub = createClient({ url: process.env.REDIS_URL });
const sub = pub.duplicate();
await Promise.all([pub.connect(), sub.connect()]);
const io = new Server(3000, {
path: '/socket.io',
cors: { origin: process.env.WEB_ORIGIN },
});
io.adapter(createAdapter(pub, sub));
io.on('connection', (socket) => {
socket.on('join', (roomId) => socket.join(String(roomId)));
socket.on('chat', ({ roomId, text }) => {
io.to(String(roomId)).emit('chat', {
roomId, text, ts: Date.now(),
});
});
});Quick ops checks
# file descriptors / limits
ulimit -n
cat /proc/$(pgrep -n node)/limits | grep 'open files'
# Redis bus
redis-cli PING
redis-cli PUBSUB CHANNELS 'ws:room:*'
redis-cli PUBLISH ws:room:42 '{"type":"chat","roomId":"42","text":"probe"}'
# two instances
PORT=8081 node scale-server.mjs &
PORT=8082 node scale-server.mjs &
# point Nginx at both; open two browsers to the same roomVerification
- Open two clients forced onto different backends (temporarily disable sticky, or connect to
:8081and:8082directly). Join the same room; a chat from A appears on B within a fraction of a second only when Redis pub/sub (or the Socket.IO adapter) is wired. redis-cli MONITOR(on a non-production box) showsPUBLISH ws:room:…when messages flow.- With sticky enabled, kill one app process: affected clients close with
1006or similar, reconnect to a healthy peer, rejoin rooms, and resume. - Leave a quiet socket open past the old default proxy timeout; with
proxy_read_timeoutraised and heartbeats running, the connection stays up. - Under a modest load test, no
Too many open files/worker_connectionserrors appear in app or Nginx logs.
Prevention: Keep Multi-Server WebSockets Healthy
- Always pair horizontal WebSocket fleets with a shared fan-out bus (Redis pub/sub or equivalent).
- Keep sticky sessions for Socket.IO / reconnect locality, but never rely on affinity alone for room delivery.
- Size
ulimit, Nginxworker_connections, and proxy timeouts for peak concurrent sockets—not average HTTP QPS. - Heartbeat both ways; document close codes and client reconnect with jitter (see websocket connection failed).
- Drain connections on deploy; avoid hard-killing all pods at once or you create a reconnect storm.
- Persist authoritative messages in the database; treat Redis as the live pipe, not the source of truth (Queues/Streams patterns in message queue architecture).
FAQs
What does websocket scaling mean in practice?
Running multiple WebSocket servers behind a load balancer so concurrent connections and CPU are shared, while a bus such as Redis pub/sub keeps room broadcasts consistent across instances.
Do I need sticky sessions if I already have Redis pub sub?
For raw ws, Redis alone is enough for fan-out; sticky still helps reconnects stay warm on one process. For Socket.IO, sticky (or a single long-lived WebSocket transport) avoids broken multi-request handshakes even with the Redis adapter.
What is a distributed WebSocket architecture?
Clients connect to any healthy instance; each instance holds only local sockets; a shared pub/sub layer copies events to every instance that has members of that room or channel.
Why do clients get close code 1006 after I add a second server?
Often a sticky miss during upgrade, a proxy that does not forward Upgrade/Connection, idle timeouts shorter than your heartbeat, or an instance crash without graceful drain.
Is Redis pub/sub durable?
No. Subscribers only receive messages while connected. Persist chat or ERP events in PostgreSQL (or use Redis Streams when you need bus-level replay).
How many sockets can one Node process hold?
It depends on RAM, message rate, and ulimit—not a fixed marketing number. Measure open FDs and event-loop lag under your own payloads, then add instances before you hit the cliff.




Comments
Join the discussion. Replies load on demand, up to 5 levels.
0 threads
No comments yet
Be the first to start the conversation.