Skip to content
|By Muhammad Wilayat Shah|12 views • 12 unique IPs

How to Scale WebSocket Connections Across Multiple Servers

Practical websocket scaling: sticky sessions vs Redis pub sub, Socket.IO adapter, Nginx upstream, heartbeats, and multi-server fan-out.

ShareLinkedInX
How to Scale WebSocket Connections Across Multiple Servers

A single Node.js process can only hold so many open sockets before file descriptors, memory, and event-loop latency become the bottleneck. Real websocket scaling means running multiple app instances behind a load balancer and making sure messages still reach every client in a room—even when those clients land on different servers. This guide covers sticky sessions, a distributed WebSocket architecture with Redis pub sub, Socket.IO adapters, Nginx upstream settings, connection limits, heartbeats, and the close codes you will see when affinity breaks.

Problem: Why WebSocket Scaling Breaks on One Server

WebSockets are long-lived TCP connections. Unlike short HTTP requests, you cannot casually spray them across random backends and expect shared in-memory room maps to work. One process stores roomId → Set<socket> in RAM. The moment you add a second process for capacity or high availability, that map is no longer global.

Teams often discover this after they already shipped a chat or live ERP notification feature (see the companion websocket chat application guide). Horizontal growth is the right next step—covered conceptually in horizontal vs vertical scaling—but WebSockets need an explicit fan-out bus, not only a bigger VM.

Symptoms of Broken Multi-Server WebSockets

  • Two users in the same room never see each other's messages when they hit different pods.
  • Browser close code 1006 (abnormal closure) after deploys, LB health flips, or idle proxy timeouts.
  • Reconnect lands on another instance; UI looks connected but room state is empty until a hard refresh.
  • Socket.IO falls back to long-polling and then fails handshake mid-upgrade because the next request hit a different server without sticky sessions.
  • Nginx or kernel logs: worker_connections are not enough, Too many open files, or upstream Connection reset by peer.
  • CPU climbs while message latency grows: one hot instance holds most sticky clients; others sit idle.
websocket scaling diagram: sticky session miss versus Redis pub/sub fan-out across multiple servers

Root Cause: Local Sockets, Sticky Misses, and Missing Fan-Out

  • Process-local state. Rooms, presence, and pending acks live in one Node heap. Other instances cannot see those sockets.
  • Sticky sessions alone are not enough. ip_hash or cookie affinity keeps a client on one server for reconnects, but peers in the same room still land elsewhere. Without a shared bus, they never meet.
  • No Redis (or equivalent) pub/sub. Publish on instance A never reaches local sockets on instance B.
  • Proxy limits. Default Nginx proxy_read_timeout, missing Upgrade/Connection headers, or low ulimit -n drop quiet connections with 1006.
  • Uneven affinity. Sticky by IP collapses behind corporate NATs; one app node becomes the bottleneck.

For when Redis pub/sub is the right bus versus durable Streams, see Redis Pub/Sub vs Streams. Broader multi-node resilience patterns are in high availability architecture.

Step-by-Step Solution: Scale WebSocket Connections Across Servers

Step 1: Decide sticky vs bus-first design

Use sticky sessions to keep a single client's reconnects and Socket.IO polling upgrades on one process. Still add Redis pub/sub (or the Socket.IO Redis adapter) so room broadcasts cross instances. Treat sticky as an optimization for reconnect locality—not as your messaging fabric.

Step 2: Put multiple Node WebSocket processes behind Nginx

Run at least two app instances (for example ports 8081–8083). Terminate TLS at Nginx or a cloud LB, then proxy /ws (or your Socket.IO path) with WebSocket upgrade headers, long read timeouts, and either ip_hash or a sticky cookie.

Step 3: Raise connection limits early

Each socket consumes a file descriptor. Raise ulimit -n (systemd LimitNOFILE), Nginx worker_connections, and OS somaxconn before load tests. Monitor open FDs per process; hitting the ceiling looks like random 1006 failures under load.

Step 4: Add Redis pub/sub fan-out (distributed WebSocket architecture)

When a process accepts a chat (or notification) message, PUBLISH it to a channel such as ws:room:{id}. Every app instance SUBSCRIBEs (or PSUBSCRIBE ws:room:*) and forwards only to its local sockets in that room. Persist history in PostgreSQL; Redis here is the live bus.

Step 5: Prefer the Socket.IO Redis adapter when you already use Socket.IO

@socket.io/redis-adapter wires the same idea for you: io.to(room).emit(…) fans out across nodes. You still want sticky sessions so the Engine.IO handshake and polling transport stay coherent.

Step 6: Heartbeat, rooms, and graceful draining

Application pings every ~25s; close silent sockets. On deploy, stop accepting new upgrades, wait for active sockets to drain (or force reconnect with backoff), then exit. Clients should rejoin rooms after onopen—patterns in websocket reconnect.

Code/Commands for WebSocket Scaling

Nginx upstream with WebSocket upgrade and sticky affinity

Code
# /etc/nginx/nginx.conf (excerpt)
events {
  worker_connections 8192;
}

upstream ws_backend {
  # sticky by client IP; swap for sticky cookie module if clients share NATs
  ip_hash;
  server 127.0.0.1:8081;
  server 127.0.0.1:8082;
  server 127.0.0.1:8083;
  keepalive 64;
}

server {
  listen 443 ssl http2;
  server_name realtime.example.com;

  location /ws {
    proxy_pass http://ws_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
  }
}

Node.js ws + Redis pub/sub across instances

Code
npm i ws redis dotenv
# REDIS_URL=redis://127.0.0.1:6379  PORT=8081
Code
// scale-server.mjs
import http from 'node:http';
import { WebSocketServer } from 'ws';
import { createClient } from 'redis';
import 'dotenv/config';

const port = Number(process.env.PORT || 8081);
const server = http.createServer();
const wss = new WebSocketServer({ server, path: '/ws' });

const pub = createClient({ url: process.env.REDIS_URL });
const sub = pub.duplicate();
await Promise.all([pub.connect(), sub.connect()]);

const rooms = new Map(); // roomId -> Set<WebSocket>

function localBroadcast(roomId, payload) {
  const set = rooms.get(roomId);
  if (!set) return;
  const data = JSON.stringify(payload);
  for (const ws of set) {
    if (ws.readyState === 1) ws.send(data);
  }
}

await sub.pSubscribe('ws:room:*', (message, channel) => {
  const roomId = channel.slice('ws:room:'.length);
  localBroadcast(roomId, JSON.parse(message));
});

wss.on('connection', (ws) => {
  ws.isAlive = true;
  ws.rooms = new Set();
  ws.on('pong', () => { ws.isAlive = true; });

  ws.on('message', async (buf) => {
    let msg; try { msg = JSON.parse(buf.toString()); } catch { return; }
    if (msg.type === 'join' && msg.roomId) {
      const roomId = String(msg.roomId);
      if (!rooms.has(roomId)) rooms.set(roomId, new Set());
      rooms.get(roomId).add(ws);
      ws.rooms.add(roomId);
      return;
    }
    if (msg.type === 'chat' && msg.roomId && msg.text) {
      const payload = {
        type: 'chat',
        roomId: String(msg.roomId),
        text: String(msg.text).slice(0, 2000),
        ts: Date.now(),
        via: process.env.PORT,
      };
      // Fan out to every instance; subscribers call localBroadcast
      await pub.publish(`ws:room:${payload.roomId}`, JSON.stringify(payload));
    }
  });

  ws.on('close', () => {
    for (const roomId of ws.rooms) {
      rooms.get(roomId)?.delete(ws);
    }
  });
});

setInterval(() => {
  for (const ws of wss.clients) {
    if (!ws.isAlive) { ws.terminate(); continue; }
    ws.isAlive = false;
    ws.ping();
  }
}, 25000);

server.listen(port, () => console.log('ws listening', port));

Socket.IO Redis adapter (alternative)

Code
import { Server } from 'socket.io';
import { createAdapter } from '@socket.io/redis-adapter';
import { createClient } from 'redis';

const pub = createClient({ url: process.env.REDIS_URL });
const sub = pub.duplicate();
await Promise.all([pub.connect(), sub.connect()]);

const io = new Server(3000, {
  path: '/socket.io',
  cors: { origin: process.env.WEB_ORIGIN },
});
io.adapter(createAdapter(pub, sub));

io.on('connection', (socket) => {
  socket.on('join', (roomId) => socket.join(String(roomId)));
  socket.on('chat', ({ roomId, text }) => {
    io.to(String(roomId)).emit('chat', {
      roomId, text, ts: Date.now(),
    });
  });
});

Quick ops checks

Code
# file descriptors / limits
ulimit -n
cat /proc/$(pgrep -n node)/limits | grep 'open files'

# Redis bus
redis-cli PING
redis-cli PUBSUB CHANNELS 'ws:room:*'
redis-cli PUBLISH ws:room:42 '{"type":"chat","roomId":"42","text":"probe"}'

# two instances
PORT=8081 node scale-server.mjs &
PORT=8082 node scale-server.mjs &
# point Nginx at both; open two browsers to the same room

Verification

  • Open two clients forced onto different backends (temporarily disable sticky, or connect to :8081 and :8082 directly). Join the same room; a chat from A appears on B within a fraction of a second only when Redis pub/sub (or the Socket.IO adapter) is wired.
  • redis-cli MONITOR (on a non-production box) shows PUBLISH ws:room:… when messages flow.
  • With sticky enabled, kill one app process: affected clients close with 1006 or similar, reconnect to a healthy peer, rejoin rooms, and resume.
  • Leave a quiet socket open past the old default proxy timeout; with proxy_read_timeout raised and heartbeats running, the connection stays up.
  • Under a modest load test, no Too many open files / worker_connections errors appear in app or Nginx logs.

Prevention: Keep Multi-Server WebSockets Healthy

  • Always pair horizontal WebSocket fleets with a shared fan-out bus (Redis pub/sub or equivalent).
  • Keep sticky sessions for Socket.IO / reconnect locality, but never rely on affinity alone for room delivery.
  • Size ulimit, Nginx worker_connections, and proxy timeouts for peak concurrent sockets—not average HTTP QPS.
  • Heartbeat both ways; document close codes and client reconnect with jitter (see websocket connection failed).
  • Drain connections on deploy; avoid hard-killing all pods at once or you create a reconnect storm.
  • Persist authoritative messages in the database; treat Redis as the live pipe, not the source of truth (Queues/Streams patterns in message queue architecture).

FAQs

What does websocket scaling mean in practice?

Running multiple WebSocket servers behind a load balancer so concurrent connections and CPU are shared, while a bus such as Redis pub/sub keeps room broadcasts consistent across instances.

Do I need sticky sessions if I already have Redis pub sub?

For raw ws, Redis alone is enough for fan-out; sticky still helps reconnects stay warm on one process. For Socket.IO, sticky (or a single long-lived WebSocket transport) avoids broken multi-request handshakes even with the Redis adapter.

What is a distributed WebSocket architecture?

Clients connect to any healthy instance; each instance holds only local sockets; a shared pub/sub layer copies events to every instance that has members of that room or channel.

Why do clients get close code 1006 after I add a second server?

Often a sticky miss during upgrade, a proxy that does not forward Upgrade/Connection, idle timeouts shorter than your heartbeat, or an instance crash without graceful drain.

Is Redis pub/sub durable?

No. Subscribers only receive messages while connected. Persist chat or ERP events in PostgreSQL (or use Redis Streams when you need bus-level replay).

How many sockets can one Node process hold?

It depends on RAM, message rate, and ulimit—not a fixed marketing number. Measure open FDs and event-loop lag under your own payloads, then add instances before you hit the cliff.

ShareLinkedInX
Back to blog

Keep reading

Newsletter

Occasional notes on Laravel, React, and shipping production systems — no spam.

Comments

Join the discussion. Replies load on demand, up to 5 levels.

0 threads

Loading editor…
Reviewed before publishing

No comments yet

Be the first to start the conversation.