Scaling Real-Time Connections
Traditional HTTP follows a request-response model, which is inefficient for real-time applications like chat, live sports updates, or collaborative editing. To push data from the server to the client instantly, we need persistent connections, most commonly achieved via WebSockets.
The Stateful Nature of WebSockets
Unlike stateless HTTP REST APIs, WebSocket connections are stateful. Once a client connects to a specific server (e.g., Server A), that TCP connection remains open. If a message needs to be sent to that user, the system must know exactly which server holds their active connection.
The Connection Manager & Pub/Sub
To scale WebSockets across millions of users, a single server cannot hold all connections. We use a cluster of WebSocket servers. The architecture requires two main components:
- Connection registry: Track each connection or device, not only one node per user:
(user, connection_id) -> node, expiry, capabilities. Heartbeats and node-generation fencing remove stale registrations after crashes. - Routing and recovery: Ephemeral Pub/Sub can notify the nodes holding a user's sockets, but it is not a durable inbox. Give durable messages per-conversation or per-recipient positions. After disconnect or node failure, the client resumes from its last acknowledged position and fetches gaps from the source of truth.
Load Balancing
Load balancers must support long-lived upgraded connections and drain them during deployments. Affinity can reduce reconnect churn but cannot be the recovery mechanism: a reconnect may land on another node. Bound outbound queues per connection; disconnect or coalesce updates for a slow client before one socket consumes unbounded memory.
Worked example: sizing a WebSocket fleet
Assume 10 million concurrent users with an average of 1.3 devices each, so about 13 million sockets. If one gateway node comfortably holds 200,000 sockets (kernel buffers plus application state at roughly 20 KB per socket is about 4 GB of RAM), we need 65 nodes at steady state. Add about 30 percent headroom so a node can be drained or lost without overloading its neighbours: roughly 85 nodes.
- Heartbeats: a 30 second ping interval across 13 million sockets is about 433,000 pings per second fleet-wide, which is cheap per node (about 5,000 per second) but must not touch a shared database.
- Node loss: 200,000 clients reconnect. With exponential backoff plus random jitter spread over 30 seconds that is about 6,700 reconnects per second, each doing auth, a registry write and a resume query. Without jitter it can be 200,000 in one or two seconds.
- Deploys: draining one node at a time with a 60 second drain window means a full rollout of 85 nodes takes well over an hour, so many teams drain several nodes in parallel and cap the reconnect rate.
Routing strategies
| Strategy | How a message finds the socket | Strength | Weakness |
|---|---|---|---|
| Broadcast to all nodes | Every node receives every message and drops the ones it does not own | No registry to keep consistent | Traffic grows with nodes times messages; fails beyond a few dozen nodes |
| Connection registry plus direct send | Look up (user, connection) to node, then RPC or per-node queue | Precise, scales well | Registry must be fenced against stale entries after crashes |
| Topic subscription per node | Each node subscribes to the channels or conversations its local users care about | Natural for group chat and live rooms | Subscription churn when users join and leave rooms quickly |
| Consistent hashing of channels | A channel server owns each channel; gateways forward to it | Ordering per channel is easy | Rebalancing on membership change must hand off state |
Resume protocol
How real systems do it
- Discord gateway sessions carry a sequence number; a client that drops sends a Resume with its session id and last sequence to receive missed events instead of a full re-sync.
- Slack separates gateway servers (hold sockets) from channel servers (own channels via consistent hashing), so fan-out and connection holding scale independently.
- Netflix Zuul Push keeps a push registry mapping client to server and treats connections as ephemeral, deliberately recycling them to rebalance load.
Failure modes to name in an interview
- Split-brain registry entries after a network partition: fence with a node generation or epoch and expire entries without heartbeats.
- Slow client: bound the outbound queue, coalesce state updates (send the latest presence, not all of them), then disconnect.
- Thundering herd after a regional failover: server-sent retry hints plus client jitter.