Key Functional Requirements
A massive chat system (like WhatsApp, Discord, or Slack) must deliver 1-on-1 and group messages with sub-second latency, support offline delivery, provide accurate delivery receipts (Sent, Delivered, Read), and handle bidirectional connections at scale.
Connection Management Architecture
Stateful WebSockets & Gateway Tier
HTTP polling introduces unacceptable overhead. Clients maintain persistent, stateful WebSocket or TCP connections. A centralized Connection Registry (often implemented using Redis or distributed hash tables) maps each userId to the specific gateway server IP currently hosting their active socket connection.
Handling Offline Messages
When User A sends a message to User B:
- The Message Router writes the message to an append-optimized distributed store (such as Cassandra or ScyllaDB) with a monotonic sequence ID per conversation, assigned by a per-conversation sequencer or a Snowflake-style generator (Cassandra and ScyllaDB have no auto-increment).
- The router checks the connection registry. If User B is offline, the message is queued for offline delivery and a push notification is dispatched via APNs/FCM.
- When User B reconnects, their client sends their last received message ID; the gateway queries the store for all subsequent messages and synchronizes them in order.
Group Chat Scaling: Fan-out Strategy
For small group chats (e.g., up to 1,024 members in WhatsApp), the message router fans out a copy to each recipient's inbox queue. For large group channels (e.g., 100,000 members in Discord), fan-out-on-write is too expensive; instead, channels use shared pub/sub event streams where clients pull from a single channel partition.
Worked example: connection and message scale
- 100 million concurrently connected clients. If each gateway server holds about 500,000 sockets (feasible with epoll or kqueue and small per-connection buffers), you need about 200 gateway servers plus headroom for rolling restarts. WhatsApp famously reported more than 2 million connections on a single Erlang server in 2012.
- Heartbeats every 30 s from 100 million clients produce about 3.3 million small packets per second on their own. This is why mobile clients stretch keepalive intervals and rely on push when the app is in the background.
- 20 billion messages per day is about 230,000 per second on average. At about 200 B each with metadata, that is about 4 TB per day before replication. It goes in a wide-column store partitioned by
(conversation_id, time_bucket), so no single partition grows without bound. Discord uses channel ID plus a 10-day bucket.
Fan-out by group size
| Conversation type | Write strategy | Read strategy | Example |
|---|---|---|---|
| 1:1 | Store once, then deliver to the recipient's gateway or push | Read the conversation partition since the cursor | WhatsApp, iMessage |
| Small group (up to about 1,000) | Store once, route to each online member's gateway | Each member syncs from their cursor | WhatsApp groups (limit raised to 1,024 members in 2022) |
| Large channel or server | Store once in the channel log, publish to a pub/sub topic | Online clients subscribe; others load recent history on open | Discord, Slack large channels |
| Broadcast | Store once; notify lazily | Pull on open | Telegram channels |
Message lifecycle and receipts
The client_msg_id makes resends idempotent. If the sender's network drops before the acknowledgement arrives, the retry is deduplicated rather than stored twice. With end-to-end encryption (the Signal protocol, used by WhatsApp), the server stores and routes only ciphertext. Group messages use sender keys, so a sender encrypts once rather than once per member.