Design: recent searches

Under the Hood: User Search History at Scale

When a user taps the search bar, mobile and web applications display their recent search history (e.g. the last 10 queries executed by that account). While conceptually straightforward, serving recent searches for 500,000,000 active users requires low latency, strict GDPR compliance, cross-device synchronization, and zero database overload.

Redis Capped Set Architecture

flowchart TD Client["Mobile Client: Taps Search Bar"] --> Edge["API Gateway"] Edge --> SearchHistoryService["Recent Search Service"] SearchHistoryService -->|"1. ZRANGE history:<user_id> 0 9 REV"| RedisCluster[("Redis Cluster (In-Memory)")] RedisCluster -->|"Returns 10 most recent queries"| SearchHistoryService subgraph MutationFlow ["When User Executes New Query: 'system design'"] UserSearch["User Search Event"] --> Worker["Async Mutation Worker"] Worker -->|"2. Atomic Lua: ZADD history:<user_id> <now_timestamp> <query> + ZREMRANGEBYRANK 0 -11"| RedisCluster Worker -->|"3. Append to User History Log"| ColdStorage[("Encrypted Cassandra / DynamoDB")] end

Under the Hood: Redis Sorted Sets vs Lists

Why Lists Fail: A simple Redis List (LPUSH + LTRIM) permits duplicate queries. If a user searches "iPhone" five times, their recent search list displays five identical "iPhone" entries. De-duplicating a list requires O(N) scans.
The Sorted Set Solution (ZSET): Redis Sorted Sets store unique members sorted by score. By storing the query string as the member and the current Unix timestamp as the score, executing ZADD history:uid timestamp query automatically de-duplicates the query and refreshes its recency to the top. Capping the history is achieved via ZREMRANGEBYRANK history:uid 0 -11.
Privacy & GDPR Compliance: Search history must support instant user purges (DEL history:uid) and automatic time-to-live expiration (e.g. EXPIRE history:uid 7776000 for 90 days retention).

Worked example: sizing

Quantity	Assumption	Result
Users with stored history	500 million
Entries per user	10
Bytes per entry	About 30 byte query plus score and encoding overhead	about 50 bytes
Per-user key	Entries plus key and object overhead	roughly 0.5 to 1 KB
Total if all users hot	500 million x about 1 KB	about 500 GB across a Redis cluster
Hot set only, 100 million daily actives	100 million x about 1 KB	about 100 GB
Write rate	5 billion searches per day	about 58,000 writes per second average

Small sorted sets (up to 128 entries by default) use Redis's compact listpack encoding, so a 10-entry history is cheap. A common optimisation is to keep only active users in Redis with a TTL and lazily reload others from the durable store on first access.

The write path, atomically

Run these as one Lua script or MULTI block so concurrent searches cannot leave the set over-length:

ZADD history:uid 1727700000 "system design" inserts or refreshes the timestamp of an existing entry.
ZREMRANGEBYRANK history:uid 0 -11 removes everything except the 10 highest scores.
EXPIRE history:uid 7776000 gives 90-day retention.

Read path: ZRANGE history:uid 0 9 REV, available since Redis 6.2, returns newest first.

Design trade-offs

Choice	Benefit	Cost
Redis only	Simplest, fastest	History lost on eviction or failover without persistence
Redis plus Cassandra or DynamoDB	Durable, cross-device	Two writes; needs async sync and idempotent upserts
Client-side storage only	No server cost, most private	No cross-device sync

Edge cases: normalise queries (case, whitespace) before using them as members; do not record searches made in incognito or private modes; and make deletion of one entry (ZREM) propagate to every store.

Cross-device sync

If a user searches on the phone and then opens the laptop, the laptop should show the new entry. Write to the durable store keyed by user ID, publish a small change event, and let the cache update or invalidate the user's key; last-write-wins on the timestamp score is sufficient because entries are independent and a slightly stale order is harmless.

More videos on this topic