Extracting and Indexing Hashtags
Hashtags power content discovery and trending topics. Processing them at scale involves real-time text parsing, indexing, and aggregation.
Extraction Pipeline
When a post is created, it is pushed to an asynchronous processing queue (e.g., Kafka). Stream processors (like Apache Flink or Storm) consume the posts, run regex to extract hashtags, normalize them (Unicode normalization such as NFC, case folding rather than ASCII-only lowercasing, stripping punctuation), and emit them as events.
The Tag Store (Inverted Index)
To find all posts for a hashtag, the system uses an Inverted Index—the same core structure used by search engines like Elasticsearch. Instead of mapping a Post to its Tags, it maps a Tag to a list of Post IDs.
In a NoSQL database like Cassandra, this looks like a table partitioned by (hashtag, time bucket, shard), with rows clustered by post_id (or time). Partitioning by hashtag alone would create unbounded hot partitions for popular tags. Reading the newest page for '#systemdesign' means querying the current bucket's shards and merging, a fast bounded read rather than an instant one.
Trending Algorithms
To calculate what is "trending," you don't just count the total volume of a tag; you calculate its velocity. An algorithm like Exponential Moving Average (EMA) or sliding window counters (often using Redis or Count-Min Sketches for memory efficiency) gives higher weight to hashtags that have suddenly spiked in usage in the last few minutes compared to their historical baseline.
Worked example: volume and hot tags
Assume 500 million posts per day, 20 percent carrying hashtags with an average of 2 tags each: 200 million postings per day, about 2,300 per second on average and perhaps 10 times that during a global event. At 16 bytes per posting (post ID plus timestamp) that is about 3.2 GB of raw index per day before replication and overhead.
During a final, one tag might see 50,000 posts per minute, about 830 writes per second into one partition, and millions of reads of its first page. Use a partition key like (tag, day, shard) with shard = hash(post_id) mod 16, so writes spread across 16 partitions; readers fetch the newest page from each shard and merge (or read a cached merged first page refreshed every few seconds).
Trending with bounded memory
A count-min sketch with width w = ceil(e / epsilon) and depth d = ceil(ln(1 / delta)) overestimates any count by at most epsilon x N with probability 1 - delta, where N is the number of events in the window. With epsilon = 0.00001 and delta = 0.01 that is w = 271,829 and d = 5, about 1.36 million 4-byte counters or 5.4 MB per window. If a 5 minute window holds 10 million tag events, the error bound is at most 100 counts. Keep a small heap of the top candidates, then score each by something like (count_now + a) / (baseline_rate + b), where the smoothing constants stop tiny tags going from 1 to 5 from looking like a 500 percent spike.
Storage options
| Store | Serves | Strength | Weakness |
|---|---|---|---|
| Wide-column table by (tag, bucket) | Newest posts for a tag | Cheap appends, simple range reads | No relevance ranking or multi-term queries |
| Search engine (Elasticsearch, OpenSearch, Lucene based) | Tag plus keyword plus filters | Rich queries and ranking | Refresh lag, heavier operations |
| Redis sorted set per hot tag | First page of hot tags | Very low latency | Memory bound; needs trimming |
| Stream processor state (Flink) | Windowed trending counts | Exactly-once windows, event time | Not a query store; publish results elsewhere |
How real systems do it
- Twitter open-sourced the twitter-text libraries so every client and server extracted hashtags identically, and its search tier (Earlybird) is an in-memory, time-partitioned inverted index built on Lucene ideas.
- Twitter's trends explicitly favour topics that are spiking now over topics that are merely always popular.