Time-Window Aggregation
In streaming analytics, we rarely care about all data since the beginning of time. We want metrics over specific intervals: "views per minute," "errors per hour."
Tumbling vs Sliding Windows
- Tumbling Windows: Fixed, non-overlapping intervals (e.g., 1:00-1:05, 1:05-1:10). An event belongs to exactly one window.
- Sliding Windows: Overlapping intervals (e.g., last 5 minutes, updated every 1 minute). An event can belong to multiple windows.
Implementation with Redis
A common pattern for a 1-minute tumbling window is to use a timestamp rounded down to the minute as the cache key: INCR api_errors:2026-09-19T17:10. This is highly efficient and requires no complex stream processing. You set an expiry (TTL) on the key to automatically clean up old data.
Stream Processors
For event-time windows, crossing the wall-clock boundary does not prove all events have arrived. Suppose a 10:00–10:05 bucket first publishes 90 views, then a mobile event with occurrence time 10:04 arrives at 10:08. A watermark and allowed-lateness policy decide whether to update the bucket to 91, emit a retraction/correction, or route the event to a late-data path. Retaining state longer improves completeness but consumes more memory and delays finality; discarding it early makes published results cheaper but permanently excludes accepted late events.
Worked example: sliding window from small buckets
Requirement: "errors in the last 5 minutes, updated every minute" for 10,000 API endpoints.
- Keep 1-minute tumbling buckets per endpoint: key
errors:endpoint:2026-09-30T10:07, incremented with INCR and given a TTL of 10 minutes. - At 10:08 the 5-minute value is the sum of buckets 10:03 through 10:07: one MGET of 5 keys.
- Memory: 10,000 endpoints x 10 live buckets = 100,000 small keys, a few MB.
This approximates a true sliding window with 1-minute granularity. If you need exactness to the second, use smaller buckets (more keys) or a sliding log of timestamps (more memory per event).
Window types compared
| Window | Definition | Event belongs to | Typical use |
|---|---|---|---|
| Tumbling | Fixed size, no overlap, such as every 5 minutes | Exactly one window | Per-minute dashboards, billing buckets |
| Hopping | Fixed size, fixed advance, such as 5 minutes every 1 minute | Size divided by advance windows, here 5 | Smoothed moving metrics |
| Sliding | Window ends at each event or query time | Every window that covers it | Rate limiting, anomaly detection |
| Session | Closes after an inactivity gap, such as 30 minutes | One session, size varies | User sessions, engagement |
Watermark trade-off in numbers
Suppose 99 percent of mobile events arrive within 2 minutes and 99.9 percent within 20 minutes. A 2-minute watermark publishes results fast but leaves about 1 percent to arrive as late corrections; a 20-minute watermark leaves about 0.1 percent late, but every dashboard is 20 minutes behind and the stream processor holds 10 times more open window state. Many teams publish early with a short watermark, emit updates as late data arrives, and mark a bucket "final" after the longer bound.
Real systems: Flink and Beam support allowed lateness and triggers that re-emit updated results; Kafka Streams uses a grace period per window; Prometheus-style systems avoid the problem by scraping at a fixed interval and defining the sample time as the scrape time.