Distributed task scheduler

Under the Hood: High-Throughput Delayed Job Scheduling

Scheduling millions of future tasks (e.g. "Send reminder email in 2 hours", "Charge card in 3 days", "Retry webhook in 30 seconds") requires coordinating delayed execution across a distributed cluster without race conditions or dropped jobs.

Redis Sorted Set (ZSET) Delayed Scheduler

Under the Hood: The Atomic Lua Script Pattern

A claim must prevent two schedulers from owning the same attempt without deleting the only durable obligation. Atomically move due IDs from the scheduled set into a leased set whose score is the lease deadline:

-- Claim due jobs without forgetting them
local due_jobs = redis.call('ZRANGEBYSCORE', KEYS[1], 0, ARGV[1], 'LIMIT', 0, ARGV[2])
if #due_jobs > 0 then
    redis.call('ZREM', KEYS[1], unpack(due_jobs))
    for _, job_id in ipairs(due_jobs) do
        redis.call('ZADD', KEYS[2], ARGV[3], job_id)
    end
    return due_jobs
end
return {}
      

On Redis Cluster, KEYS[1] and KEYS[2] must hash to the same slot, so name them with a hash tag such as {jobs}:scheduled and {jobs}:leased; otherwise the script fails with a CROSSSLOT error.

After claiming, publish each job with its stable ID. Only clear the lease after the broker confirms acceptance. If the scheduler crashes first, a reaper returns expired leases to the scheduled set. If publication succeeded but its acknowledgement was lost, retry may publish a duplicate, so the execution queue or worker must deduplicate by job ID. Removing a job before this recoverable handoff creates a lost-job window.

Hierarchical Hashed Wheel Timers

For in-process ultra-high-frequency scheduling (sub-millisecond to seconds), distributed databases use the Hashed Wheel Timer algorithm (Varghese & Lauck). A circular array of buckets advances like a clock hand and can provide constant-time insertion and tick work under specific implementations. Cancellation cost and worst-case bucket work depend on the data structure and timer distribution; this is an in-process timer, not a durable distributed scheduler.

Worked example: sizing a delayed-job store

A reminders product schedules 50 million jobs per day, spread unevenly, with 5 times the average during the evening peak.

• Average rate: 50,000,000 / 86,400 = about 580 jobs due per second. Peak is about 2,900 per second.
• Pending jobs (scheduled up to 7 days ahead): up to about 350 million entries. In a Redis sorted set at about 100 bytes each that is about 35 GB, so shard by job ID hash into, say, 16 sorted sets (about 2.2 GB each), with one scheduler owning each shard.
• Poll loop: each shard's owner runs the claim script every 100 ms with LIMIT 500, a capacity of about 5,000 per second per shard, far above the peak of about 180 per shard.
• Durability: Redis with AOF everysec can lose about a second of new schedules on a crash. For payment retries, keep the source of truth in Postgres or DynamoDB and use Redis only as an index.

Implementation options

Correctness checklist

• Every job has a stable ID, and workers deduplicate on (job ID, scheduled time).
• The reaper returns jobs whose lease has expired to the scheduled state.
• The scheduler owner per shard is fenced (a lease epoch stored with each claim).
• Clock skew: compute "now" on the store where possible (Redis TIME, or now() in the database), not on each client.
• Missed-run policy for recurring jobs after downtime: run once, run all missed, or skip.

How real systems do it

• Airbnb's Dynein keeps delayed jobs durably in DynamoDB and hands due jobs to SQS for execution. Slack rebuilt its job queue by putting Kafka in front of Redis, so that job enqueues survive Redis failures.
• Kafka uses a hierarchical timing wheel in-process for request timeouts (purgatory). It is not used for durable scheduling.