Under the Hood: Orchestrating Complex Business Processes
Real-world engineering workflows (e.g. customer onboarding, food delivery fulfillment, ML model training pipelines) consist of multiple steps that must execute across heterogeneous services. These steps have dependencies forming a Directed Acyclic Graph (DAG).
The Temporal Event-Sourced Orchestration Architecture
Under the Hood: Deterministic Workflow Replay
Traditional orchestrators store current state in database tables (e.g. step = 'STEP_3'). Modern engines like Temporal and Cadence use Event Sourcing:
- Workflows are written as standard code (Go, TypeScript, Java). Workers normally keep running workflows cached in memory and apply only new events. When a worker fails or restarts, or a cached workflow is evicted, a worker rebuilds the workflow state by replaying the recorded event history from the beginning.
- During replay, previously executed external calls (Activities) are not re-executed; their recorded results are returned instantaneously from the history log.
- The Determinism Rule: Workflow code must be 100% deterministic (no random UUID generation, direct system clock reads, or direct HTTP calls inside the workflow function). Non-deterministic operations must reside in Activities, or use the SDK's deterministic equivalents (for example
workflow.Nowandworkflow.SideEffectin Go,Workflow.currentTimeMillis()andWorkflow.randomUUID()in Java).
Worked example: an order saga with compensation
The Temporal history for a failure path might read: WorkflowStarted, ActivityScheduled ChargeCard, ActivityCompleted ChargeCard (txn t_55), ActivityScheduled ReserveInventory, ActivityFailed ReserveInventory (OutOfStock), ActivityScheduled RefundCard, ActivityCompleted RefundCard, WorkflowCompleted (failed). If the worker crashes after ChargeCard completes, a new worker replays the history: the ChargeCard call returns t_55 from history without charging again, and execution continues at ReserveInventory.
Engine comparison
| Engine | Model | Best for | State storage | Watch out for |
|---|---|---|---|---|
| Airflow | Python-defined static DAGs, scheduled runs | Batch ETL, ML pipelines | Metadata DB of task states | Scheduler latency of seconds. Not for per-request flows |
| Temporal / Cadence | Workflows as code, event-sourced replay | Long-running business processes, sagas | History per workflow (Cassandra, Postgres, MySQL) | Determinism rules, history size limits (about 50,000 events), versioning |
| AWS Step Functions | JSON state machine (Amazon States Language) | Serverless glue on AWS | Managed | Per-transition cost. Standard workflows run up to 1 year |
| Argo Workflows | DAG of Kubernetes pods | CI/CD, ML training on Kubernetes | Kubernetes CRDs | Pod startup overhead per step |
| Choreography (events only) | No central orchestrator | Loosely coupled, simple flows | Spread across services | Hard to see or change end-to-end flow |
Operational concerns
- Versioning: changing workflow code while old executions are in flight breaks replay. Use Temporal's patching or versioning APIs, or run old and new side by side.
- Continue-as-new for long-lived workflows (subscriptions, polling), so history does not grow without bound.
- Compensation is not rollback: a refund is a new transaction that may itself fail, so compensations need retries and alerting too.
Who uses it
- Uber built Cadence, and Temporal is its fork by the original authors. Stripe, Netflix, Snap and Coinbase use Temporal for payments, infrastructure and CI workflows.
- Airbnb created Airflow for data pipelines. It is now an Apache project.