Under the Hood: Asynchronous Schema Evolution
In synchronous APIs, if a client sends an invalid field, the server returns an immediate HTTP 400. In asynchronous event-driven architectures, once an event is published to Kafka or RabbitMQ, it is persisted to disk and consumed by dozens of downstream microservices hours or days later. A malformed or incompatible event stays in the log: consumers can stall on it as a poison message unless they route it to a dead-letter topic, and every replay hits it again. Event Contracts formalize asynchronous payloads.
Schema Registry Enforcement Flow
Schema Evolution Compatibility Modes
- Backward Compatibility (Default): Newer schema code can read events written with older schemas. Consumers must be upgraded before producers. New fields must have default values.
- Forward Compatibility: Older schema code can read events written with newer schemas. Producers must be upgraded before consumers. Fields may only be deleted if they had defaults.
- Full Compatibility: Schemas are both backward and forward compatible. Old and new producers and consumers can operate concurrently in any upgrade order without downtime.
Worked example: adding a field to OrderPlaced
Version 1 of the Avro schema for OrderPlaced has order_id, customer_id and total_cents. The team wants to add currency.
- Add with a default (
"currency", type string, default "USD"). New consumers reading old events fill in "USD" (backward compatible). Old consumers reading new events ignore the unknown field (forward compatible). So this change is fully compatible and can be deployed in any order. - Add without a default: new consumers cannot read old events that lack the field, so Schema Registry rejects it under BACKWARD mode.
- Rename total_cents to amount: not compatible in either direction. Add
amountas a new field, dual-populate both for a migration period, then remove the old one only when no consumer reads it, or publishOrderPlaced.v2to a new topic.
On the wire, Confluent's serializers prefix each Kafka message with 1 magic byte (0) and a 4-byte schema ID. The consumer looks up the writer's schema by ID (and caches it), then resolves it against its own reader schema. That is what lets old and new versions coexist in one topic.
Compatibility modes
| Mode | Guarantee | Allowed changes (Avro) | Upgrade order |
|---|---|---|---|
| BACKWARD (Confluent default) | New schema can read data written with the previous schema | Delete fields, add fields with defaults | Consumers first |
| FORWARD | Previous schema can read data written with the new schema | Add fields, delete fields that had defaults | Producers first |
| FULL | Both directions | Add or delete only fields with defaults | Any order |
| *_TRANSITIVE | Same, checked against all earlier versions, not just the latest | Same as above | Needed when consumers replay old history |
| NONE | No checks | Anything | Coordinated deploys only |
What else belongs in the contract
- Identity and deduplication: a unique event ID so at-least-once delivery can be deduplicated.
- Ordering key: which key determines the partition, and therefore which events are ordered relative to each other (for example all events for one order_id).
- Semantics: whether the event is a notification (go fetch the details) or carries full state, and whether it can be replayed safely.
- Retention and privacy: how long events are kept and which fields contain personal data that must be deletable.
Failure modes
- Poison messages: route events that fail deserialization or validation to a dead-letter topic with the error, and alert.
- Semantic breaks that pass schema checks, such as changing units or meaning; review them like breaking API changes.
- Protobuf evolution: never reuse a field number; reserve removed numbers and names.