← DDIA

Chapter 04 · Encoding & Evolution

Encoding & Evolution

You deployed a new field. Half your servers are old. Nothing broke. Why?

Two versions of your code are alive at once, reading each other's data.

Three acts: 1. wire 2. directions 3. rules

On the wire

Same record. Same three fields. Three encodings.

No schema. Field names ship in every payload.

{ userId: 1234, userName: "Martin", favouriteNumber: 1337 }
{"userId":1234,"userName":"Martin","favouriteNumber":1337}
schema · none · self-describing 0 bytes

Binary. Field tags ride the wire; names live in the schema.

{ userId: 1234, userName: "Martin", favouriteNumber: 1337 }
1 userId · int32 2 userName · string 3 favouriteNumber · int32
08 D2 0912 06 Martin18 B9 0A
tags on wire · names in schema 0 bytes

Binary, no tags. Writer's and reader's schemas resolve field by field.

{ userId: 1234, userName: "Martin", favouriteNumber: 1337 }
writer userId · long userName · string favouriteNumber · long
reader userId · long userName · string favouriteNumber · long
D2 090C MartinB9 0A
no tags · resolved by writer + reader schema pair 0 bytes

Two directions

Rolling deploy: old and new code trade rows at the same time.

v1
v1
v1
v1
v1
v1
t0all v1
t1rolling
t2all v2
database · shared record
id: 42 name: "Bill" favNum: 1337
↑ forward old code reads new field · ignores it
↓ backward new code reads old row · fills default

both directions must hold, at the same moment

Takeaway

Old code writing to new. New code reading old. Both directions must hold, at the same moment.

Add a field safely.

01

Optional or default

New fields ship as optional or with a default value.

OLD CODE writes without the field · reads still work.
NEW CODE reads old rows · fills in the default.
BREAKS IF you ship a required field with no default.
02

Tag identity is forever

Field tags (Protobuf) or names (Avro) never get reused. Retire them; don't recycle.

SAFE mark the tag reserved · pick the next free one.
UNSAFE reuse tag N for a new field of a different type.
WHY the wire is stable · old data still uses the old meaning.
03

Readers tolerate the unknown

Unknown fields are preserved or ignored, never rejected.

NEW WRITER adds field 47.
OLD READER skips field 47 quietly · round-trips it if writing back.
BREAKS IF the reader throws on any field it doesn't recognise.

Follow all three and the fleet can be any mix of versions at any moment.

── Reference ──

Six formats · one evolution matrix · four things to remember.

Encoding formats, side by side
Format Bytes* Schema Evolution Human-readable Tags on wire Notable users
JSON 81 none tolerant readers yes field names web APIs everywhere
XML ~150 XSD (optional) tolerant readers yes tag names legacy enterprise, SOAP
MessagePack 66 none tolerant readers no field names Fluentd, polyglot msgpack libraries
Thrift 59 .thrift IDL tag numbers stable no field tags Facebook, Cassandra RPC
Protobuf 33 .proto IDL tag numbers stable no field tags Google, gRPC, Kubernetes
Avro 32 .avsc writer/reader schema resolution no none (schema ships) Kafka, Hadoop, Confluent

* Bytes for the sample record used in DDIA §4.1 Figures 4-1 through 4-6.

Safe schema evolution operations by format
Operation JSON / XML Protobuf / Thrift Avro
Add optional field safe safe (with default) safe (with default in reader)
Remove optional field safe safe (leave tag number reserved) safe (with default in reader)
Add required field breaks forward compat breaks forward compat writers must upgrade first
Change field type dangerous limited (varint-compatible only) limited (per Avro resolution rules)
Rename field breaks tag stays; symbol can change field aliases supported

JSON / XML → schemaless, verbose, universal; tolerant readers do the compatibility work.

Protobuf / Thrift → schema with tag numbers; tags on wire enable evolution.

Avro → schema per file; writer's schema ships; no tags on wire.

Backward vs forward → new code reading old data vs old code reading new data; both must hold during rolling deploy.