Steps
Status: accepted · decided 2026-11-13 · tracked in core#268 · part of Migrating stored IR across format versions
The question. What is a migration made of, so that it is accurate for every old format, never loses what a version recorded, and never claims to know what the old format did not?
The model
A migration moves a whole store from format N to the current format C, one step at a time. Steps never touch bytes or hashes. They work on a model of one version, which the format's codec decodes from the store and encodes back:
bytes ──decode_N──▶ Version_N ──step N→N+1──▶ Version_N+1 ──…──▶ Version_C ──encode_C──▶ bytes
(frozen types of N) (frozen types of C)
A Version_N holds the commit's own fields (version, release, author, timestamp, message), the decoded
payloads (schemas, vendored dependencies, config, docs, facts) and the unknown list described
below. A reference to another version, a parent or a release.since, is a version id (its place in
the chain), not a hash. The engine turns ids back into hashes when it encodes, so a step author never handles a
hash and can't break the Merkle structure. See hashes.
The intermediate formats never reach the disk. A store goes from N to C in memory, version by version, and is written once.
A step
A step N → N+1 is three things, written against the frozen types of N and N+1
(formats), never against today's types:
| Part | What it is |
|---|---|
up |
A function from a Version_N to a Version_N+1. Total: it returns a value for every value the format's types can express, not only for what a real build produces. Deterministic: the same input, the same output, on every platform. It has no I/O, clock, environment, randomness, thread or hash-map order. Every map in the model is ordered. |
down |
The witness: a function from the image of up back to Version_N, with down(up(v)) = v exactly. See the witness. |
unknown |
The list of paths in Version_N+1 that up writes as unknown because N never recorded them, each with the reason. Written down in the format's SCHEMA.md and read by the tests. |
A step may read only the version it is given and, for facts that live in history, the versions before it in the same
chain. It may not recompute what the current core would compute (a guarantee set, a wire number, a shape hash): that
is code that changes, and a step is frozen. What a version recorded is carried across as data, and what has to be
recomputed is recomputed on read, by whichever core is reading. For example the facts blob is a record of what
the build computed under the rules in force; a step carries it, and history_of_commit already recomputes it when it
is absent or unnumbered.
Steps may not delete user data
up keeps every field of N. A field that N+1 renamed or retyped is mapped. A field that N+1 has no place for is
kept in the unknown list with its original value (see below). The witness checks this.
Unknown
Where N+1 needs something N never recorded, up writes an unknown. It never writes a default that looks
complete: a framing the author never chose, a length T[] for an array whose length was lost, an empty set of
causes for a guarantee whose causes weren't stored. This is the rule
core#251 set for dependency configs ("read as unknown, never as
defaults that look complete") and the vocabulary's for a
dependency that can't be seen, applied to a whole version.
An unknown has two possible carriers. Proposed: both, in this order of preference.
- Typed, in the new format. Where N+1's own type has room for it, the field is
Known<T>: a value, orUnknown { reason }. The compiler then forces every reader to handle it. Format 4's chains are the example (format 4). A format that adds a field is written so that anything a later step might have to leave unknown is already aKnown<T>. - An
unknownentry in the root tree, for a retype whose new type has no room: a blob listing, for each unknown, itspathinto the model, thestepthat wrote it, thereason, and theoriginal, the bytes N recorded, where N recorded something. The field itself holds a placeholder that the entry marks as one. This is what lets a step exist where the new type has no "unknown" variant and can't gain one without a bump: the entry keeps the old value (say, afloatthat N froze without a width) beside a placeholder. Format 3'sTypeRefis an example of a type with no room, and a test-only format has one on purpose (proof).
unknown is an additive entry, a row in format 3's ledger with the absence reading none. A version without it
has no unknowns.
Ordering constraint. A reader that doesn't know the entry would read a store that carries unknowns as complete. So
the readers of unknown ship before any step that writes one. See the rollout.
Reading an unknown
Consumers don't see the entry or the placeholder. The accessors on a version return Known<T>, with no Deref and
no default, so an unknown can't be forgotten. Every use then has a rule, and it is the strictest reading:
| A consumer needs | The field is unknown | Result |
|---|---|---|
| a protocol's given-up guarantees | the set or a cause | counts as given up; the build that depends on it refuses |
| a protocol's framing or format | the declaration | the strictest facts, as Encodings does when complete is false (core#262) |
| a bound | the maximum | unbounded |
| a type, compared | its width or length | breaking, or "can't tell" if the rule has that |
| a span | where it was written | no span; diagnostics name the version instead |
| generating code or documentation | anything the output prints | fails closed: the generator or doc stops with "unknown (migrated from format N)", format 4 |
A comparison that read an unknown says so: it lists the version and the path in the same unseen list core#262
added, and its bump is at least a patch. See history.
Some things are never unknown, because without them a version isn't one: its version, its release record, its
parents, and the schema units' names. A step that can't produce them fails, and the migration refuses.
The witness
down(up(v)) = v is the proof, on every real version, that a step lost nothing. Decided: every step has one. It is
cheap to write for the steps that are likely: one that wraps a value, renames a field or adds a field with an unknown
has a trivial inverse, and a step that moves a value into the unknown list keeps the original there so down can put
it back. A step that can't state its inverse is exactly the one to look at.
It is checked in two places. The tests run it over fixtures and random values (proof). And the engine runs it over every version of the user's own store that it migrates, before it swaps anything (running), so a store the corpus didn't predict is still checked. The witness also has a second use, for pins.
Chains
migrate(N → C) = up(C-1→C) ∘ … ∘ up(N→N+1). A later step may resolve an unknown an earlier one wrote, only from
data in the version. The unknowns of a chain are the union of its steps' lists minus what a later step resolved,
and the engine writes that list.
The engine has one chain per store and one version at a time. It does not support skipping steps: a "shortcut" from 1 to 3 is a second implementation of the same function, and the composition law (proof) exists to catch the two disagreeing.
What cannot be migrated
Today's formats are out of scope (index): they reset at 1.0. Until then,
a store the system can't migrate is never touched by a command on its own. The command that finds it stops and says why;
the user decides, and the only thing that moves it is the user's comline store reset, which renames it to a backup.
| Store | Why | What a command does |
|---|---|---|
| A store in a pre-1.0 format with no step to the current one (the unstamped bincode store, formats 1 and 2, older rows of format 3) | Nobody writes those steps. Format 0 is positional: a decode of the wrong layout can succeed and give a different variant, and nothing in the store can check a decode against. | Stops, naming comline store reset (running): the store is renamed to .comline.format-N.bak/, a new one starts at the next build, deps-cache/ is kept |
| An unstamped store that decodes as CBOR | The stamp was lost, so the format isn't known. | Stops, unless the user gives one: comline store migrate --assume-format N. The decode must succeed for every object |
| A damaged store (a missing or corrupt object, #182) | A migration needs every object. | Stops, naming the file and how it failed |
| A store newer than the binary | The binary can't read it. | Stops, naming the format and the comline that reads it |
A history can later be rebuilt from Git with comline store replay (proposed; may not be worth building). It builds a
new history with the current core, one build per release tag or commit the user chooses, never claims to be the old
one, and numbers versions by the current rules.
Support
| Format | Status |
|---|---|
| 1.0 and later | every format migrates to the next: that is the guarantee this system exists to give |
| 3 (as it stands when the framework ships) | 3→4, the first client: it carries real stores, and exists to prove the system |
| 0 to 2, and older rows of 3 | not migrated; the user resets them (comline store reset), and the 1.0 release refuses them |
| test-only formats | in the test tree, never written by a release: the proof |
Where the code lives
Decided: a workspace member comline-ir-formats (publish = false at first) beside oracle/, holding one module
per format with its SCHEMA.md, types, codec and ledger, and the steps between them. It has no dependency on
comline-core, the way the oracle has none, so that a frozen type can't reach for today's. A step N → N+1 lives
beside the frozen N+1 types. Those exist from the day N+1 is introduced, and are kept equal to the live types by the
live ≡ frozen(current) test until N+2 seals them.
Core depends on the crate to read old stores. Test-only formats live in its test tree, not in the shipped modules. The
engine and the CLI commands are in core. The crate's only dependencies are serde, ciborium, blake3 and
lz4_flex, so the IR-only build, which the wasm32 consumers use, can carry it: only for in-memory, read-only
migration, never to write a store.
Decided here
- New formats use
Known<T>for everything they add that a later step might have to leave unknown. Theunknownentry is for retypes that have no room. - The
unknownentry's paths are JSON pointers into the model: the frozen types already derive serde, and a path then reads the same in the format's document and in a test. - The
formatscrate also carries the unknown readers and the steps for in-memory, read-only migration in the IR-only build (wasm), and never the engine that writes.