Skip to content

How generators are built

The question. Every Comline generator today is hand-written Rust that pushes strings: generate_rust and generate_typescript are 1,120 and 1,083 lines of push_str over frozen IR. That was a proof of concept, not a commitment. Should generation keep that shape, move to templates, move to syntax trees, move out of Rust entirely — or stop being Comline's job at all?

Five proposals answer it, written to the same shape so they can be read side by side. Proposal A is now accepted and shipped — see What shipped at the end.

Approach In one line Assessment
Proposal A Shared resolution kernel a symbol/module layer in comline-codegen; generators map a resolved model to syntax accepted — shipped
Proposal B Templates per-language template files replace push_str complementary to A, not a substitute
Proposal C Syntax trees build a real AST per target and pretty-print it narrow value; one cheap piece worth taking
Proposal D Target-language plugins protoc-style: each generator is a program in its own language blocked by a current constraint
Proposal E Adopt an existing model translate Comline IR to Smithy/protobuf and inherit their generators assessed, not recommended

What the evidence says

The question is usually framed as how should output be spelled — strings, templates, or trees. Comline's own defect history says that is the wrong axis.

Every generation bug found so far was a missing-model bug:

Defect Cause
Cross-schema references emitted as bare names with no imports — fixed in comline-rust#28, then independently again in comline-typescript#32 no symbol resolution
use ns::Name as Alias resolving to crate::ns::Alias — fixed in comline-rust#39 no symbol resolution
error_type_names same-file-only in both generators, falling back to UnknownError<n> no cross-schema lookup
ModuleTree and NamespaceTree written twice, each refusing different things no shared module model
lib/foo accepted by the Rust generator, clobbering the crate root — only the exact name lib was guarded no shared module model
Framing selection drifting across three implementations until centralised in annotations::canonical_value duplicated knowledge
map_str_type, manifest dependency logic, and external_std filtering each written twice duplicated knowledge

Now test each approach against that list. Templates would have prevented none of them. Syntax trees would have prevented none of them — a field typed Request with no import is syntactically perfect TypeScript. Those tools prevent malformed output; every defect above was well-formed and semantically wrong.

What prevents them is a layer that knows where a symbol comes from and records the dependency when you name it. That layer was hand-rolled twice already, as build_import_map plus a used-name set.

The safety net is shaped wrong, too. conformance::Fixture is one schema and one namespace, so the corpus cannot express a cross-schema reference at all — which is why the same bug shipped twice under a green test suite.

The marginal cost of a language is the other symptom. generation/lib-gen holds four abandoned starts — c, lua, luau, python, between 3 and 108 lines. Adding a target currently means re-deriving module trees, import resolution, primitive maps and manifest emission by hand. Nobody finished.

One constraint narrows the field

playground/wasm compiles the generators to wasm32 and runs them in the browser, deliberately: its manifest says the playground runs that code, "not a re-implementation."

That is what blocks Proposal D, otherwise the strongest strategic option — it would let a TypeScript developer write the TypeScript generator with TypeScript's own tooling, and would retire the git-rev pin cascade across seven repositories. It also rules out heavyweight AST toolchains in C and kills E, whose toolchains are JVM-based.

Two facts make the remaining options cheaper than they look: Handlebars is already a dependency of comline-core (utils/templating.rs, used for layout) and therefore already proven under wasm; and syn + prettyplease are pure Rust, so Rust output could be parsed and re-printed for free formatting and a syntax-validity check.

The recommendation

Take A, and take it in two steps — the corpus first, because it both proves the diagnosis and guards the refactor that follows. B becomes cheap and safe afterwards and can be decided on its own merits; the formatter half of C is worth taking regardless. Keep D reachable by keeping the kernel's request boundary serialisable — core already speaks CBOR — rather than by building it now.

What shipped

Both steps are done, in the order the recommendation gave.

Step 1 — the corpus first. conformance::Fixture became a package rather than a single schema (generation#47), which is what let the corpus express the thing it previously could not: a cross-schema reference. Five fixtures went in with it — cross_schema, import_alias, import_glob, nested_namespace, external_std.

Step 2 — the kernel, then one migration per generator. generation#48 added four modules to comline-codegen with unit tests and deliberately no consumers; comline-typescript#34 and comline-rust#40 then moved each generator onto them, one reviewable diff each against a stable target.

Module Holds
symbols Origin, SymbolTable, UseTracker, error_names — what a bare name refers to, and what a rendering reached for
modules the TreeLayout trait and NamespaceTree — the lib-mode module tree, parameterised by target
stdlib is_std_namespace and split — std's implicit presence and when to declare the dependency
naming pascal, camel, screaming_snake — three conversions that were character-identical copies

The kernel resolves; each generator renders. Rust writes crate::<ns>::Name inline and has no import block; TypeScript writes the bare name plus an import type header. That is not duplication left behind — it is the one thing the two targets genuinely disagree about, and the split now says so. TreeLayout is down to four target decisions; a fifth appearing there would be the signal the sharing was wrong.

What it cost and closed. The TypeScript generator lost 199 lines net and the Rust generator 55, with every conformance golden byte-identical in both cases. The module-tree row above closed on the way past: the shared tree checks a namespace's first segment, so lib/foo is refused as well as lib.

One claim from the analysis above did not survive contact. The TypeScript generator expanded glob use declarations itself, and that was dead code: core's resolve_use_declaration already emits the ::* marker plus one leaf per declared name whenever it resolved the namespace, and it resolves against the exact collection the CLI builds a request's schema list from. The test covering that expansion was asserting against IR core never produces — the same hand-built-fixture trap that produced a false "glob bug" during the corpus work.

Still open, by choice. B is now cheap and safe and can be decided on its own merits; the formatter half of C is still worth taking; D stays reachable while the request boundary remains serialisable.