Skip to content

Green tests over stale generated files

Tests exercise the code and say nothing about the committed outputs derived from it, so the build is green and the artifacts are wrong.

→ had a green pipeline over generated files that no longer matched their sources

cost when it happens
medium
how often
common
area
verification
documentation
recognised by
platform-team
ai-native-team
enterprise
engineering-lead

The moment

Everything is green. The tests pass, the linter is quiet, and four committed files that are generated from the code describe the code as it was before the last change. Nothing in the pipeline looks at them, because they are not code and they are not tests.

Why it happens

Generated files that are committed occupy an awkward category: they are reviewed like source and produced like output. Regenerating them is a step, and a step that is not run by the same thing that runs the tests will be skipped by whoever is in a hurry — which, increasingly, is a worker that was asked to change the source and did exactly that.

Why a better model does not fix it

Regeneration is a build step, not a judgement. A worker that did not run it did not fail to reason; it was never told, and telling it every time is the manual synchronisation the generation was supposed to remove.

What it costs

Reviewers read a stale artifact and approve it. Consumers of the artifact — a site, a client generator, another tool — serve last month's truth. And the tree is no longer reproducible: running the generator produces a diff, which everyone learns to ignore.

What Majordomus does

The derivation graph is written down once, in dependency order, so a person and CI run the same thing in the same sequence. Every committed derived artifact has a check that regenerates it and compares: generate --check names each file that differs or is missing and exits non-zero, and the site's data has the same check against the hash of its inputs. Running the whole derivation twice changes nothing, which is what the check compares against.

Before and after

before   tests: green      docs/generated/*.json: from two commits ago

after    $ scripts/derive-check
         stale: docs/generated/openapi.json (differs)
                site/data/registry/registry.json (differs)
         exit 10

What it does not do

It does not decide what should be generated, and it will not regenerate silently during a check — a check that writes is not a check. It names what is stale and the command that fixes it.

What this looks like

Concrete situations, one per audience. Each is declared in the moment's front matter, so the before and the after are data rather than prose a page could drift from.

  1. A worker that changed the source only ai-native-team

    before
    A worker changes a declaration, the tests pass, and the committed reference derived from it is now a description of the previous version.
    after
    `generate --check` names every artifact that differs from what its source produces, and CI runs it.

  2. A derivation with an order platform-team

    before
    Regeneration is done in the wrong order, so one output is built from a stale input and looks current.
    after
    The derivation graph is written down once, run in dependency order, and proved by re-running it and comparing.

  3. Reviewing an artifact that was not rebuilt enterprise

    before
    A generated document is reviewed and approved while the source it claims to describe has moved.
    after
    The artifact is regenerated in CI and the tree is refused if it differs; approval applies to something current by construction.

How you would know

The observable symptoms this moment declares. They are the questionnaire on the index and the input of majordomus why diagnose; nothing else defines them.

  • A change landed without regenerating the files derived from it. forgot-to-regenerate
  • The pipeline is green and a committed generated file is out of date. green-but-stale
  • Regenerating derived files is a step somebody has to remember. regeneration-is-manual

Where this lives in the tool

Everything below is read out of this moment's own front matter and resolved against the repository. A name here that did not exist would fail validation.

the commands that answer it

the capabilities of the executable that answer it

what it supervises — derived from the claims below

the claims that back this page, and the evidence behind each

the rules that govern it

  • project.derived-files-regenerated
  • project.derived-once
  • majordomus.projection-integrity
  • project.interfaces-are-projections

the use cases that show the way out

If this one is familiar, so is the next

What this moment names, what names it, and what shares its area, audience or tags. The second and third are derived; only the first is written down.

All 38, and how they connect to the tool →