Skip to content

The documented command that no longer works

Examples are written once, in prose, and nothing ever executes them again.

→ pasted a command from the documentation and watched it fail

cost when it happens
medium
how often
common
area
documentation
verification
recognised by
open-source-maintainer
platform-team
agency
ai-native-team

The moment

Somebody copies the command from the getting-started page. It exits 2 with a usage error on an option that was renamed in the spring. The page has not been wrong for long enough for anyone to complain, and it has been wrong for everyone who tried it.

Why it happens

An example in a document is a string. Nothing runs it, so nothing can notice when it stops working. The command it describes lives in code that changes for good reasons, and the string does not participate in those changes.

Why a better model does not fix it

A worker reading the documentation reproduces the broken invocation faithfully, then diagnoses the failure it caused. The documentation was the input, and it was wrong; no amount of capability recovers from a wrong specification without paying for the detour.

What it costs

The most expensive minute in a project: the first one, for someone deciding whether this is maintained. After that, a steady drip of support questions whose answer is "the docs are out of date".

What Majordomus does

The command line is declared once, and the examples are declared beside it as argument vectors with what each must show — an exit code, a fragment of output, a JSON pointer that must resolve. The crate's own tests execute every one of them against the built binary in a disposable repository. The reference document, the machine-readable command model and the pages on the site are renderings of that same declaration, so a rename reaches all of them at once. A command that can be run and carries no executed example does not pass validation.

Before and after

before   docs: majordomus check --explain-all      (renamed in April)

after    every example is argv + Expect, executed by the suite:
         $ cargo test --test cli_examples
         running 27 examples against the built binary ... ok

What it does not do

It does not check prose, and it cannot tell whether an example is a useful one. It makes an example that no longer works a test failure rather than a reader's discovery.

What this looks like

Concrete situations, one per audience. Each is declared in the moment's front matter, so the before and the after are data rather than prose a page could drift from.

  1. The first command a newcomer runs agency

    before
    The quick-start command fails on an option that was renamed a release ago; the newcomer''s first impression is that nothing here is maintained.
    after
    Every documented example is an argument vector the test suite executes against the built binary, with what it must print.

  2. A rename that reached five documents platform-team

    before
    An option is renamed; five documents show the old one and are found one complaint at a time.
    after
    The reference is rendered from the declaration and the examples beside it; there is no second copy to miss.

  3. A worker following the documentation ai-native-team

    before
    A worker builds a script around a documented invocation that has not worked for months.
    after
    A command with no executed example does not pass the crate''s own gate, so the documentation cannot get ahead of the binary.

How you would know

The observable symptoms this moment declares. They are the questionnaire on the index and the input of majordomus why diagnose; nothing else defines them.

  • A command copied from the documentation failed this month. example-fails
  • The examples in the documentation are not executed by anything. examples-never-run
  • An option was renamed and the places that show it were updated by hand, or not at all. flag-renamed

Where this lives in the tool

Everything below is read out of this moment's own front matter and resolved against the repository. A name here that did not exist would fail validation.

the commands that answer it

the capabilities of the executable that answer it

what it supervises — derived from the claims below

the claims that back this page, and the evidence behind each

the rules that govern it

  • project.native-cli-documented
  • project.use-case-evidence
  • majordomus.command-surface
  • majordomus.command-coverage

the use cases that show the way out

If this one is familiar, so is the next

What this moment names, what names it, and what shares its area, audience or tags. The second and third are derived; only the first is written down.

All 38, and how they connect to the tool →