Token economics are measured from matched runs and published only past a declared rule

82. Token economics are measured from matched runs and published only past a declared rule

Context

The economics document refused, from the first version, to state any token saving: nothing measured one. The planned milestones (execution-telemetry, cost-per-accepted-outcome) described what measuring would take and stopped there. Meanwhile the value Majordomus is most often assumed to have — fewer tokens per task — remained a belief, and a belief is exactly what this repository exists to turn into a checked fact or retire.

Three things made a measurement possible now. Claude Code reports provider usage per request in its stream-json output, headless and without an API key. The context compiler (devcontext) exposes what it considered and what it selected. And the claims matrix already separates guaranteed from planned statements, so a number can be bound to evidence rather than typed.

Decision

1. One calculator. apps/majordomus-cli/src/economics reads declarations and raw run records and derives every pair, metric, interval and verdict. The CLI, HTTP, OpenAPI, MCP, the Cockpit, the generated report and the site are projections of summarize; none computes a number. Derived values are never stored; raw facts are.
2. Five measurement classes — observed, counted, derived, estimated, counterfactual — on every metric, and a derived metric names the class of its inputs. Context reduction (counted) and total-token change (observed) are separate metrics that are never merged.
3. Matched pairs judged by hidden tests. A live suite runs each task in a control (the fixture as a competent team keeps it) and a treatment (the same fixture with Majordomus installed as a user installs it). A pair counts only when both runs passed every success gate; every other pair is reported with its reason. Reference solutions prove the gates can be failed and passed.
4. A declared publication rule. The methodology states, before any evidence, the minimum valid pairs, categories, repetitions, valid-pair rate and interval width, and that evidence must be current and clean. Below the rule the verdict is generated as "no verified total-token-savings claim", with the preliminary observation labelled.
5. Freshness by digest. Each record carries a digest of the files its suite depends on; a change makes the evidence stale and removes any guarantee bound to it.
6. A gate on prose. economics check refuses a quantity next to the economics vocabulary in hand-written prose, and a bound claim marked guaranteed whose metric does not stand.

Consequences

• The repository can now say what it measured, including that Majordomus costs more on some tasks, which the rule requires to be shown rather than averaged away.
• Live evidence costs provider usage and wall-clock time; it runs on request (economics run), not on every commit. The deterministic suite and the whole calculator are tested without a provider.
• The treatment excludes the MCP server and git hooks. Measuring them is a new treatment and a new methodology version, not an edit.
• The tokenizer that counts context (o200k_base) is not every model's; counted numbers say which one they are, and are never compared with provider-reported tokens.