Economics

Majordomus does not state a token-saving percentage it has not measured, and this document contains none. The numbers are in generated/economics.md, written by majordomus generate economics from recorded runs, and on the site's economics page, which renders the same generated data. Typed into prose, a number about tokens, context or cost is refused by majordomus economics check.

The question

For the same task, from the same repository state, with the same model, the same tools and the same acceptance tests: how many tokens does a coding session consume with Majordomus installed and without it?

The unit is tokens per successfully completed task. A run that saves tokens by failing is not a saving, so a pair of runs counts only when both passed every success gate.

What is measured, and how each number is labelled

Every number the subsystem produces carries one of five classes, and a number computed from several inputs is never stronger than the weakest of them:

Two measurements exist, and they answer different questions:

• Total-token reduction (effective_token_reduction, derived from observed usage). Matched live runs of a task corpus, control against treatment. This is the only kind of number that could support a claim about what Majordomus saves.
• Context selection (context_reduction_ratio, derived from counted tokens). For every issue of this repository's plan, what the context compiler selected out of what it judged relevant. No model is involved. It is not total savings, and every surface says so.

Control and treatment

The canonical definitions are in .ai/repo/benchmarks/economics/methodology.yaml.

• Control, without Majordomus. The fixture repository as a competent team keeps it: a README, a short CLAUDE.md pointing at the conventions, conventions and decision records in docs/, a green test suite. Claude Code runs headless with the model the suite names, its built-in tools and permissions, and no MCP server, user settings or skills.
• Treatment, with Majordomus as a user installs it. The same fixture after majordomus init, majordomus update (the bootstrap added to the fixture's own CLAUDE.md as a generated region) and majordomus capture install --provider claude-code, committed. Everything else is identical. The MCP server and the git hooks are deliberately excluded from this first treatment.

Both arms are isolated from the operator's own configuration in the same way, so neither inherits instructions, connectors or memory the other lacks.

The task corpus

Seven tasks in .ai/repo/benchmarks/economics/tasks/, each standing for a development pattern this repository sees: a defect whose fix is written down in a decision record, a small additive feature, a change across four modules and a persisted format, a suite broken by a rename where a convention decides the right fix, a versioned storage change, a user-reported regression, and a feature built across two sessions where the second starts from what the first left.

Each task names its fixture, an optional setup patch, a verify command, hidden acceptance tests copied in only after the last session, and a reference solution. majordomus economics references proves, with no model involved, that every hidden test fails on the task's starting state and passes on its reference — so a failed run says something about the agent, not about the test.

From runs to a statement

1. majordomus economics run --suite pilot runs every task in both arms and records one file per run: the provider's usage per request and per model, as reported (message by message, deduplicated), the harness's own cost projection, tool calls, and the verdict of each success gate. Nothing a session said or read is recorded. A recorded run is evidence and is never written over: a run already recorded is skipped, and --force is refused. A run that should not count is excluded in the methodology with a reason, and another repetition is recorded in its place.
2. The calculator pairs control and treatment runs of the same task and repetition, checks that they are comparable (suite version, the model requested and the models that answered, fixture, harness version, configuration, revision, session count) and valid (both succeeded, both report usage), and names the reason for every pair that is not.
3. Per valid pair: reduction = 1 - treatment / control. Every per-pair and aggregate token figure on every surface is this reduction: positive means Majordomus used fewer tokens, and a negative reduction means Majordomus used more tokens. It is reported as it is, and no surface flips its sign.
4. Across pairs: the distribution (median, quartiles, 95th percentile, mean, standard deviation) and a percentile bootstrap interval of the median with a recorded seed, and the same by task category, session count and complexity. The repetitions of one task are not independent, so the interval resamples whole tasks rather than single pairs; each interval names the method it was computed with.
5. The verdict: a quantitative total-token claim is allowed only when every threshold of the methodology's publication rule holds — enough valid pairs, enough task categories with enough pairs each, enough repetitions of every task in each arm, a high enough valid-pair rate, a narrow enough interval, current evidence, and no run from a dirty tree. The rule is evaluated for the primary metric over all the evidence, so only that metric, unfiltered, can be verified. Below that, every surface states that no verified total-token-savings claim is available, beside the preliminary observation labelled as preliminary.

No run is excluded unless the methodology names it with a reason, and the primary metric is then shown with and without it (effective_token_reduction.including_excluded).

When evidence goes stale

Every record carries a digest of the files its suite depends on: the methodology, the corpus, the fixture, and the parts of Majordomus that shape what a treatment session is given (the skeleton init writes, the provider templates, the briefing and handover code). When any of them changes, the evidence is reported as stale, and a claim bound to it loses its guarantee until the suite is run again. Evidence recorded under another methodology version is incompatible and never pooled.

The proof plan

The token benchmark answers one question. The proof tests that are to settle the rest (accepted results, wall time, human interventions, CI reruns, spend, recovery, onboarding) are registered in the plan file proof-plan.yaml under .ai/repo/benchmarks/ (schema proof-plan/v1) before any of their records exists, and economics::proof::load validates the plan fail-closed: it returns the plan or every finding, and an absent file is the finding "no proof plan is registered", never an empty valid plan.

Each test names its question, the claims of docs/CLAIMS.yaml it settles, its dimension, the words a result is stated in, and its instrument: built, or planned with the issues that build it. Once its experiment is designed, a test gains a protocol: a version, the time it was registered, a change log entry for every version, and a decision of one of three kinds.

A protocol carries registered_digest: the SHA-256 of its own canonical form, which is the protocol as written without that field, with object keys sorted and no whitespace. The same function, economics::proof::digest, digests a suite's registration block. The validator refuses a digest that does not match, naming the test and both digests, and also an unknown dimension or decision kind, a duplicate test id, a claim docs/CLAIMS.yaml lacks, a paired protocol whose suite or registration digest does not resolve, and a version without a change log entry.

A record counts toward a test only when it carries that test's protocol digest and was recorded after registered_at (economics::proof::counts_toward); otherwise the answer says why not: no protocol yet, another protocol, too early, or a time it cannot read. Each protocol is frozen by its own digest rather than by the plan file's, so registering one test's protocol never makes another test's records stale. For the same reason the plan file is a freshness input of no suite, and the validator refuses a suite whose freshness_inputs could select it, read as git ls-files reads them: a directory above it, a wildcard or pathspec magic that matches it, or a list with no including pathspec, which selects every tracked file.

Reproducing it

majordomus economics references         # every task can be failed and passed; no model
majordomus economics measure            # the context suite; deterministic, no model
majordomus economics run --suite pilot  # live sessions; needs a Claude Code login, spends usage
majordomus economics summary            # every figure, recomputed from the recorded runs
majordomus economics explain effective_token_reduction
majordomus economics check              # refuse an unsupported claim
majordomus economics tasks --format json  # each task's wall time, refusals and episodes

The same answers are served at GET /api/v1/economics, /api/v1/economics/explain, /api/v1/economics/runs, /api/v1/economics/check and /api/v1/economics/tasks, by the MCP tools majordomus_economics*, and on the Cockpit's Economics page. All of them are projections of one calculator, apps/majordomus-cli/src/economics.

Each task's delivery economics

majordomus economics tasks answers one record per task. It folds the ledger of every checkout that git worktree list reports, together with the archives a ledger rotation left beside it (.ai/local/state/ledger.<time>.jsonl.archived). Each record states:

• the task's issue and intent, checkout and branch;
• its first task.started and its last task.finished (or, while open, its last task.refused), and the wall seconds between them;
• how many times finish refused it;
• the episodes that carried it, each with the span of its lines about the task.

A task with no task.finished line reads open, never finished.

Interventions, CI reruns, spend and acceptance are on every record as not_observed, with the reason and the issue that will supply each one (I3377 to I3381). There is no value, and that is not a zero.

The answer also lists every ledger file it read, with its line and skipped-line counts, and every checkout it could not read, with the reason. --since and --until filter tasks by their start; a ledger is still read in full.

What this does not measure

• A large repository. The fixture is small by design, so that verifying a run takes seconds; context selection matters most where there is much to select from, and that is not measured.
• Other harnesses and models than the ones a suite names. Provider counts are never compared across providers.
• The MCP server, mesh coordination between parallel workers, and the finish contract's effect on rework. Each would be its own treatment.
• Money. Cost is the harness's own projection at run time, not a bill, and historical runs are never re-priced.
• A corrected treatment. The treatment is Majordomus as a user who installs only the shell tool gets it, and its generated bootstrap advises commands that installation cannot run (majordomus worktree). What a session spends on them is part of what that user gets, and it is measured as such, not corrected for.
• The corpus as it is now. The context suite counts the corpus at the revision it recorded, not at the current one; a later change to the plan or to the files it selects from moves nothing until the suite is measured again.

The hypotheses

The benchmark was built to test hypotheses recorded in the methodology before any evidence existed. They are hypotheses, and the generated report lists them as such beside whatever the evidence shows.