Module runner 

The live runner: a task, run by a real coding harness, once without Majordomus and once with it, from the same fixture, with the same model, judged by the same hidden tests.

For each run the runner:

1. takes the provenance the record will carry — the repository’s revision, the digest of the suite’s freshness inputs, the harness’s version — before anything else happens;
2. copies the task’s fixture into a fresh workspace, applies the task’s setup patch, and commits it — the control’s starting state. The workspace is repo/ inside a new directory of the system temporary directory named by a digest: nothing in its path says which task, which arm or which benchmark it belongs to, and nothing beside it holds a hidden test, a transcript or another run’s workspace;
3. for the treatment only, installs Majordomus the way a user does (init, update, capture install --provider claude-code) and commits that too;
4. runs every session of the task in the harness (Claude Code, headless, stream-json), with the user’s own settings, MCP servers and skills shut out for both arms alike, each session under a wall-clock limit;
5. runs the task’s verify command, copies the hidden acceptance tests in, runs it again, and checks no test file the fixture started with was deleted;
6. refuses the run if the freshness inputs or the harness changed while it was in flight, deletes the workspace, and writes one EconomicsRun with the provider’s usage as reported, and nothing a session said. A record, once written, is never written over.

The harness is a parameter: a test passes a replaying stand-in and exercises the whole pipeline — workspace, sessions, gates, record — without a provider, a credential or a cent. The transcripts stay in the work directory, <work dir>/<suite>/<run id>/, outside the repository and outside every workspace.

use majordomus_cli::economics::runner::run_id;
assert_eq!(run_id("vat-rounding", "baseline", 2), "vat-rounding--baseline--r2");

Structs

RunOptions

Everything a live suite run is told. The work directory must lie outside root, and run refuses the options otherwise, because the transcripts it collects there quote prompts and code that never belong in the repository’s history.

Functions

check_references

The reference solution of every task applied to its fixture: the hidden tests must fail on the starting state and pass on the reference, or a failing run would say nothing about the agent. Returns one line per task, and Err listing every task that breaks the rule. Each task is prepared under <work_dir>/references/<task> with the control’s starting state and removed once checked; no harness and no provider is involved, so the check costs nothing to repeat.

record_path

Where a run’s record is written: one JSON file per run id under the suite’s directory of the evidence tree. run skips a job whose record already exists here, which is what makes an interrupted suite resumable, and never writes over one.

run

Run a live suite: every job not yet recorded, parallel at a time, control and treatment of one task and repetition side by side so that both meet the provider under the same conditions. Returns the records written and the jobs that failed to produce one. Refuses, before any workspace exists, force (a recorded run is never written over), an unknown suite, a suite that is not live, a task the suite names but nobody declared, and a work directory — or a system temporary directory — inside the repository.

run_id

The id of a run: task, variant, repetition. One file per id; a recorded run is never overwritten.