Skip to content

majordomus bench

Time every public command of the registry, cold and warm, and compare with the baseline.

Time every public command of the registry, cold and warm, and compare with the baseline.

  • generated-output-mutating
  • system
  • --json

The targets are the registry's public commands; the scenarios are the command fixtures when the suite is present. A run is local evidence; only --write-baseline writes the tracked baseline, and --check refuses a regression over the policy's thresholds.

Syntax

majordomus bench [<command>...] [--samples <n>] [--warmup <n>] [--mode cold|warm|both] [--format text|json] [--no-save] [--list] [--check] [--write-baseline [--force]]

Lifecycle

  1. Set up
  2. Begin
  3. Work
  4. Verify
  5. Conclude
  6. Inspect

Other commands at this stage:doctor, watch, history, search, doctrine, version, knowledge, rules.

Reads

  • share/commands.yaml
  • test/fixtures/commands/
  • .ai/repo/policy.yaml
  • .ai/repo/benchmarks/baseline.json

Writes

  • .ai/local/benchmarks/
  • .ai/repo/benchmarks/baseline.json

Exit codes

Only the codes this command can produce. The full contract is on the commands index.

  • 0 ok
  • 2 usage
  • 10 contract unmet
  • 12 missing artifact
  • 13 internal error
  • 15 refused

Performance as executable evidence

bench times every public command of the registry in a disposable repository, cold and warm, and records the distribution rather than an average. The targets are the registry; nothing here keeps a list.

The targets, derived from the registry

installed
yes

Given this repository

# A repository with Majordomus installed and projections generated, and one commit of work.
"$MJ" init >/dev/null
"$MJ" update >/dev/null
mkdir -p lib docs
echo a > lib/a
echo d > docs/d
git add . && git commit -qm base

Run

$ majordomus bench --list

Output contains

  • ^doctor +read-only
  • ^start +state-mutating

exit 0

Every public command is a target with its class and the scenario it will run; the harness itself is the one public command that is not a target.

One target, one cold run

installed
yes

Given this repository

# A repository with Majordomus installed and projections generated, and one commit of work.
"$MJ" init >/dev/null
"$MJ" update >/dev/null
mkdir -p lib docs
echo a > lib/a
echo d > docs/d
git add . && git commit -qm base

Run

$ majordomus bench version --samples 1 --warmup 0 --mode cold --no-save

Output contains

  • ^version +cold +ok +1

exit 0

A read-only command is run once cold in a prepared repository; the row carries the status of the run and the percentiles of its samples, in milliseconds of the child process alone.

A command that is not in the registry

installed
yes

Given this repository

# A repository with Majordomus installed and projections generated, and one commit of work.
"$MJ" init >/dev/null
"$MJ" update >/dev/null
mkdir -p lib docs
echo a > lib/a
echo d > docs/d
git add . && git commit -qm base

Run

$ majordomus bench nosuch

Output contains

  • not a public command of the registry

exit 12

A target is a public command of share/commands.yaml and nothing else, so a name the registry does not carry is a missing artifact, exit 12, with the command that lists what exists.

Reference

Time every public command of the registry, cold and warm, and compare with the baseline.

The targets are the public commands of share/commands.yaml and nothing else: a command added to the registry is a target from that moment, and the harness never times itself. Each target runs in a disposable repository. When the tool's own suite is present, the scenario is the first scenario of the command's fixture under test/fixtures/commands/, so what is timed is what the site demonstrates and the suite executes; otherwise it is the bare command in an installed repository.

A read-only command is run once cold, then benchmark.warmup times unsampled, then benchmark.samples times warm, all in one repository. A command that mutates state gets a fresh repository per sample, so every sample of it is cold and it has no warm distribution. A sample is the wall-clock time of the child process alone; setup is never inside the clock. Every distribution is recorded with its count, minimum, p50, p90, p95, p99, maximum, mean and standard deviation; nothing is averaged away.

  • bench times every target; bench <command>... only those.
  • --list prints the targets with their class and scenario, --format json as data.
  • --samples <n>, --warmup <n> override the policy for this run only; --mode cold|warm|both selects the distributions recorded.
  • --format json prints the run as one document with schema majordomus/benchmark-result/v1: the run id, the commit and whether the tree was dirty, the platform, the profile and one result per target and mode.
  • Every run is written under .ai/local/benchmarks/ (runs/<run-id>.json, latest.json, one line per run in history.jsonl) unless --no-save. That is local evidence: ignored by git, never a baseline.
  • --write-baseline writes the run under .ai/repo/benchmarks/rust/, as baseline.<platform>.json — one baseline per platform, since a duration compared across machines compares the machines — with schema majordomus/benchmark-baseline/v1, and prints the old and new p50/p95/p99 per target. It refuses a dirty tree without --force, because a baseline records a commit.
  • --check compares the run with the baseline under benchmark.regression: for each target and mode, a p50, p95 or p99 more than the threshold fraction over the baseline is a FAIL naming the metric, both values and the threshold, and the command exits 10. No baseline exits 12; a baseline with another schema is not comparable and exits 15.

MJ_TIMING=1 on any command prints the phases and work counters of that run on stderr, which is how a slow target is taken apart.

$ majordomus bench --list
command      class                      scenario
init         generated-output-mutating  fixture fresh
doctor       read-only                  fixture not-wired
...

$ majordomus bench doctor version
command      mode  status       n     p50     p95     p99     max  scenario
doctor       cold  ok           1    2711    2711    2711    2711  fixture not-wired
doctor       warm  ok          10    2640    2790    2790    2790  fixture not-wired
version      cold  ok           1      41      41      41      41  fixture prints
version      warm  ok          10      38      45      45      45  fixture prints

slowest by warm p95 (cold where warm does not apply):
  doctor       p95 2790 ms
  version      p95 45 ms

run b-20260905T031200Z-9f1c saved as .ai/local/benchmarks/runs/b-20260905T031200Z-9f1c.json

Exit 2 on a usage error, 12 when a named target is not a public command of the registry, 13 when a target did not run cleanly (its row says setup-failed or the exit code it produced), 10, 12 and 15 from --check as above.

Evidence

Every case below runs in CI on Linux and macOS. The site refuses to build if a public command has no behavioural case, no negative case, or no demonstration.

Behaviour

Failure modes

Implementation

Demonstration