I0826 — bench --check refuses a regression by policy
Compare a fresh run against the baseline under the policy's benchmark.regression thresholds (relative p50, p95, p99) and exit non-zero naming the command, the metric, both values and the threshold.
DONE wave 4 · p0 · implementation profile · runs alone
Part of M004 — Performance is executable evidence, and the hot path does no canonical work twice.
Objective
Compare a fresh run against the baseline under the policy's benchmark.regression thresholds (relative p50, p95, p99) and exit non-zero naming the command, the metric, both values and the threshold.
Why
Without a gate the next slow validator arrives unnoticed, exactly as the current ones did.
Current state
No comparison exists.
Desired state
Deterministic report: metric, baseline, current, absolute and relative delta, threshold, result; incompatible baselines are reported as not comparable rather than as a number.
Scope
- lib/bench.sh
- share/allow/policy.txt
- share/skeleton/policy.yaml
- .ai/repo/policy.yaml
- test/cases/82_bench_check.sh
Out of scope
- Significance testing
Dependencies
What waits on this
- I0830DONE docs/PERFORMANCE.md, README, CONTRIBUTING and CLI reference
- I0831BLOCKED Performance claims at their proven status
- I0835BLOCKED CI runs the structural performance checks
- I0838READY bench compare and the top regressions report
Acceptance criteria
- A baseline edited to make the current run a regression makes the check exit non-zero with the named metric
- A compatible run within thresholds passes
- A baseline with another schema version is not comparable
Validation
- bash test/run.sh 82_bench_check
Evidence required
- regression_refused
Evidence
| covers | type | command | result | at commit |
|---|---|---|---|---|
| regression_refused | test | bash test/run.sh 82_bench_check | case 82 asserts --check exits 12 without a baseline, 15 for a baseline of another schema, 0 with OK lines when the same command is measured twice on one machine, 10 with FAIL lines naming p50, p95 and p99, both values and the policy threshold when the baseline is edited to claim one millisecond, that raising benchmark.regression.p50 in the policy turns the same edit into a pass, and that a target the baseline does not know is INFO, never compared; baseline rows reach awk through a file because BSD awk refuses a newline in a -v value | next |
Risk
Thresholds live in the policy and are read like every other policy key: declared, no reader-side default.
Timeline
- started
- 2026-09-05T03:47:33Z
- verified
- 2026-09-05T03:47:33Z
- completed
- 2026-09-05T03:47:33Z
Those three fields, the evidence above and the state of the dependencies are all the status is made of. There is no status field to disagree with them.
Canonical record: .ai/repo/project/issues/I0826.yaml. Read it back with majordomus plan show I0826.