I2068 — A verdict that depends on the runner says so, and one cold sample is not a regression
Make the timing verdicts this repository's CI and its adopters' hooks hand down distinguish a slower change from a slower machine: a benchmark regression is judged from more than one cold sample and against the load it was measured under, and a case whose cost sits within a few percent of its kill bound is reported as a budget risk before it fails a suite whose every case passed.
DONE wave 0 · p2 · implementation profile · parallel safe
Part of M004 — Performance is executable evidence, and the hot path does no canonical work twice.
Objective
Make the timing verdicts this repository's CI and its adopters' hooks hand down distinguish a slower change from a slower machine: a benchmark regression is judged from more than one cold sample and against the load it was measured under, and a case whose cost sits within a few percent of its kill bound is reported as a budget risk before it fails a suite whose every case passed.
Why
On 2026-10-08 three verdicts failed changes that ran none of the measured code: test/cases/82_bench_check.sh failed #823 on `version cold — regression: p50 9 -> 23 ms` from one sample on a saturated runner; test/cases/95_executable_reference.sh cost 3465 s against a 3450 s budget on a slow runner and failed batch 3's suite aggregate although every case passed (rerun once, green); and prismatic-platform's pre-push page-load check reported violations at load ~30 that passed alone and on retry. Each was rerun by hand, and each rerun costs a real run fifteen jobs on a shared pool.
Current state
82_bench_check compares one cold sample with a committed macOS baseline at a fixed percentage threshold and records no load; the suite aggregate refuses a case over its per-case budget with no margin warning beforehand (the budget is held below the 3600 s kill bound by case 417); nothing in a timing verdict names the runner, its load or the number of samples.
Desired state
A timing failure states the samples it rests on, the runner and its load; a single cold sample over threshold is reported as a warning to re-measure, and a regression needs the repeat to agree; a case whose recorded cost is within 5% of its budget is reported by the suite before the run that crosses it; the 3600 s kill bound and case 417's guarantee are unchanged, and no threshold is loosened.
Scope
- test/cases/82_bench_check.sh
- test/cases/95_executable_reference.sh
- test/cases/417_the_suite_finishes_inside_its_bound.sh
- scripts/ci/suite-budget
- apps/majordomus-cli/src/bench
Dependencies
None. This issue is a root of the graph.
What waits on this
Acceptance criteria
- A bench check with one cold sample over threshold reports a warning naming the sample count and does not fail; two agreeing samples over threshold fail as today
- Every timing finding names the runner, the sample count and the load average it was measured under
- The suite reports a case whose recorded cost is within 5% of the per-case budget, by name, before any run exceeds it
- Case 417 and the kill bound are unchanged, and no regression threshold is raised
Validation
- bash test/run.sh 82_bench_check
- bash test/run.sh 417_the_suite_finishes_inside_its_bound
Evidence required
- a_single_cold_sample_is_a_warning_and_two_are_a_verdict
- a_case_near_its_budget_is_named_before_it_fails
Evidence
| covers | type | command | result | at commit |
|---|---|---|---|---|
| a_single_cold_sample_is_a_warning_and_two_are_a_verdict | test | bash test/run.sh 82_bench_check && cargo test --manifest-path apps/majordomus-cli/Cargo.toml --test bench a_single_cold_sample_is_a_warning_and_two_are_a_verdict | ok 82_bench_check: one cold sample over threshold is WARN on 1 of 1 sample(s) and exits 0; --cold-samples 2 both over is FAIL on 2 of 2 and exits 10; every finding names samples, runner and load. Rust check: 1 sample WARN, 2 agreeing FAIL, verdict line names runner and load | 44ad51e |
| a_case_near_its_budget_is_named_before_it_fails | test | bash test/run.sh 3084_a_case_near_its_budget_is_named 417_the_suite_finishes_inside_its_bound | ok both: suite-budget names by case every cost within budget.near_percent (5) of the per-case budget, in this run and as recorded in suite-durations.tsv, as NEAR without changing the exit status; case 417, the 5300 s bound and every budget unchanged | 44ad51e |
| a_single_cold_sample_is_a_warning_and_two_are_a_verdict | test | bash test/run.sh 82_bench_check && cargo test --manifest-path apps/majordomus-cli/Cargo.toml --test bench | 82_bench_check ok; --test bench 9 passed, 0 failed on master 707a04c7ce8 (2026-10-17) | 707a04c |
| a_case_near_its_budget_is_named_before_it_fails | test | bash test/run.sh 3084_a_case_near_its_budget_is_named 417_the_suite_finishes_inside_its_bound | both ok on master 707a04c7ce8 (2026-10-17) | 707a04c |
Timeline
- started
- 2026-10-16T07:54:07Z
- verified
- 2026-10-16T07:54:13Z
- completed
- 2026-10-17T21:35:27Z
Those three fields, the evidence above and the state of the dependencies are all the status is made of. There is no status field to disagree with them.
Canonical record: .ai/repo/project/issues/I2068.yaml. Read it back with majordomus plan show I2068.