I3133 — The economics benchmark measures acceptance: how many sessions declared done while the task's hidden tests fail, with and without the finish contract

✕BLOKOVÁNO wave 1 · p2 · deep-work profile · parallel safe

Part of the-product-shows-what-it-caught — What finish caught is counted from the ledger and compared on the benchmark, never asserted.

Cíl

Add an acceptance metric to the economics methodology over the existing pilot corpus: per arm, the fraction of sessions whose final state the agent called done while the hidden acceptance tests fail, and add a treatment variant that includes the finish contract and hooks the first treatment excludes; the generator publishes it only past the methodology's existing rule.

Proč

Assessments of 2026-10-02, 10-11 and 10-13 asked for identical tasks with and without Majordomus, measuring false completion, not only tokens. The corpus already has hidden tests that decide whether a 'done' is done; docs/ECONOMICS.md lists 'the finish contract's effect on rework' as unmeasured. Under version 1's pairing rule, the runs this metric counts are invalid, and without a declaration that the runner observes, the metric would only repeat the failed-run table already in docs/generated/economics.md. Proposed in ADR 0208 (The product's value is stated as the false dones its finish caught, beside the friction it cost) (proposed; it carries no authority until the owner accepts it). Sources: chatgpt-majordomus:c176#9, chatgpt-majordomus:c226#18, chatgpt-majordomus:c256#5, chatgpt-majordomus:c194#3, chatgpt-majordomus:c001#2, chatgpt-majordomus:c119#5, chatgpt-majordomus:c126#1, chatgpt-majordomus:c179#11, chatgpt-majordomus:c179#13, chatgpt-majordomus:c198#14.

Aktuální stav

.ai/repo/benchmarks/economics/methodology.yaml (version 1, primary metric effective_token_reduction, a pair valid only when both runs succeeded); seven tasks with hidden tests; docs/generated/economics.md at 707a04c7ce8 reports 0 valid live pairs under methodology 1 and already tabulates failed runs per arm.

Cílový stav

Methodology version 2 declares the acceptance metric and the finish treatment; the calculator in apps/majordomus-cli/src/economics computes it; the report says unmeasured until pairs exist.

Rozsah

• .ai/repo/benchmarks/economics/methodology.yaml
• apps/majordomus-cli/src/economics
• docs/ECONOMICS.md
• docs/generated/economics.md
• apps/majordomus-cli/src/economics/runner.rs
• apps/majordomus-cli/tests/economics.rs
• .ai/repo/project/intents
• .ai/repo/benchmarks/economics/suites/pilot.yaml
• .ai/repo/benchmarks/economics/README.md
• test/cases/430_economics.sh
• test/cases/431_economics_claims_gated.sh

Mimo rozsah

• Recording the live runs (needs a provider account and the owner's budget)
• Cross-provider comparison
• A ledger-side false-acceptance signal (reopen or revert after a completed finish); no event records a reopen

Závislosti

• I0401✕BLOKOVÁNO Definovat přijatý výsledek a spojit ho s jeho měřeními, aniž by se spojení ukládalo

Co na tom čeká

• I3276✕BLOCKED The time from an intent's proposal to its verified satisfaction is derived from the records, with the interventions on the way
• I3370✕BLOCKED The corpus suite is registered and committed before its first run, and the T2 protocol names its registration digest
• I3371✕BLOCKED Economics summary decides a registered suite's primary outcome by its registered test, reports secondary differences without a verdict word, and refuses typed figures about them

Kritéria přijetí

• majordomus economics references still proves every hidden test fails on start and passes on reference
• With a fixture of paired runs the calculator reports the acceptance metric per arm; with none it reports no claim
• majordomus economics check refuses a prose figure for the metric
• Methodology version 2 declares an acceptance pairing of its own: a pair is valid when both sessions ran to their end and the hidden tests ran on both final states; a hidden-test failure is the measured outcome, never a reason to invalidate the pair; the pairing has its own min_valid_pairs
• 'Declared done' is defined the same way in both arms as a completion declaration the shared task prompt requires and the runner parses; the finish-contract variant also requires an accepted `majordomus finish --outcome completed`, applied by the runner rather than left to the agent's choice
• The denominator is the arm's sessions in valid acceptance pairs, and the report prints it beside every figure
• The treatment-arm count of finish accepted while hidden tests fail is reported under the name false acceptance
• Criterion acceptance-is-a-benchmark-treatment of the intent names false acceptance, and carries evidence: test with ref apps/majordomus-cli/tests/economics.rs

Validace

• cargo test --test economics
• majordomus economics check
• majordomus generate --check

Požadované důkazy

• test_acceptance_metric

Důkazy

Nic nezaznamenáno. Za každým tokenem výše musí stát příkaz nebo artefakt, než lze tento úkol dokončit; vyprávění se odmítá.

Časová osa

zahájeno

—

ověřeno

—

dokončeno

—

Tato tři pole, důkazy výše a stav závislostí jsou vše, z čeho se stav skládá. Neexistuje žádné pole stavu, které by s nimi mohlo být v rozporu.

Canonical record: .ai/repo/project/issues/I3133.yaml. Read it back with majordomus plan show I3133.