HTTP API: economics

One success gate's verdict on a run.

use majordomus_cli::economics::model::EconomicsCheck;
let c = EconomicsCheck { id: "tests_kept".into(), passed: true, detail: "0 deleted".into() };
let back: EconomicsCheck = serde_json::from_value(serde_json::to_value(&c).unwrap()).unwrap();
assert_eq!(back, c);

The answer of economics.check, the same over the CLI, HTTP and MCP: how much was read and every refusal. ok is true exactly when findings is empty. scanned counts the hand-written files read, docs/CLAIMS.yaml included and generated files excluded; claims_checked counts the methodology's claim bindings, whatever their status.

use majordomus_cli::economics::claims::{check, EconomicsCheckReport};
let dir = tempfile::tempdir().unwrap();
let git = std::process::Command::new("git")
    .env_remove("GIT_DIR").env_remove("GIT_WORK_TREE").env_remove("GIT_INDEX_FILE")
    .arg("-C").arg(dir.path()).args(["init", "-q"]).status().unwrap();
assert!(git.success());
let report: EconomicsCheckReport = check(dir.path());
assert!(report.ok && report.findings.is_empty(), "an empty repository claims nothing");
let json = serde_json::to_value(&report).unwrap();
assert_eq!(json["claims_checked"], 0);
assert_eq!(serde_json::from_value::<EconomicsCheckReport>(json).unwrap(), report);

How a number came to be. The classes are ordered from strongest to weakest, and a value computed from several inputs is never stronger than the weakest of them.

use majordomus_cli::economics::model::EconomicsClass;
assert_eq!(serde_json::to_value(EconomicsClass::Counterfactual).unwrap(), "counterfactual");
let parsed: EconomicsClass = serde_json::from_value("observed".into()).unwrap();
assert!(parsed < EconomicsClass::Estimated, "a stronger class sorts first");

The context suite's latest measurement, in brief.

use std::collections::BTreeMap;
use majordomus_cli::economics::model::{EconomicsContextView, EconomicsTokenizer};
let v = EconomicsContextView {
    revision: "8d37d011ef".into(), measured_at: "2026-09-24T10:00:00Z".into(),
    tokenizer: EconomicsTokenizer {
        encoding: "o200k_base".into(), implementation: "tiktoken-rs 0.12.0".into(),
    },
    seeds: 2, candidate_tokens: 20_000, considered_tokens: 300_000, selected_tokens: 15_000,
    estimated_selected_tokens: 16_000,
    excluded_tokens: BTreeMap::from([("budget".into(), 5_000)]),
    seeds_over_budget_counted: 0, budget_tokens: 8_000,
};
assert_eq!(v.selected_tokens + v.excluded_tokens["budget"], v.candidate_tokens);
assert_eq!(serde_json::to_value(&v).unwrap()["tokenizer"]["encoding"], "o200k_base");

The shape of a set of per-pair values.

use majordomus_cli::economics::model::EconomicsDistribution;
let d: EconomicsDistribution = serde_json::from_value(serde_json::json!({
    "n": 1, "mean": 0.4, "median": 0.4, "p25": 0.4, "p75": 0.4, "p95": 0.4,
    "min": 0.4, "max": 0.4
}))
.unwrap();
assert!(d.sd.is_none(), "one value has no sample standard deviation");
assert!(d.min <= d.p25 && d.p25 <= d.median && d.median <= d.p75 && d.p75 <= d.max);

One run excluded under the outlier rule.

use majordomus_cli::economics::model::{EconomicsExclusion, EconomicsOutlierPolicy};
let o: EconomicsOutlierPolicy = serde_json::from_value(serde_json::json!({
    "rule": "A run is excluded only by naming it here with a reason.",
    "excluded": [{ "run": "vat-rounding--baseline--r2", "reason": "provider outage" }]
}))
.unwrap();
let first: &EconomicsExclusion = &o.excluded[0];
assert_eq!(first.run, "vat-rounding--baseline--r2");
let unexplained = serde_json::json!({ "run": "vat-rounding--baseline--r1" });
assert!(serde_json::from_value::<EconomicsExclusion>(unexplained).is_err());

The answer of economics.explain: one metric, and everything it rests on.

A repository that declares no benchmark methodology measures nothing, so it has no metric to explain: it is answered as economics.summary answers it — present: false, the statement why, no metric and no number — rather than refused, because a repository that does not benchmark itself is not a malformed request. A metric id that a declared methodology does not have is still refused.

use majordomus_cli::economics::model::{EconomicsClass, EconomicsExplanation};
let e: EconomicsExplanation = serde_json::from_value(serde_json::json!({
    "present": true,
    "metric": { "id": "context_reduction_ratio", "title": "Context reduction",
        "class": "derived", "inputs": "counted", "unit": "ratio",
        "formula": "median over seeds of 1 - selected_tokens / candidate_tokens",
        "status": "not_measured", "n": 0, "suite": "context" },
    "class_meaning": "Arithmetic over observed or counted values.", "methodology": 1,
    "suites": [], "pairs": [], "excluded": [], "variants": [],
    "reproduce": ["majordomus economics measure --suite context"]
}))
.unwrap();
assert_eq!(e.metric.unwrap().inputs, Some(EconomicsClass::Counted));
assert!(e.pairs.is_empty(), "a counted metric rests on seeds, not on pairs");

let none: EconomicsExplanation = serde_json::from_value(serde_json::json!({
    "present": false, "statement": "No verified total-token-savings claim is available",
    "class_meaning": "", "suites": [], "pairs": [], "excluded": [], "variants": [],
    "reproduce": []
}))
.unwrap();
assert!(none.metric.is_none() && none.methodology.is_none(), "nothing declared, no number");

One thing the check refuses. detail names the remedy, so a finding can be acted on without reading the methodology. A finding about a claim or a file rather than a line carries no line, and its JSON omits the key. The kinds:

• unsupported_quantity: a number typed into hand-written prose next to the economics vocabulary.
• claim_unsupported: a claim binding that does not hold — the bound claim is missing from docs/CLAIMS.yaml, or it is guaranteed while its metric does not stand where the binding says or its evidence is not current.
• claim_unbound: a guaranteed claim says tokens, context or cost go down and no binding ties it to a metric.
• claim_duplicate: one claim id appears more than once in docs/CLAIMS.yaml.
• declarations_unreadable: the benchmark declarations cannot be read, so no binding can be shown to hold.
• claims_unreadable: docs/CLAIMS.yaml exists and cannot be read or parsed.
• scan_failed: the prose could not be listed or a file in it could not be read, so nothing can be said to be clean.

use majordomus_cli::economics::claims::EconomicsFinding;
let finding = EconomicsFinding {
    kind: "claim_unsupported".into(),
    path: "docs/CLAIMS.yaml".into(),
    line: None,
    detail: "set the claim to planned, or record the evidence".into(),
};
let json = serde_json::to_value(&finding).unwrap();
assert!(json.get("line").is_none(), "no line, no key");
assert_eq!(serde_json::from_value::<EconomicsFinding>(json).unwrap(), finding);

How recorded evidence stands against the repository as it is now.

use majordomus_cli::economics::model::EconomicsFreshness;
assert_eq!(serde_json::to_value(EconomicsFreshness::NoEvidence).unwrap(), "no_evidence");
let f: EconomicsFreshness = serde_json::from_value("incompatible".into()).unwrap();
assert_eq!(f, EconomicsFreshness::Incompatible);

A recorded measurement over time: one entry per recorded context run.

use majordomus_cli::economics::model::EconomicsHistoryPoint;
let p: EconomicsHistoryPoint = serde_json::from_value(serde_json::json!({
    "suite": "context", "methodology": 1, "revision": "8d37d011ef",
    "at": "2026-09-24T10:00:00Z", "metric": "context_reduction_ratio", "value": 0.62, "n": 40
}))
.unwrap();
assert_eq!((p.methodology, p.n), (1, 40));
assert_eq!(serde_json::to_value(&p).unwrap()["metric"], "context_reduction_ratio");

A statement the benchmark exists to test. Never a result.

use majordomus_cli::economics::model::EconomicsHypothesis;
let h: EconomicsHypothesis = serde_json::from_value(serde_json::json!({
    "id": "continuity",
    "statement": "Work that spans sessions gains the most.",
    "status": "untested"
}))
.unwrap();
assert_eq!(h.status, "untested");
assert_eq!(serde_json::to_value(&h).unwrap()["id"], "continuity");

A bootstrap interval, with everything needed to reproduce it.

use majordomus_cli::economics::model::EconomicsInterval;
let i: EconomicsInterval = serde_json::from_value(serde_json::json!({
    "level_bp": 9500, "low": 0.21, "high": 0.38,
    "method": "bootstrap percentile of the median", "resamples": 10000, "seed": 20260924
}))
.unwrap();
assert!(i.low <= i.high);
assert_eq!(serde_json::to_value(&i).unwrap()["seed"], 20260924, "the seed travels with it");

One metric: its value, where it comes from, and how much it can bear.

use majordomus_cli::economics::model::{EconomicsClass, EconomicsMetric, EconomicsMetricStatus};
let m: EconomicsMetric = serde_json::from_value(serde_json::json!({
    "id": "effective_token_reduction", "title": "Effective token reduction",
    "class": "derived", "inputs": "observed", "unit": "ratio",
    "formula": "median of 1 - treatment / control", "status": "not_measured", "n": 0,
    "suite": "pilot"
}))
.unwrap();
assert_eq!(m.inputs, Some(EconomicsClass::Observed));
assert!(m.value.is_none() && m.status == EconomicsMetricStatus::NotMeasured);
assert!(serde_json::to_value(&m).unwrap().get("value").is_none(), "no value, not a zero");

Where a metric stands against the publication rule: whether its value may back a claim, is a result too thin to claim anything, or was never measured at all.

use majordomus_cli::economics::model::EconomicsMetricStatus;
assert_eq!(serde_json::to_value(EconomicsMetricStatus::NotMeasured).unwrap(), "not_measured");
let s: EconomicsMetricStatus = serde_json::from_value("preliminary".into()).unwrap();
assert_ne!(s, EconomicsMetricStatus::Verified, "a preliminary result is not a claim");

A control run and its treatment run, and what follows from them.

use majordomus_cli::economics::model::{EconomicsPair, EconomicsPairStatus};
let p: EconomicsPair = serde_json::from_value(serde_json::json!({
    "suite": "pilot", "task": "vat-rounding", "category": "bugfix", "sessions": 1,
    "repetition": 2, "control": "vat-rounding--baseline--r2", "status": "missing",
    "reasons": ["no treatment run recorded"]
}))
.unwrap();
assert_eq!(p.status, EconomicsPairStatus::Missing);
assert!(p.treatment.is_none() && p.token_reduction.is_none(), "no side, no reduction");

Pair counts of a suite, so that nothing is dropped out of sight.

use majordomus_cli::economics::model::EconomicsPairCounts;
let c = EconomicsPairCounts {
    declared: 14, attempted: 12, valid: 9, control_failed: 1,
    treatment_failed: 1, both_failed: 0, other: 1,
};
let accounted = c.valid + c.control_failed + c.treatment_failed + c.both_failed + c.other;
assert_eq!(accounted, c.attempted, "every attempted pair is in exactly one bucket");
assert_eq!(EconomicsPairCounts::default().declared, 0);

Why a pair does or does not count.

use majordomus_cli::economics::model::EconomicsPairStatus;
assert_eq!(serde_json::to_value(EconomicsPairStatus::ControlFailed).unwrap(), "control_failed");
let s: EconomicsPairStatus = serde_json::from_value("usage_unavailable".into()).unwrap();
assert_ne!(s, EconomicsPairStatus::Valid);

When a quantitative total-token claim may be published. Every threshold must hold.

use majordomus_cli::economics::model::EconomicsPublicationPolicy;
let rule = serde_json::json!({
    "min_valid_pairs": 30, "min_categories": 4, "min_pairs_per_category": 5,
    "min_repetitions": 3, "min_valid_pair_rate_bp": 8000, "max_interval_width_bp": 2000,
    "require_current": true
});
let p: EconomicsPublicationPolicy = serde_json::from_value(rule.clone()).unwrap();
assert!(p.require_current && p.min_valid_pairs == 30);
let mut lax = rule;
lax.as_object_mut().unwrap().remove("require_current");
assert!(serde_json::from_value::<EconomicsPublicationPolicy>(lax).is_err(), "no default");

The answer of economics.runs: the raw facts every metric is computed from.

use majordomus_cli::economics::model::EconomicsRunList;
let l: EconomicsRunList =
    serde_json::from_value(serde_json::json!({ "count": 0, "runs": [] })).unwrap();
assert_eq!(l.count, l.runs.len());
assert!(l.diagnostics.is_empty(), "no diagnostics listed means none");

One run in brief, with its derived totals.

use majordomus_cli::economics::model::EconomicsRunView;
let v: EconomicsRunView = serde_json::from_value(serde_json::json!({
    "id": "vat-rounding--majordomus--r1", "suite": "pilot", "task": "vat-rounding",
    "variant": "majordomus", "repetition": 1, "model": "claude-sonnet-5",
    "revision": "8d37d011ef", "completed": false, "checks": [], "sessions": 1,
    "path": ".ai/repo/benchmarks/economics/runs/pilot/vat-rounding--majordomus--r1.json"
}))
.unwrap();
assert!(v.usage.is_none(), "a run whose provider reported no usage has no totals");
assert!(v.path.ends_with(&format!("{}.json", v.id)));

A metric's value within one segment of the pairs.

use majordomus_cli::economics::model::EconomicsSegment;
let s: EconomicsSegment = serde_json::from_value(serde_json::json!({
    "dimension": "category", "value": "multi-session", "n": 0
}))
.unwrap();
assert!(s.token_reduction.is_none(), "a segment with no valid pair has no distribution");

A suite and the state of its evidence.

use majordomus_cli::economics::model::{
    EconomicsFreshness, EconomicsPairCounts, EconomicsSuiteView,
};
let v = EconomicsSuiteView {
    id: "pilot".into(), kind: "live".into(), version: 1, title: "Pilot".into(),
    model: Some("claude-sonnet-5".into()), freshness: EconomicsFreshness::NoEvidence,
    freshness_detail: Some("no run recorded".into()), runs: 0,
    pairs: EconomicsPairCounts { declared: 14, ..EconomicsPairCounts::default() },
    revisions: vec![], harnesses: vec![], models_reported: vec![],
};
let json = serde_json::to_value(&v).unwrap();
assert_eq!(json["freshness"], "no_evidence");
assert_eq!(json["pairs"]["declared"], 14, "declared pairs show even when none ran");

The answer of economics.summary: every metric, where each comes from, and the one statement the evidence allows.

use majordomus_cli::economics::model::EconomicsSummary;
let s: EconomicsSummary = serde_json::from_value(serde_json::json!({
    "present": false,
    "verdict": { "publishable": false, "statement": "No methodology is declared." },
    "metrics": [], "suites": [], "pairs": [], "segments": [], "history": [],
    "variants": [], "hypotheses": []
}))
.unwrap();
assert!(s.methodology.is_none() && s.publication.is_none() && s.diagnostics.is_empty());
let json = serde_json::to_value(&s).unwrap();
assert!(json.get("methodology").is_none(), "an absent methodology is absent, not null");

The identity of a tokenizer, recorded with every count it produced.

use majordomus_cli::economics::model::EconomicsTokenizer;
let t: EconomicsTokenizer = serde_json::from_value(serde_json::json!({
    "encoding": "o200k_base", "implementation": "tiktoken-rs 0.12.0"
}))
.unwrap();
let other = EconomicsTokenizer { implementation: "tiktoken-rs 0.13.0".into(), ..t.clone() };
assert_ne!(t, other, "the same encoding from another implementation is another tokenizer");

A run's totals, derived from its raw usage.

use majordomus_cli::economics::model::EconomicsUsage;
let u = EconomicsUsage {
    input_uncached: 10, cache_write: 90, cache_read: 900, output: 50,
    total_input: 1000, total: 1050, requests: 3, ..EconomicsUsage::default()
};
assert_eq!(u.total_input, u.input_uncached + u.cache_write + u.cache_read);
let json = serde_json::to_value(&u).unwrap();
assert!(json.get("reasoning").is_none() && json.get("cost_microusd").is_none());

One arm of the experiment: what the session runs with, and what it runs without.

use majordomus_cli::economics::model::EconomicsVariant;
let v: EconomicsVariant = serde_json::from_value(serde_json::json!({
    "id": "baseline", "role": "control", "title": "Without Majordomus",
    "description": "The fixture repository as a competent team keeps it."
}))
.unwrap();
assert_eq!(v.role, "control");
assert!(v.excludes.is_empty(), "an arm that names no exclusion excludes nothing");

The one sentence the evidence allows, and why it allows no stronger one.

use majordomus_cli::economics::model::EconomicsVerdict;
let v: EconomicsVerdict = serde_json::from_value(serde_json::json!({
    "publishable": false,
    "statement": "No quantitative token claim: 9 valid pairs of the 30 the rule requires.",
    "unmet": ["min_valid_pairs: 9 < 30"]
}))
.unwrap();
assert!(!v.publishable);
assert_eq!(v.unmet.len(), 1, "every threshold not met is named");