Skip to content

economics.summary

Every metric of the economics benchmark with its value, measurement class, sample size, distribution and bootstrap interval; every pair of runs with its status (valid, control failed, treatment failed, incomparable, ...); results by segment; the state of each suite's evidence (current, stale, incompatible); and the one statement the publication rule allows. Context reduction and total-token reduction are separate metrics and are never merged.

Every metric of the economics benchmark with its value, measurement class, sample size, distribution and bootstrap interval; every pair of runs with its status (valid, control failed, treatment failed, incomparable, ...); results by segment; the state of each suite's evidence (current, stale, incompatible); and the one statement the publication rule allows. Context reduction and total-token reduction are separate metrics and are never merged.

Exposure

Where this one definition is reachable. Absence is explicit: a surface not listed does not carry it.

surfaceas
MCPtool majordomus_economics
resource majordomus://economics
HTTPGET /api/v1/economics operationId economics.summary
command linemajordomus economics summary

Schemas

The canonical input and output, from the Rust types; the MCP tool schema and the OpenAPI parameters and responses are derived from these.

input · EconomicsQuery

The input of economics.summary: optional narrowing of the pairs and metrics. The verdict is always about all the evidence, whatever is narrowed, and a narrowed metric is never verified: the publication rule was evaluated over all of it, not over a slice.

use majordomus_cli::economics::model::EconomicsQuery;
let all: EconomicsQuery = serde_json::from_value(serde_json::json!({})).unwrap();
assert_eq!(all, EconomicsQuery::default());
let q: EconomicsQuery =
    serde_json::from_value(serde_json::json!({ "suite": "pilot", "category": "bugfix" }))
        .unwrap();
assert_eq!(q.suite.as_deref(), Some("pilot"));
let typo = serde_json::json!({ "suites": "pilot" });
let refused = serde_json::from_value::<EconomicsQuery>(typo);
assert!(refused.is_err(), "a misspelt filter is refused");
propertytyperequireddescription
category string | null no Only tasks of this category.
model string | null no Only runs that asked for this model.
suite string | null no Only this suite.
task string | null no Only this task.
JSON Schema 
{
  "additionalProperties": false,
  "description": "The input of `economics.summary`: optional narrowing of the pairs and metrics. The\nverdict is always about all the evidence, whatever is narrowed, and a narrowed metric\nis never verified: the publication rule was evaluated over all of it, not over a slice.\n\n```\nuse majordomus_cli::economics::model::EconomicsQuery;\nlet all: EconomicsQuery = serde_json::from_value(serde_json::json!({})).unwrap();\nassert_eq!(all, EconomicsQuery::default());\nlet q: EconomicsQuery =\n    serde_json::from_value(serde_json::json!({ \"suite\": \"pilot\", \"category\": \"bugfix\" }))\n        .unwrap();\nassert_eq!(q.suite.as_deref(), Some(\"pilot\"));\nlet typo = serde_json::json!({ \"suites\": \"pilot\" });\nlet refused = serde_json::from_value::<EconomicsQuery>(typo);\nassert!(refused.is_err(), \"a misspelt filter is refused\");\n```",
  "properties": {
    "category": {
      "description": "Only tasks of this category.",
      "type": [
        "string",
        "null"
      ]
    },
    "model": {
      "description": "Only runs that asked for this model.",
      "type": [
        "string",
        "null"
      ]
    },
    "suite": {
      "description": "Only this suite.",
      "type": [
        "string",
        "null"
      ]
    },
    "task": {
      "description": "Only this task.",
      "type": [
        "string",
        "null"
      ]
    }
  },
  "title": "EconomicsQuery",
  "type": "object"
}

output · EconomicsSummary

The answer of economics.summary: every metric, where each comes from, and the one statement the evidence allows.

use majordomus_cli::economics::model::EconomicsSummary;
let s: EconomicsSummary = serde_json::from_value(serde_json::json!({
    "present": false,
    "verdict": { "publishable": false, "statement": "No methodology is declared." },
    "metrics": [], "suites": [], "pairs": [], "segments": [], "history": [],
    "variants": [], "hypotheses": []
}))
.unwrap();
assert!(s.methodology.is_none() && s.publication.is_none() && s.diagnostics.is_empty());
let json = serde_json::to_value(&s).unwrap();
assert!(json.get("methodology").is_none(), "an absent methodology is absent, not null");
propertytyperequireddescription
context one of 2 no The context suite in brief.
diagnostics array no Problems reading the declarations or the evidence. default []
history array yes The metric over recorded measurements.
hypotheses array yes The hypotheses, as hypotheses.
methodology integer | null no The methodology version.
metrics array yes Every metric.
pairs array yes Every pair, valid or not.
present boolean yes Whether a methodology is declared here at all.
primary_metric string | null no The primary metric's id.
publication one of 2 no The publication rule.
question string | null no The question.
segments array yes Token reduction by segment.
suites array yes Every suite and its evidence.
variants array yes The arms of the experiment.
verdict reference yes What the total-token evidence allows to be said.
JSON Schema 
{
  "$defs": {
    "EconomicsClass": {
      "description": "How a number came to be. The classes are ordered from strongest to weakest, and a value\ncomputed from several inputs is never stronger than the weakest of them.\n\n```\nuse majordomus_cli::economics::model::EconomicsClass;\nassert_eq!(serde_json::to_value(EconomicsClass::Counterfactual).unwrap(), \"counterfactual\");\nlet parsed: EconomicsClass = serde_json::from_value(\"observed\".into()).unwrap();\nassert!(parsed < EconomicsClass::Estimated, \"a stronger class sorts first\");\n```",
      "oneOf": [
        {
          "const": "observed",
          "description": "Returned by the provider or its harness at run time and recorded unmodified.",
          "type": "string"
        },
        {
          "const": "counted",
          "description": "Computed deterministically from bytes on disk with a named, pinned tokenizer.",
          "type": "string"
        },
        {
          "const": "derived",
          "description": "Arithmetic over observed or counted values.",
          "type": "string"
        },
        {
          "const": "estimated",
          "description": "A number standing in for one nobody measured. Never compared, never published.",
          "type": "string"
        },
        {
          "const": "counterfactual",
          "description": "What a mechanism would have avoided, modelled rather than observed.",
          "type": "string"
        }
      ]
    },
    "EconomicsContextView": {
      "description": "The context suite's latest measurement, in brief.\n\n```\nuse std::collections::BTreeMap;\nuse majordomus_cli::economics::model::{EconomicsContextView, EconomicsTokenizer};\nlet v = EconomicsContextView {\n    revision: \"8d37d011ef\".into(), measured_at: \"2026-09-24T10:00:00Z\".into(),\n    tokenizer: EconomicsTokenizer {\n        encoding: \"o200k_base\".into(), implementation: \"tiktoken-rs 0.12.0\".into(),\n    },\n    seeds: 2, candidate_tokens: 20_000, considered_tokens: 300_000, selected_tokens: 15_000,\n    estimated_selected_tokens: 16_000,\n    excluded_tokens: BTreeMap::from([(\"budget\".into(), 5_000)]),\n    seeds_over_budget_counted: 0, budget_tokens: 8_000,\n};\nassert_eq!(v.selected_tokens + v.excluded_tokens[\"budget\"], v.candidate_tokens);\nassert_eq!(serde_json::to_value(&v).unwrap()[\"tokenizer\"][\"encoding\"], \"o200k_base\");\n```",
      "properties": {
        "budget_tokens": {
          "description": "The compiler's budget.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "candidate_tokens": {
          "description": "Counted candidate tokens (relevant: selected or cut by the budget), all seeds.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "considered_tokens": {
          "description": "Counted tokens of everything the compiler reached, all seeds.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "estimated_selected_tokens": {
          "description": "The compiler's own estimate of the selected tokens, all seeds.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "excluded_tokens": {
          "additionalProperties": {
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          },
          "description": "Counted tokens left out, by reason, all seeds.",
          "type": "object"
        },
        "measured_at": {
          "description": "When.",
          "type": "string"
        },
        "revision": {
          "description": "The revision measured.",
          "type": "string"
        },
        "seeds": {
          "description": "Seeds measured.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "seeds_over_budget_counted": {
          "description": "Seeds whose counted selection exceeds the compiler's budget.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "selected_tokens": {
          "description": "Counted selected tokens, all seeds.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "tokenizer": {
          "$ref": "#/$defs/EconomicsTokenizer",
          "description": "The tokenizer."
        }
      },
      "required": [
        "revision",
        "measured_at",
        "tokenizer",
        "seeds",
        "candidate_tokens",
        "considered_tokens",
        "selected_tokens",
        "estimated_selected_tokens",
        "excluded_tokens",
        "seeds_over_budget_counted",
        "budget_tokens"
      ],
      "type": "object"
    },
    "EconomicsDistribution": {
      "description": "The shape of a set of per-pair values.\n\n```\nuse majordomus_cli::economics::model::EconomicsDistribution;\nlet d: EconomicsDistribution = serde_json::from_value(serde_json::json!({\n    \"n\": 1, \"mean\": 0.4, \"median\": 0.4, \"p25\": 0.4, \"p75\": 0.4, \"p95\": 0.4,\n    \"min\": 0.4, \"max\": 0.4\n}))\n.unwrap();\nassert!(d.sd.is_none(), \"one value has no sample standard deviation\");\nassert!(d.min <= d.p25 && d.p25 <= d.median && d.median <= d.p75 && d.p75 <= d.max);\n```",
      "properties": {
        "max": {
          "description": "Largest.",
          "format": "double",
          "type": "number"
        },
        "mean": {
          "description": "Arithmetic mean.",
          "format": "double",
          "type": "number"
        },
        "median": {
          "description": "Median.",
          "format": "double",
          "type": "number"
        },
        "min": {
          "description": "Smallest.",
          "format": "double",
          "type": "number"
        },
        "n": {
          "description": "How many values.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "p25": {
          "description": "25th percentile.",
          "format": "double",
          "type": "number"
        },
        "p75": {
          "description": "75th percentile.",
          "format": "double",
          "type": "number"
        },
        "p95": {
          "description": "95th percentile.",
          "format": "double",
          "type": "number"
        },
        "sd": {
          "description": "Sample standard deviation; absent for one value.",
          "format": "double",
          "type": [
            "number",
            "null"
          ]
        }
      },
      "required": [
        "n",
        "mean",
        "median",
        "p25",
        "p75",
        "p95",
        "min",
        "max"
      ],
      "type": "object"
    },
    "EconomicsFreshness": {
      "description": "How recorded evidence stands against the repository as it is now.\n\n```\nuse majordomus_cli::economics::model::EconomicsFreshness;\nassert_eq!(serde_json::to_value(EconomicsFreshness::NoEvidence).unwrap(), \"no_evidence\");\nlet f: EconomicsFreshness = serde_json::from_value(\"incompatible\".into()).unwrap();\nassert_eq!(f, EconomicsFreshness::Incompatible);\n```",
      "oneOf": [
        {
          "const": "current",
          "description": "Recorded under this methodology, and nothing it depends on changed since.",
          "type": "string"
        },
        {
          "const": "stale",
          "description": "Recorded under this methodology, but a mechanism it measured changed since.",
          "type": "string"
        },
        {
          "const": "incompatible",
          "description": "Recorded under another methodology version: not comparable at all.",
          "type": "string"
        },
        {
          "const": "no_evidence",
          "description": "Nothing recorded.",
          "type": "string"
        }
      ]
    },
    "EconomicsHistoryPoint": {
      "description": "A recorded measurement over time: one entry per recorded context run.\n\n```\nuse majordomus_cli::economics::model::EconomicsHistoryPoint;\nlet p: EconomicsHistoryPoint = serde_json::from_value(serde_json::json!({\n    \"suite\": \"context\", \"methodology\": 1, \"revision\": \"8d37d011ef\",\n    \"at\": \"2026-09-24T10:00:00Z\", \"metric\": \"context_reduction_ratio\", \"value\": 0.62, \"n\": 40\n}))\n.unwrap();\nassert_eq!((p.methodology, p.n), (1, 40));\nassert_eq!(serde_json::to_value(&p).unwrap()[\"metric\"], \"context_reduction_ratio\");\n```",
      "properties": {
        "at": {
          "description": "When.",
          "type": "string"
        },
        "methodology": {
          "description": "The methodology version, so that incompatible versions are never drawn as one line.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        },
        "metric": {
          "description": "The metric.",
          "type": "string"
        },
        "n": {
          "description": "How many values it rests on.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "revision": {
          "description": "The revision.",
          "type": "string"
        },
        "suite": {
          "description": "The suite.",
          "type": "string"
        },
        "value": {
          "description": "Its value.",
          "format": "double",
          "type": "number"
        }
      },
      "required": [
        "suite",
        "methodology",
        "revision",
        "at",
        "metric",
        "value",
        "n"
      ],
      "type": "object"
    },
    "EconomicsHypothesis": {
      "description": "A statement the benchmark exists to test. Never a result.\n\n```\nuse majordomus_cli::economics::model::EconomicsHypothesis;\nlet h: EconomicsHypothesis = serde_json::from_value(serde_json::json!({\n    \"id\": \"continuity\",\n    \"statement\": \"Work that spans sessions gains the most.\",\n    \"status\": \"untested\"\n}))\n.unwrap();\nassert_eq!(h.status, \"untested\");\nassert_eq!(serde_json::to_value(&h).unwrap()[\"id\"], \"continuity\");\n```",
      "properties": {
        "id": {
          "description": "A slug.",
          "type": "string"
        },
        "statement": {
          "description": "What is hypothesised.",
          "type": "string"
        },
        "status": {
          "description": "`untested`, `supported`, `refuted` — set by a person reading the evidence.",
          "type": "string"
        }
      },
      "required": [
        "id",
        "statement",
        "status"
      ],
      "type": "object"
    },
    "EconomicsInterval": {
      "description": "A bootstrap interval, with everything needed to reproduce it.\n\n```\nuse majordomus_cli::economics::model::EconomicsInterval;\nlet i: EconomicsInterval = serde_json::from_value(serde_json::json!({\n    \"level_bp\": 9500, \"low\": 0.21, \"high\": 0.38,\n    \"method\": \"bootstrap percentile of the median\", \"resamples\": 10000, \"seed\": 20260924\n}))\n.unwrap();\nassert!(i.low <= i.high);\nassert_eq!(serde_json::to_value(&i).unwrap()[\"seed\"], 20260924, \"the seed travels with it\");\n```",
      "properties": {
        "high": {
          "description": "Upper bound.",
          "format": "double",
          "type": "number"
        },
        "level_bp": {
          "description": "Confidence in basis points.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        },
        "low": {
          "description": "Lower bound.",
          "format": "double",
          "type": "number"
        },
        "method": {
          "description": "How it was computed: a cluster bootstrap over tasks for every metric over pairs or\nruns, whose repetitions of one task are not independent, and an independent\nbootstrap over seeds for the context suite.",
          "type": "string"
        },
        "resamples": {
          "description": "Resamples.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        },
        "seed": {
          "description": "Generator seed.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        }
      },
      "required": [
        "level_bp",
        "low",
        "high",
        "method",
        "resamples",
        "seed"
      ],
      "type": "object"
    },
    "EconomicsMetric": {
      "description": "One metric: its value, where it comes from, and how much it can bear.\n\n```\nuse majordomus_cli::economics::model::{EconomicsClass, EconomicsMetric, EconomicsMetricStatus};\nlet m: EconomicsMetric = serde_json::from_value(serde_json::json!({\n    \"id\": \"effective_token_reduction\", \"title\": \"Effective token reduction\",\n    \"class\": \"derived\", \"inputs\": \"observed\", \"unit\": \"ratio\",\n    \"formula\": \"median of 1 - treatment / control\", \"status\": \"not_measured\", \"n\": 0,\n    \"suite\": \"pilot\"\n}))\n.unwrap();\nassert_eq!(m.inputs, Some(EconomicsClass::Observed));\nassert!(m.value.is_none() && m.status == EconomicsMetricStatus::NotMeasured);\nassert!(serde_json::to_value(&m).unwrap().get(\"value\").is_none(), \"no value, not a zero\");\n```",
      "properties": {
        "class": {
          "$ref": "#/$defs/EconomicsClass",
          "description": "The measurement class."
        },
        "distribution": {
          "anyOf": [
            {
              "$ref": "#/$defs/EconomicsDistribution"
            },
            {
              "type": "null"
            }
          ],
          "description": "Their distribution."
        },
        "evidence": {
          "default": [],
          "description": "The runs, pairs or seeds it rests on.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "formula": {
          "description": "The formula, in words.",
          "type": "string"
        },
        "id": {
          "description": "The metric's id.",
          "type": "string"
        },
        "inputs": {
          "anyOf": [
            {
              "$ref": "#/$defs/EconomicsClass"
            },
            {
              "type": "null"
            }
          ],
          "description": "For a derived metric, the class of what it was derived from: a reduction derived from\nobserved usage is not the same evidence as one derived from counted bytes."
        },
        "interval": {
          "anyOf": [
            {
              "$ref": "#/$defs/EconomicsInterval"
            },
            {
              "type": "null"
            }
          ],
          "description": "The interval, when enough values exist."
        },
        "n": {
          "description": "How many values it rests on.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "not": {
          "description": "What the metric is not, where it is easily mistaken for something else.",
          "type": [
            "string",
            "null"
          ]
        },
        "status": {
          "$ref": "#/$defs/EconomicsMetricStatus",
          "description": "Where it stands."
        },
        "suite": {
          "description": "The suite.",
          "type": "string"
        },
        "title": {
          "description": "A name.",
          "type": "string"
        },
        "unit": {
          "description": "`ratio`, `tokens`, `calls`.",
          "type": "string"
        },
        "value": {
          "description": "The headline value; absent when nothing was measured.",
          "format": "double",
          "type": [
            "number",
            "null"
          ]
        },
        "warnings": {
          "default": [],
          "description": "What a reader must know before using it.",
          "items": {
            "type": "string"
          },
          "type": "array"
        }
      },
      "required": [
        "id",
        "title",
        "class",
        "unit",
        "formula",
        "status",
        "n",
        "suite"
      ],
      "type": "object"
    },
    "EconomicsMetricStatus": {
      "description": "Where a metric stands against the publication rule: whether its value may back a claim,\nis a result too thin to claim anything, or was never measured at all.\n\n```\nuse majordomus_cli::economics::model::EconomicsMetricStatus;\nassert_eq!(serde_json::to_value(EconomicsMetricStatus::NotMeasured).unwrap(), \"not_measured\");\nlet s: EconomicsMetricStatus = serde_json::from_value(\"preliminary\".into()).unwrap();\nassert_ne!(s, EconomicsMetricStatus::Verified, \"a preliminary result is not a claim\");\n```",
      "oneOf": [
        {
          "const": "verified",
          "description": "Measured, and meets the publication rule. Only the primary metric over all the\nevidence can be: the rule is evaluated for it and for nothing else, so no secondary\nmetric and no narrowed slice is ever verified.",
          "type": "string"
        },
        {
          "const": "preliminary",
          "description": "Measured, but no claim may rest on it: below the publication rule, narrowed by a\nquery, or a metric the rule was never evaluated for. A result, not a claim.",
          "type": "string"
        },
        {
          "const": "measured",
          "description": "Measured deterministically; the publication rule for sampled data does not apply.",
          "type": "string"
        },
        {
          "const": "not_measured",
          "description": "Nothing recorded to compute it from.",
          "type": "string"
        }
      ]
    },
    "EconomicsPair": {
      "description": "A control run and its treatment run, and what follows from them.\n\n```\nuse majordomus_cli::economics::model::{EconomicsPair, EconomicsPairStatus};\nlet p: EconomicsPair = serde_json::from_value(serde_json::json!({\n    \"suite\": \"pilot\", \"task\": \"vat-rounding\", \"category\": \"bugfix\", \"sessions\": 1,\n    \"repetition\": 2, \"control\": \"vat-rounding--baseline--r2\", \"status\": \"missing\",\n    \"reasons\": [\"no treatment run recorded\"]\n}))\n.unwrap();\nassert_eq!(p.status, EconomicsPairStatus::Missing);\nassert!(p.treatment.is_none() && p.token_reduction.is_none(), \"no side, no reduction\");\n```",
      "properties": {
        "category": {
          "description": "The task's category.",
          "type": "string"
        },
        "continuation_reduction": {
          "description": "The same on the tokens of the last session only: what continuing cost.",
          "format": "double",
          "type": [
            "number",
            "null"
          ]
        },
        "control": {
          "description": "The control run's id, when recorded.",
          "type": [
            "string",
            "null"
          ]
        },
        "control_usage": {
          "anyOf": [
            {
              "$ref": "#/$defs/EconomicsUsage"
            },
            {
              "type": "null"
            }
          ],
          "description": "The control's totals."
        },
        "cost_reduction": {
          "description": "The same on the harness's cost projection.",
          "format": "double",
          "type": [
            "number",
            "null"
          ]
        },
        "first_request_overhead": {
          "description": "First-request input of the treatment minus that of the control: what the treatment\nadds before the model has done anything.",
          "format": "int64",
          "type": [
            "integer",
            "null"
          ]
        },
        "reasons": {
          "default": [],
          "description": "Why not, when it does not.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "repetition": {
          "description": "The repetition.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        },
        "sessions": {
          "description": "Sessions the task spans.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "status": {
          "$ref": "#/$defs/EconomicsPairStatus",
          "description": "Whether it counts."
        },
        "suite": {
          "description": "The suite.",
          "type": "string"
        },
        "task": {
          "description": "The task.",
          "type": "string"
        },
        "token_reduction": {
          "description": "`1 - treatment / control` on total tokens. Negative: the treatment cost more.\nPresent on a valid pair, and on an excluded pair that would otherwise be valid, so\nthat the outlier rule's effect can be shown both ways; only valid pairs enter a\nmetric other than `effective_token_reduction.including_excluded`. The same holds for\nevery reduction and overhead below.",
          "format": "double",
          "type": [
            "number",
            "null"
          ]
        },
        "tool_call_reduction": {
          "description": "The same on tool calls.",
          "format": "double",
          "type": [
            "number",
            "null"
          ]
        },
        "treatment": {
          "description": "The treatment run's id, when recorded.",
          "type": [
            "string",
            "null"
          ]
        },
        "treatment_usage": {
          "anyOf": [
            {
              "$ref": "#/$defs/EconomicsUsage"
            },
            {
              "type": "null"
            }
          ],
          "description": "The treatment's totals."
        }
      },
      "required": [
        "suite",
        "task",
        "category",
        "sessions",
        "repetition",
        "status"
      ],
      "type": "object"
    },
    "EconomicsPairCounts": {
      "description": "Pair counts of a suite, so that nothing is dropped out of sight.\n\n```\nuse majordomus_cli::economics::model::EconomicsPairCounts;\nlet c = EconomicsPairCounts {\n    declared: 14, attempted: 12, valid: 9, control_failed: 1,\n    treatment_failed: 1, both_failed: 0, other: 1,\n};\nlet accounted = c.valid + c.control_failed + c.treatment_failed + c.both_failed + c.other;\nassert_eq!(accounted, c.attempted, \"every attempted pair is in exactly one bucket\");\nassert_eq!(EconomicsPairCounts::default().declared, 0);\n```",
      "properties": {
        "attempted": {
          "description": "Pairs with at least one run recorded.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "both_failed": {
          "description": "Pairs where both failed.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "control_failed": {
          "description": "Pairs whose control failed.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "declared": {
          "description": "Pairs the suite declares.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "other": {
          "description": "Pairs not comparable, missing a side, lacking usage, or excluded.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "treatment_failed": {
          "description": "Pairs whose treatment failed.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "valid": {
          "description": "Pairs that count.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        }
      },
      "required": [
        "declared",
        "attempted",
        "valid",
        "control_failed",
        "treatment_failed",
        "both_failed",
        "other"
      ],
      "type": "object"
    },
    "EconomicsPairStatus": {
      "description": "Why a pair does or does not count.\n\n```\nuse majordomus_cli::economics::model::EconomicsPairStatus;\nassert_eq!(serde_json::to_value(EconomicsPairStatus::ControlFailed).unwrap(), \"control_failed\");\nlet s: EconomicsPairStatus = serde_json::from_value(\"usage_unavailable\".into()).unwrap();\nassert_ne!(s, EconomicsPairStatus::Valid);\n```",
      "oneOf": [
        {
          "const": "valid",
          "description": "Both runs succeeded, are comparable and report usage.",
          "type": "string"
        },
        {
          "const": "control_failed",
          "description": "The control run failed a success gate.",
          "type": "string"
        },
        {
          "const": "treatment_failed",
          "description": "The treatment run failed a success gate.",
          "type": "string"
        },
        {
          "const": "both_failed",
          "description": "Both failed.",
          "type": "string"
        },
        {
          "const": "usage_unavailable",
          "description": "A run reports no usage to compare.",
          "type": "string"
        },
        {
          "const": "incomparable",
          "description": "The runs differ in something the methodology requires them to share.",
          "type": "string"
        },
        {
          "const": "missing",
          "description": "One side was never recorded.",
          "type": "string"
        },
        {
          "const": "excluded",
          "description": "A run is excluded under the outlier rule.",
          "type": "string"
        }
      ]
    },
    "EconomicsPublicationPolicy": {
      "description": "When a quantitative total-token claim may be published. Every threshold must hold.\n\n```\nuse majordomus_cli::economics::model::EconomicsPublicationPolicy;\nlet rule = serde_json::json!({\n    \"min_valid_pairs\": 30, \"min_categories\": 4, \"min_pairs_per_category\": 5,\n    \"min_repetitions\": 3, \"min_valid_pair_rate_bp\": 8000, \"max_interval_width_bp\": 2000,\n    \"require_current\": true\n});\nlet p: EconomicsPublicationPolicy = serde_json::from_value(rule.clone()).unwrap();\nassert!(p.require_current && p.min_valid_pairs == 30);\nlet mut lax = rule;\nlax.as_object_mut().unwrap().remove(\"require_current\");\nassert!(serde_json::from_value::<EconomicsPublicationPolicy>(lax).is_err(), \"no default\");\n```",
      "properties": {
        "max_interval_width_bp": {
          "description": "Width of the interval, in basis points of reduction, at most.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        },
        "min_categories": {
          "description": "Task categories with enough pairs, at least.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "min_pairs_per_category": {
          "description": "Pairs a category needs to count towards `min_categories`.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "min_repetitions": {
          "description": "Repetitions of every task and variant, at least.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        },
        "min_valid_pair_rate_bp": {
          "description": "Valid pairs over attempted pairs, in basis points, at least.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        },
        "min_valid_pairs": {
          "description": "Valid matched pairs, at least.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "require_current": {
          "description": "Whether the evidence must be current (its mechanism unchanged since it was recorded).",
          "type": "boolean"
        }
      },
      "required": [
        "min_valid_pairs",
        "min_categories",
        "min_pairs_per_category",
        "min_repetitions",
        "min_valid_pair_rate_bp",
        "max_interval_width_bp",
        "require_current"
      ],
      "type": "object"
    },
    "EconomicsSegment": {
      "description": "A metric's value within one segment of the pairs.\n\n```\nuse majordomus_cli::economics::model::EconomicsSegment;\nlet s: EconomicsSegment = serde_json::from_value(serde_json::json!({\n    \"dimension\": \"category\", \"value\": \"multi-session\", \"n\": 0\n}))\n.unwrap();\nassert!(s.token_reduction.is_none(), \"a segment with no valid pair has no distribution\");\n```",
      "properties": {
        "dimension": {
          "description": "`category`, `sessions`, `complexity`.",
          "type": "string"
        },
        "n": {
          "description": "Valid pairs in it.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "token_reduction": {
          "anyOf": [
            {
              "$ref": "#/$defs/EconomicsDistribution"
            },
            {
              "type": "null"
            }
          ],
          "description": "Distribution of per-pair token reduction."
        },
        "value": {
          "description": "The segment.",
          "type": "string"
        }
      },
      "required": [
        "dimension",
        "value",
        "n"
      ],
      "type": "object"
    },
    "EconomicsSuiteView": {
      "description": "A suite and the state of its evidence.\n\n```\nuse majordomus_cli::economics::model::{\n    EconomicsFreshness, EconomicsPairCounts, EconomicsSuiteView,\n};\nlet v = EconomicsSuiteView {\n    id: \"pilot\".into(), kind: \"live\".into(), version: 1, title: \"Pilot\".into(),\n    model: Some(\"claude-sonnet-5\".into()), freshness: EconomicsFreshness::NoEvidence,\n    freshness_detail: Some(\"no run recorded\".into()), runs: 0,\n    pairs: EconomicsPairCounts { declared: 14, ..EconomicsPairCounts::default() },\n    revisions: vec![], harnesses: vec![], models_reported: vec![],\n};\nlet json = serde_json::to_value(&v).unwrap();\nassert_eq!(json[\"freshness\"], \"no_evidence\");\nassert_eq!(json[\"pairs\"][\"declared\"], 14, \"declared pairs show even when none ran\");\n```",
      "properties": {
        "freshness": {
          "$ref": "#/$defs/EconomicsFreshness",
          "description": "Where its evidence stands."
        },
        "freshness_detail": {
          "description": "Why, when not current.",
          "type": [
            "string",
            "null"
          ]
        },
        "harnesses": {
          "default": [],
          "description": "The harness versions that ran it.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "id": {
          "description": "The suite.",
          "type": "string"
        },
        "kind": {
          "description": "`live` or `context`.",
          "type": "string"
        },
        "model": {
          "description": "The model, for a live suite.",
          "type": [
            "string",
            "null"
          ]
        },
        "models_reported": {
          "default": [],
          "description": "The models the harness reported.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "pairs": {
          "$ref": "#/$defs/EconomicsPairCounts",
          "description": "Pair counts."
        },
        "revisions": {
          "default": [],
          "description": "The revisions the evidence was recorded at.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "runs": {
          "description": "Runs recorded.",
          "format": "uint",
          "minimum": 0,
          "type": "integer"
        },
        "title": {
          "description": "A name.",
          "type": "string"
        },
        "version": {
          "description": "Its version.",
          "format": "uint32",
          "minimum": 0,
          "type": "integer"
        }
      },
      "required": [
        "id",
        "kind",
        "version",
        "title",
        "freshness",
        "runs",
        "pairs"
      ],
      "type": "object"
    },
    "EconomicsTokenizer": {
      "description": "The identity of a tokenizer, recorded with every count it produced.\n\n```\nuse majordomus_cli::economics::model::EconomicsTokenizer;\nlet t: EconomicsTokenizer = serde_json::from_value(serde_json::json!({\n    \"encoding\": \"o200k_base\", \"implementation\": \"tiktoken-rs 0.12.0\"\n}))\n.unwrap();\nlet other = EconomicsTokenizer { implementation: \"tiktoken-rs 0.13.0\".into(), ..t.clone() };\nassert_ne!(t, other, \"the same encoding from another implementation is another tokenizer\");\n```",
      "properties": {
        "encoding": {
          "description": "The encoding: `o200k_base`.",
          "type": "string"
        },
        "implementation": {
          "description": "The implementation and its pinned version.",
          "type": "string"
        }
      },
      "required": [
        "encoding",
        "implementation"
      ],
      "type": "object"
    },
    "EconomicsUsage": {
      "description": "A run's totals, derived from its raw usage.\n\n```\nuse majordomus_cli::economics::model::EconomicsUsage;\nlet u = EconomicsUsage {\n    input_uncached: 10, cache_write: 90, cache_read: 900, output: 50,\n    total_input: 1000, total: 1050, requests: 3, ..EconomicsUsage::default()\n};\nassert_eq!(u.total_input, u.input_uncached + u.cache_write + u.cache_read);\nlet json = serde_json::to_value(&u).unwrap();\nassert!(json.get(\"reasoning\").is_none() && json.get(\"cost_microusd\").is_none());\n```",
      "properties": {
        "cache_read": {
          "description": "Input read from the cache.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "cache_write": {
          "description": "Input written to the cache.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "cost_microusd": {
          "description": "The harness's cost projection, in millionths of a dollar, when every session had one.",
          "format": "uint64",
          "minimum": 0,
          "type": [
            "integer",
            "null"
          ]
        },
        "input_uncached": {
          "description": "Input neither read from nor written to the cache.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "output": {
          "description": "Output, reasoning included.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "reasoning": {
          "description": "Reasoning tokens, when every request reported them.",
          "format": "uint64",
          "minimum": 0,
          "type": [
            "integer",
            "null"
          ]
        },
        "requests": {
          "description": "Provider requests.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "tool_calls": {
          "description": "Tool calls.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "total": {
          "description": "All input and output.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        },
        "total_input": {
          "description": "All input.",
          "format": "uint64",
          "minimum": 0,
          "type": "integer"
        }
      },
      "required": [
        "input_uncached",
        "cache_write",
        "cache_read",
        "output",
        "total_input",
        "total",
        "requests",
        "tool_calls"
      ],
      "type": "object"
    },
    "EconomicsVariant": {
      "description": "One arm of the experiment: what the session runs with, and what it runs without.\n\n```\nuse majordomus_cli::economics::model::EconomicsVariant;\nlet v: EconomicsVariant = serde_json::from_value(serde_json::json!({\n    \"id\": \"baseline\", \"role\": \"control\", \"title\": \"Without Majordomus\",\n    \"description\": \"The fixture repository as a competent team keeps it.\"\n}))\n.unwrap();\nassert_eq!(v.role, \"control\");\nassert!(v.excludes.is_empty(), \"an arm that names no exclusion excludes nothing\");\n```",
      "properties": {
        "description": {
          "description": "Exactly what the arm consists of.",
          "type": "string"
        },
        "excludes": {
          "default": [],
          "description": "What the arm deliberately does not include.",
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "id": {
          "description": "`baseline`, `majordomus`, ...",
          "type": "string"
        },
        "role": {
          "description": "`control` or `treatment`.",
          "type": "string"
        },
        "title": {
          "description": "A short name.",
          "type": "string"
        }
      },
      "required": [
        "id",
        "role",
        "title",
        "description"
      ],
      "type": "object"
    },
    "EconomicsVerdict": {
      "description": "The one sentence the evidence allows, and why it allows no stronger one.\n\n```\nuse majordomus_cli::economics::model::EconomicsVerdict;\nlet v: EconomicsVerdict = serde_json::from_value(serde_json::json!({\n    \"publishable\": false,\n    \"statement\": \"No quantitative token claim: 9 valid pairs of the 30 the rule requires.\",\n    \"unmet\": [\"min_valid_pairs: 9 < 30\"]\n}))\n.unwrap();\nassert!(!v.publishable);\nassert_eq!(v.unmet.len(), 1, \"every threshold not met is named\");\n```",
      "properties": {
        "publishable": {
          "description": "Whether a quantitative total-token claim meets the publication rule.",
          "type": "boolean"
        },
        "statement": {
          "description": "The strongest statement the evidence supports, generated, never typed.",
          "type": "string"
        },
        "unmet": {
          "default": [],
          "description": "Every publication threshold not met.",
          "items": {
            "type": "string"
          },
          "type": "array"
        }
      },
      "required": [
        "publishable",
        "statement"
      ],
      "type": "object"
    }
  },
  "description": "The answer of `economics.summary`: every metric, where each comes from, and the one\nstatement the evidence allows.\n\n```\nuse majordomus_cli::economics::model::EconomicsSummary;\nlet s: EconomicsSummary = serde_json::from_value(serde_json::json!({\n    \"present\": false,\n    \"verdict\": { \"publishable\": false, \"statement\": \"No methodology is declared.\" },\n    \"metrics\": [], \"suites\": [], \"pairs\": [], \"segments\": [], \"history\": [],\n    \"variants\": [], \"hypotheses\": []\n}))\n.unwrap();\nassert!(s.methodology.is_none() && s.publication.is_none() && s.diagnostics.is_empty());\nlet json = serde_json::to_value(&s).unwrap();\nassert!(json.get(\"methodology\").is_none(), \"an absent methodology is absent, not null\");\n```",
  "properties": {
    "context": {
      "anyOf": [
        {
          "$ref": "#/$defs/EconomicsContextView"
        },
        {
          "type": "null"
        }
      ],
      "description": "The context suite in brief."
    },
    "diagnostics": {
      "default": [],
      "description": "Problems reading the declarations or the evidence.",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "history": {
      "description": "The metric over recorded measurements.",
      "items": {
        "$ref": "#/$defs/EconomicsHistoryPoint"
      },
      "type": "array"
    },
    "hypotheses": {
      "description": "The hypotheses, as hypotheses.",
      "items": {
        "$ref": "#/$defs/EconomicsHypothesis"
      },
      "type": "array"
    },
    "methodology": {
      "description": "The methodology version.",
      "format": "uint32",
      "minimum": 0,
      "type": [
        "integer",
        "null"
      ]
    },
    "metrics": {
      "description": "Every metric.",
      "items": {
        "$ref": "#/$defs/EconomicsMetric"
      },
      "type": "array"
    },
    "pairs": {
      "description": "Every pair, valid or not.",
      "items": {
        "$ref": "#/$defs/EconomicsPair"
      },
      "type": "array"
    },
    "present": {
      "description": "Whether a methodology is declared here at all.",
      "type": "boolean"
    },
    "primary_metric": {
      "description": "The primary metric's id.",
      "type": [
        "string",
        "null"
      ]
    },
    "publication": {
      "anyOf": [
        {
          "$ref": "#/$defs/EconomicsPublicationPolicy"
        },
        {
          "type": "null"
        }
      ],
      "description": "The publication rule."
    },
    "question": {
      "description": "The question.",
      "type": [
        "string",
        "null"
      ]
    },
    "segments": {
      "description": "Token reduction by segment.",
      "items": {
        "$ref": "#/$defs/EconomicsSegment"
      },
      "type": "array"
    },
    "suites": {
      "description": "Every suite and its evidence.",
      "items": {
        "$ref": "#/$defs/EconomicsSuiteView"
      },
      "type": "array"
    },
    "variants": {
      "description": "The arms of the experiment.",
      "items": {
        "$ref": "#/$defs/EconomicsVariant"
      },
      "type": "array"
    },
    "verdict": {
      "$ref": "#/$defs/EconomicsVerdict",
      "description": "What the total-token evidence allows to be said."
    }
  },
  "required": [
    "present",
    "verdict",
    "metrics",
    "suites",
    "pairs",
    "segments",
    "history",
    "variants",
    "hypotheses"
  ],
  "title": "EconomicsSummary",
  "type": "object"
}

Policies

benchmark
required — a target on every transport the exposure declares; the cases are the input type's
cache
process — up to 8 entries in the process, 5s each, scoped by the registry fingerprint

Benchmark targets

Derived from the registry for this repository: one requirement per transport, and the cases the input type provides. The whole matrix is on the benchmarks page.

transportstatecasestargets
directcovered2default cold+warm
narrowed cold+warm
mcpcovered2default cold+warm
narrowed cold+warm
httpcovered2default cold+warm
narrowed cold+warm

Searches the titles and descriptions of every page this site publishes.