economics.summary
Každá metrika benchmarku ekonomiky s hodnotou, třídou měření, velikostí vzorku, rozdělením a bootstrapovým intervalem; každá dvojice běhů s jejím stavem (valid, control failed, treatment failed, incomparable, ...); výsledky podle segmentů; stav důkazů každé sady (current, stale, incompatible); a jediné tvrzení, které pravidlo publikace dovoluje. Snížení kontextu a snížení celkových tokenů jsou samostatné metriky a nikdy se neslučují.
Každá metrika benchmarku ekonomiky s hodnotou, třídou měření, velikostí vzorku, rozdělením a bootstrapovým intervalem; každá dvojice běhů s jejím stavem (valid, control failed, treatment failed, incomparable, ...); výsledky podle segmentů; stav důkazů každé sady (current, stale, incompatible); a jediné tvrzení, které pravidlo publikace dovoluje. Snížení kontextu a snížení celkových tokenů jsou samostatné metriky a nikdy se neslučují.
- query
- behaviorally_verified
- module economics
- #economics
- #evidence
- #benchmark
Zpřístupnění
Kde je tato jedna definice dosažitelná. Absence je explicitní: povrch, který není uveden, ji nenese.
| povrch | jako |
|---|---|
| MCP | nástroj majordomus_economics zdroj majordomus://economics |
| HTTP | GET /api/v1/economics operationId economics.summary |
| příkazová řádka | majordomus economics summary |
Schémata
Kanonický vstup a výstup z typů Rustu; schéma nástroje MCP a parametry a odpovědi OpenAPI se odvozují z nich.
input · EconomicsQuery
Vstup
economics.summary: volitelné zúžení dvojic a metrik. Verdikt je vždy o všech důkazech, ať je zúženo cokoli, a zúžená metrika nikdy není verified: pravidlo publikace se vyhodnotilo nad vším, ne nad výřezem.
use majordomus_cli::economics::model::EconomicsQuery;
let all: EconomicsQuery = serde_json::from_value(serde_json::json!({})).unwrap();
assert_eq!(all, EconomicsQuery::default());
let q: EconomicsQuery =
serde_json::from_value(serde_json::json!({ "suite": "pilot", "category": "bugfix" }))
.unwrap();
assert_eq!(q.suite.as_deref(), Some("pilot"));
let typo = serde_json::json!({ "suites": "pilot" });
let refused = serde_json::from_value::<EconomicsQuery>(typo);
assert!(refused.is_err(), "a misspelt filter is refused");| vlastnost | typ | povinné | popis |
|---|---|---|---|
| kategorie | string | null | ne | Pouze úlohy této kategorie. |
| model | string | null | ne | Pouze běhy, které si tento model vyžádaly. |
| sada testů | string | null | ne | Pouze tato sada. |
| úloha | string | null | ne | Pouze tato úloha. |
JSON Schema
{
"additionalProperties": false,
"description": "The input of `economics.summary`: optional narrowing of the pairs and metrics. The\nverdict is always about all the evidence, whatever is narrowed, and a narrowed metric\nis never verified: the publication rule was evaluated over all of it, not over a slice.\n\n```\nuse majordomus_cli::economics::model::EconomicsQuery;\nlet all: EconomicsQuery = serde_json::from_value(serde_json::json!({})).unwrap();\nassert_eq!(all, EconomicsQuery::default());\nlet q: EconomicsQuery =\n serde_json::from_value(serde_json::json!({ \"suite\": \"pilot\", \"category\": \"bugfix\" }))\n .unwrap();\nassert_eq!(q.suite.as_deref(), Some(\"pilot\"));\nlet typo = serde_json::json!({ \"suites\": \"pilot\" });\nlet refused = serde_json::from_value::<EconomicsQuery>(typo);\nassert!(refused.is_err(), \"a misspelt filter is refused\");\n```",
"properties": {
"category": {
"description": "Only tasks of this category.",
"type": [
"string",
"null"
]
},
"model": {
"description": "Only runs that asked for this model.",
"type": [
"string",
"null"
]
},
"suite": {
"description": "Only this suite.",
"type": [
"string",
"null"
]
},
"task": {
"description": "Only this task.",
"type": [
"string",
"null"
]
}
},
"title": "EconomicsQuery",
"type": "object"
}output · EconomicsSummary
Odpověď
economics.summary: každá metrika, odkud pochází, a jediné tvrzení, které důkazy dovolují.
use majordomus_cli::economics::model::EconomicsSummary;
let s: EconomicsSummary = serde_json::from_value(serde_json::json!({
"present": false,
"verdict": { "publishable": false, "statement": "No methodology is declared." },
"metrics": [], "suites": [], "pairs": [], "segments": [], "history": [],
"variants": [], "hypotheses": []
}))
.unwrap();
assert!(s.methodology.is_none() && s.publication.is_none() && s.diagnostics.is_empty());
let json = serde_json::to_value(&s).unwrap();
assert!(json.get("methodology").is_none(), "an absent methodology is absent, not null");| vlastnost | typ | povinné | popis |
|---|---|---|---|
| kontext | jedno ze 2 | ne | Sada kontextu ve zkratce. |
| diagnostika | array | ne | Problémy při čtení deklarací nebo důkazů. výchozí [] |
| history | array | ano | Metrika nad zaznamenanými měřeními. |
| hypotézy | array | ano | Hypotézy jako hypotézy. |
| metodika | integer | null | ne | Verze metodiky. |
| metriky | array | ano | Každá metrika. |
| dvojice | array | ano | Každá dvojice, platná i neplatná. |
| přítomno | boolean | ano | Zda je zde vůbec deklarována nějaká metodika. |
| primary_metric | string | null | ne | Id primární metriky. |
| publikace | jedno ze 2 | ne | Pravidlo publikace. |
| otázka | string | null | ne | Otázka. |
| segmenty | array | ano | Snížení počtu tokenů podle segmentu. |
| sady | array | ano | Každá sada a její důkazy. |
| varianty | array | ano | Větve experimentu. |
| verdikt | reference | ano | Co dovolují důkazy o celkovém počtu tokenů říct. |
JSON Schema
{
"$defs": {
"EconomicsClass": {
"description": "How a number came to be. The classes are ordered from strongest to weakest, and a value\ncomputed from several inputs is never stronger than the weakest of them.\n\n```\nuse majordomus_cli::economics::model::EconomicsClass;\nassert_eq!(serde_json::to_value(EconomicsClass::Counterfactual).unwrap(), \"counterfactual\");\nlet parsed: EconomicsClass = serde_json::from_value(\"observed\".into()).unwrap();\nassert!(parsed < EconomicsClass::Estimated, \"a stronger class sorts first\");\n```",
"oneOf": [
{
"const": "observed",
"description": "Returned by the provider or its harness at run time and recorded unmodified.",
"type": "string"
},
{
"const": "counted",
"description": "Computed deterministically from bytes on disk with a named, pinned tokenizer.",
"type": "string"
},
{
"const": "derived",
"description": "Arithmetic over observed or counted values.",
"type": "string"
},
{
"const": "estimated",
"description": "A number standing in for one nobody measured. Never compared, never published.",
"type": "string"
},
{
"const": "counterfactual",
"description": "What a mechanism would have avoided, modelled rather than observed.",
"type": "string"
}
]
},
"EconomicsContextView": {
"description": "The context suite's latest measurement, in brief.\n\n```\nuse std::collections::BTreeMap;\nuse majordomus_cli::economics::model::{EconomicsContextView, EconomicsTokenizer};\nlet v = EconomicsContextView {\n revision: \"8d37d011ef\".into(), measured_at: \"2026-09-24T10:00:00Z\".into(),\n tokenizer: EconomicsTokenizer {\n encoding: \"o200k_base\".into(), implementation: \"tiktoken-rs 0.12.0\".into(),\n },\n seeds: 2, candidate_tokens: 20_000, considered_tokens: 300_000, selected_tokens: 15_000,\n estimated_selected_tokens: 16_000,\n excluded_tokens: BTreeMap::from([(\"budget\".into(), 5_000)]),\n seeds_over_budget_counted: 0, budget_tokens: 8_000,\n};\nassert_eq!(v.selected_tokens + v.excluded_tokens[\"budget\"], v.candidate_tokens);\nassert_eq!(serde_json::to_value(&v).unwrap()[\"tokenizer\"][\"encoding\"], \"o200k_base\");\n```",
"properties": {
"budget_tokens": {
"description": "The compiler's budget.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"candidate_tokens": {
"description": "Counted candidate tokens (relevant: selected or cut by the budget), all seeds.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"considered_tokens": {
"description": "Counted tokens of everything the compiler reached, all seeds.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"estimated_selected_tokens": {
"description": "The compiler's own estimate of the selected tokens, all seeds.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"excluded_tokens": {
"additionalProperties": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"description": "Counted tokens left out, by reason, all seeds.",
"type": "object"
},
"measured_at": {
"description": "When.",
"type": "string"
},
"revision": {
"description": "The revision measured.",
"type": "string"
},
"seeds": {
"description": "Seeds measured.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seeds_over_budget_counted": {
"description": "Seeds whose counted selection exceeds the compiler's budget.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"selected_tokens": {
"description": "Counted selected tokens, all seeds.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"tokenizer": {
"$ref": "#/$defs/EconomicsTokenizer",
"description": "The tokenizer."
}
},
"required": [
"revision",
"measured_at",
"tokenizer",
"seeds",
"candidate_tokens",
"considered_tokens",
"selected_tokens",
"estimated_selected_tokens",
"excluded_tokens",
"seeds_over_budget_counted",
"budget_tokens"
],
"type": "object"
},
"EconomicsDistribution": {
"description": "The shape of a set of per-pair values.\n\n```\nuse majordomus_cli::economics::model::EconomicsDistribution;\nlet d: EconomicsDistribution = serde_json::from_value(serde_json::json!({\n \"n\": 1, \"mean\": 0.4, \"median\": 0.4, \"p25\": 0.4, \"p75\": 0.4, \"p95\": 0.4,\n \"min\": 0.4, \"max\": 0.4\n}))\n.unwrap();\nassert!(d.sd.is_none(), \"one value has no sample standard deviation\");\nassert!(d.min <= d.p25 && d.p25 <= d.median && d.median <= d.p75 && d.p75 <= d.max);\n```",
"properties": {
"max": {
"description": "Largest.",
"format": "double",
"type": "number"
},
"mean": {
"description": "Arithmetic mean.",
"format": "double",
"type": "number"
},
"median": {
"description": "Median.",
"format": "double",
"type": "number"
},
"min": {
"description": "Smallest.",
"format": "double",
"type": "number"
},
"n": {
"description": "How many values.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"p25": {
"description": "25th percentile.",
"format": "double",
"type": "number"
},
"p75": {
"description": "75th percentile.",
"format": "double",
"type": "number"
},
"p95": {
"description": "95th percentile.",
"format": "double",
"type": "number"
},
"sd": {
"description": "Sample standard deviation; absent for one value.",
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"n",
"mean",
"median",
"p25",
"p75",
"p95",
"min",
"max"
],
"type": "object"
},
"EconomicsFreshness": {
"description": "How recorded evidence stands against the repository as it is now.\n\n```\nuse majordomus_cli::economics::model::EconomicsFreshness;\nassert_eq!(serde_json::to_value(EconomicsFreshness::NoEvidence).unwrap(), \"no_evidence\");\nlet f: EconomicsFreshness = serde_json::from_value(\"incompatible\".into()).unwrap();\nassert_eq!(f, EconomicsFreshness::Incompatible);\n```",
"oneOf": [
{
"const": "current",
"description": "Recorded under this methodology, and nothing it depends on changed since.",
"type": "string"
},
{
"const": "stale",
"description": "Recorded under this methodology, but a mechanism it measured changed since.",
"type": "string"
},
{
"const": "incompatible",
"description": "Recorded under another methodology version: not comparable at all.",
"type": "string"
},
{
"const": "no_evidence",
"description": "Nothing recorded.",
"type": "string"
}
]
},
"EconomicsHistoryPoint": {
"description": "A recorded measurement over time: one entry per recorded context run.\n\n```\nuse majordomus_cli::economics::model::EconomicsHistoryPoint;\nlet p: EconomicsHistoryPoint = serde_json::from_value(serde_json::json!({\n \"suite\": \"context\", \"methodology\": 1, \"revision\": \"8d37d011ef\",\n \"at\": \"2026-09-24T10:00:00Z\", \"metric\": \"context_reduction_ratio\", \"value\": 0.62, \"n\": 40\n}))\n.unwrap();\nassert_eq!((p.methodology, p.n), (1, 40));\nassert_eq!(serde_json::to_value(&p).unwrap()[\"metric\"], \"context_reduction_ratio\");\n```",
"properties": {
"at": {
"description": "When.",
"type": "string"
},
"methodology": {
"description": "The methodology version, so that incompatible versions are never drawn as one line.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"metric": {
"description": "The metric.",
"type": "string"
},
"n": {
"description": "How many values it rests on.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"revision": {
"description": "The revision.",
"type": "string"
},
"suite": {
"description": "The suite.",
"type": "string"
},
"value": {
"description": "Its value.",
"format": "double",
"type": "number"
}
},
"required": [
"suite",
"methodology",
"revision",
"at",
"metric",
"value",
"n"
],
"type": "object"
},
"EconomicsHypothesis": {
"description": "A statement the benchmark exists to test. Never a result.\n\n```\nuse majordomus_cli::economics::model::EconomicsHypothesis;\nlet h: EconomicsHypothesis = serde_json::from_value(serde_json::json!({\n \"id\": \"continuity\",\n \"statement\": \"Work that spans sessions gains the most.\",\n \"status\": \"untested\"\n}))\n.unwrap();\nassert_eq!(h.status, \"untested\");\nassert_eq!(serde_json::to_value(&h).unwrap()[\"id\"], \"continuity\");\n```",
"properties": {
"id": {
"description": "A slug.",
"type": "string"
},
"statement": {
"description": "What is hypothesised.",
"type": "string"
},
"status": {
"description": "`untested`, `supported`, `refuted` — set by a person reading the evidence.",
"type": "string"
}
},
"required": [
"id",
"statement",
"status"
],
"type": "object"
},
"EconomicsInterval": {
"description": "A bootstrap interval, with everything needed to reproduce it.\n\n```\nuse majordomus_cli::economics::model::EconomicsInterval;\nlet i: EconomicsInterval = serde_json::from_value(serde_json::json!({\n \"level_bp\": 9500, \"low\": 0.21, \"high\": 0.38,\n \"method\": \"bootstrap percentile of the median\", \"resamples\": 10000, \"seed\": 20260924\n}))\n.unwrap();\nassert!(i.low <= i.high);\nassert_eq!(serde_json::to_value(&i).unwrap()[\"seed\"], 20260924, \"the seed travels with it\");\n```",
"properties": {
"high": {
"description": "Upper bound.",
"format": "double",
"type": "number"
},
"level_bp": {
"description": "Confidence in basis points.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"low": {
"description": "Lower bound.",
"format": "double",
"type": "number"
},
"method": {
"description": "How it was computed: a cluster bootstrap over tasks for every metric over pairs or\nruns, whose repetitions of one task are not independent, and an independent\nbootstrap over seeds for the context suite.",
"type": "string"
},
"resamples": {
"description": "Resamples.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"description": "Generator seed.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"level_bp",
"low",
"high",
"method",
"resamples",
"seed"
],
"type": "object"
},
"EconomicsMetric": {
"description": "One metric: its value, where it comes from, and how much it can bear.\n\n```\nuse majordomus_cli::economics::model::{EconomicsClass, EconomicsMetric, EconomicsMetricStatus};\nlet m: EconomicsMetric = serde_json::from_value(serde_json::json!({\n \"id\": \"effective_token_reduction\", \"title\": \"Effective token reduction\",\n \"class\": \"derived\", \"inputs\": \"observed\", \"unit\": \"ratio\",\n \"formula\": \"median of 1 - treatment / control\", \"status\": \"not_measured\", \"n\": 0,\n \"suite\": \"pilot\"\n}))\n.unwrap();\nassert_eq!(m.inputs, Some(EconomicsClass::Observed));\nassert!(m.value.is_none() && m.status == EconomicsMetricStatus::NotMeasured);\nassert!(serde_json::to_value(&m).unwrap().get(\"value\").is_none(), \"no value, not a zero\");\n```",
"properties": {
"class": {
"$ref": "#/$defs/EconomicsClass",
"description": "The measurement class."
},
"distribution": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsDistribution"
},
{
"type": "null"
}
],
"description": "Their distribution."
},
"evidence": {
"default": [],
"description": "The runs, pairs or seeds it rests on.",
"items": {
"type": "string"
},
"type": "array"
},
"formula": {
"description": "The formula, in words.",
"type": "string"
},
"id": {
"description": "The metric's id.",
"type": "string"
},
"inputs": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsClass"
},
{
"type": "null"
}
],
"description": "For a derived metric, the class of what it was derived from: a reduction derived from\nobserved usage is not the same evidence as one derived from counted bytes."
},
"interval": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsInterval"
},
{
"type": "null"
}
],
"description": "The interval, when enough values exist."
},
"n": {
"description": "How many values it rests on.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not": {
"description": "What the metric is not, where it is easily mistaken for something else.",
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/$defs/EconomicsMetricStatus",
"description": "Where it stands."
},
"suite": {
"description": "The suite.",
"type": "string"
},
"title": {
"description": "A name.",
"type": "string"
},
"unit": {
"description": "`ratio`, `tokens`, `calls`.",
"type": "string"
},
"value": {
"description": "The headline value; absent when nothing was measured.",
"format": "double",
"type": [
"number",
"null"
]
},
"warnings": {
"default": [],
"description": "What a reader must know before using it.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"id",
"title",
"class",
"unit",
"formula",
"status",
"n",
"suite"
],
"type": "object"
},
"EconomicsMetricStatus": {
"description": "Where a metric stands against the publication rule: whether its value may back a claim,\nis a result too thin to claim anything, or was never measured at all.\n\n```\nuse majordomus_cli::economics::model::EconomicsMetricStatus;\nassert_eq!(serde_json::to_value(EconomicsMetricStatus::NotMeasured).unwrap(), \"not_measured\");\nlet s: EconomicsMetricStatus = serde_json::from_value(\"preliminary\".into()).unwrap();\nassert_ne!(s, EconomicsMetricStatus::Verified, \"a preliminary result is not a claim\");\n```",
"oneOf": [
{
"const": "verified",
"description": "Measured, and meets the publication rule. Only the primary metric over all the\nevidence can be: the rule is evaluated for it and for nothing else, so no secondary\nmetric and no narrowed slice is ever verified.",
"type": "string"
},
{
"const": "preliminary",
"description": "Measured, but no claim may rest on it: below the publication rule, narrowed by a\nquery, or a metric the rule was never evaluated for. A result, not a claim.",
"type": "string"
},
{
"const": "measured",
"description": "Measured deterministically; the publication rule for sampled data does not apply.",
"type": "string"
},
{
"const": "not_measured",
"description": "Nothing recorded to compute it from.",
"type": "string"
}
]
},
"EconomicsPair": {
"description": "A control run and its treatment run, and what follows from them.\n\n```\nuse majordomus_cli::economics::model::{EconomicsPair, EconomicsPairStatus};\nlet p: EconomicsPair = serde_json::from_value(serde_json::json!({\n \"suite\": \"pilot\", \"task\": \"vat-rounding\", \"category\": \"bugfix\", \"sessions\": 1,\n \"repetition\": 2, \"control\": \"vat-rounding--baseline--r2\", \"status\": \"missing\",\n \"reasons\": [\"no treatment run recorded\"]\n}))\n.unwrap();\nassert_eq!(p.status, EconomicsPairStatus::Missing);\nassert!(p.treatment.is_none() && p.token_reduction.is_none(), \"no side, no reduction\");\n```",
"properties": {
"category": {
"description": "The task's category.",
"type": "string"
},
"continuation_reduction": {
"description": "The same on the tokens of the last session only: what continuing cost.",
"format": "double",
"type": [
"number",
"null"
]
},
"control": {
"description": "The control run's id, when recorded.",
"type": [
"string",
"null"
]
},
"control_usage": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsUsage"
},
{
"type": "null"
}
],
"description": "The control's totals."
},
"cost_reduction": {
"description": "The same on the harness's cost projection.",
"format": "double",
"type": [
"number",
"null"
]
},
"first_request_overhead": {
"description": "First-request input of the treatment minus that of the control: what the treatment\nadds before the model has done anything.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"reasons": {
"default": [],
"description": "Why not, when it does not.",
"items": {
"type": "string"
},
"type": "array"
},
"repetition": {
"description": "The repetition.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"sessions": {
"description": "Sessions the task spans.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"status": {
"$ref": "#/$defs/EconomicsPairStatus",
"description": "Whether it counts."
},
"suite": {
"description": "The suite.",
"type": "string"
},
"task": {
"description": "The task.",
"type": "string"
},
"token_reduction": {
"description": "`1 - treatment / control` on total tokens. Negative: the treatment cost more.\nPresent on a valid pair, and on an excluded pair that would otherwise be valid, so\nthat the outlier rule's effect can be shown both ways; only valid pairs enter a\nmetric other than `effective_token_reduction.including_excluded`. The same holds for\nevery reduction and overhead below.",
"format": "double",
"type": [
"number",
"null"
]
},
"tool_call_reduction": {
"description": "The same on tool calls.",
"format": "double",
"type": [
"number",
"null"
]
},
"treatment": {
"description": "The treatment run's id, when recorded.",
"type": [
"string",
"null"
]
},
"treatment_usage": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsUsage"
},
{
"type": "null"
}
],
"description": "The treatment's totals."
}
},
"required": [
"suite",
"task",
"category",
"sessions",
"repetition",
"status"
],
"type": "object"
},
"EconomicsPairCounts": {
"description": "Pair counts of a suite, so that nothing is dropped out of sight.\n\n```\nuse majordomus_cli::economics::model::EconomicsPairCounts;\nlet c = EconomicsPairCounts {\n declared: 14, attempted: 12, valid: 9, control_failed: 1,\n treatment_failed: 1, both_failed: 0, other: 1,\n};\nlet accounted = c.valid + c.control_failed + c.treatment_failed + c.both_failed + c.other;\nassert_eq!(accounted, c.attempted, \"every attempted pair is in exactly one bucket\");\nassert_eq!(EconomicsPairCounts::default().declared, 0);\n```",
"properties": {
"attempted": {
"description": "Pairs with at least one run recorded.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"both_failed": {
"description": "Pairs where both failed.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"control_failed": {
"description": "Pairs whose control failed.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"declared": {
"description": "Pairs the suite declares.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"other": {
"description": "Pairs not comparable, missing a side, lacking usage, or excluded.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"treatment_failed": {
"description": "Pairs whose treatment failed.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"description": "Pairs that count.",
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"declared",
"attempted",
"valid",
"control_failed",
"treatment_failed",
"both_failed",
"other"
],
"type": "object"
},
"EconomicsPairStatus": {
"description": "Why a pair does or does not count.\n\n```\nuse majordomus_cli::economics::model::EconomicsPairStatus;\nassert_eq!(serde_json::to_value(EconomicsPairStatus::ControlFailed).unwrap(), \"control_failed\");\nlet s: EconomicsPairStatus = serde_json::from_value(\"usage_unavailable\".into()).unwrap();\nassert_ne!(s, EconomicsPairStatus::Valid);\n```",
"oneOf": [
{
"const": "valid",
"description": "Both runs succeeded, are comparable and report usage.",
"type": "string"
},
{
"const": "control_failed",
"description": "The control run failed a success gate.",
"type": "string"
},
{
"const": "treatment_failed",
"description": "The treatment run failed a success gate.",
"type": "string"
},
{
"const": "both_failed",
"description": "Both failed.",
"type": "string"
},
{
"const": "usage_unavailable",
"description": "A run reports no usage to compare.",
"type": "string"
},
{
"const": "incomparable",
"description": "The runs differ in something the methodology requires them to share.",
"type": "string"
},
{
"const": "missing",
"description": "One side was never recorded.",
"type": "string"
},
{
"const": "excluded",
"description": "A run is excluded under the outlier rule.",
"type": "string"
}
]
},
"EconomicsPublicationPolicy": {
"description": "When a quantitative total-token claim may be published. Every threshold must hold.\n\n```\nuse majordomus_cli::economics::model::EconomicsPublicationPolicy;\nlet rule = serde_json::json!({\n \"min_valid_pairs\": 30, \"min_categories\": 4, \"min_pairs_per_category\": 5,\n \"min_repetitions\": 3, \"min_valid_pair_rate_bp\": 8000, \"max_interval_width_bp\": 2000,\n \"require_current\": true\n});\nlet p: EconomicsPublicationPolicy = serde_json::from_value(rule.clone()).unwrap();\nassert!(p.require_current && p.min_valid_pairs == 30);\nlet mut lax = rule;\nlax.as_object_mut().unwrap().remove(\"require_current\");\nassert!(serde_json::from_value::<EconomicsPublicationPolicy>(lax).is_err(), \"no default\");\n```",
"properties": {
"max_interval_width_bp": {
"description": "Width of the interval, in basis points of reduction, at most.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"min_categories": {
"description": "Task categories with enough pairs, at least.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"min_pairs_per_category": {
"description": "Pairs a category needs to count towards `min_categories`.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"min_repetitions": {
"description": "Repetitions of every task and variant, at least.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"min_valid_pair_rate_bp": {
"description": "Valid pairs over attempted pairs, in basis points, at least.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"min_valid_pairs": {
"description": "Valid matched pairs, at least.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"require_current": {
"description": "Whether the evidence must be current (its mechanism unchanged since it was recorded).",
"type": "boolean"
}
},
"required": [
"min_valid_pairs",
"min_categories",
"min_pairs_per_category",
"min_repetitions",
"min_valid_pair_rate_bp",
"max_interval_width_bp",
"require_current"
],
"type": "object"
},
"EconomicsSegment": {
"description": "A metric's value within one segment of the pairs.\n\n```\nuse majordomus_cli::economics::model::EconomicsSegment;\nlet s: EconomicsSegment = serde_json::from_value(serde_json::json!({\n \"dimension\": \"category\", \"value\": \"multi-session\", \"n\": 0\n}))\n.unwrap();\nassert!(s.token_reduction.is_none(), \"a segment with no valid pair has no distribution\");\n```",
"properties": {
"dimension": {
"description": "`category`, `sessions`, `complexity`.",
"type": "string"
},
"n": {
"description": "Valid pairs in it.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"token_reduction": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsDistribution"
},
{
"type": "null"
}
],
"description": "Distribution of per-pair token reduction."
},
"value": {
"description": "The segment.",
"type": "string"
}
},
"required": [
"dimension",
"value",
"n"
],
"type": "object"
},
"EconomicsSuiteView": {
"description": "A suite and the state of its evidence.\n\n```\nuse majordomus_cli::economics::model::{\n EconomicsFreshness, EconomicsPairCounts, EconomicsSuiteView,\n};\nlet v = EconomicsSuiteView {\n id: \"pilot\".into(), kind: \"live\".into(), version: 1, title: \"Pilot\".into(),\n model: Some(\"claude-sonnet-5\".into()), freshness: EconomicsFreshness::NoEvidence,\n freshness_detail: Some(\"no run recorded\".into()), runs: 0,\n pairs: EconomicsPairCounts { declared: 14, ..EconomicsPairCounts::default() },\n revisions: vec![], harnesses: vec![], models_reported: vec![],\n};\nlet json = serde_json::to_value(&v).unwrap();\nassert_eq!(json[\"freshness\"], \"no_evidence\");\nassert_eq!(json[\"pairs\"][\"declared\"], 14, \"declared pairs show even when none ran\");\n```",
"properties": {
"freshness": {
"$ref": "#/$defs/EconomicsFreshness",
"description": "Where its evidence stands."
},
"freshness_detail": {
"description": "Why, when not current.",
"type": [
"string",
"null"
]
},
"harnesses": {
"default": [],
"description": "The harness versions that ran it.",
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"description": "The suite.",
"type": "string"
},
"kind": {
"description": "`live` or `context`.",
"type": "string"
},
"model": {
"description": "The model, for a live suite.",
"type": [
"string",
"null"
]
},
"models_reported": {
"default": [],
"description": "The models the harness reported.",
"items": {
"type": "string"
},
"type": "array"
},
"pairs": {
"$ref": "#/$defs/EconomicsPairCounts",
"description": "Pair counts."
},
"revisions": {
"default": [],
"description": "The revisions the evidence was recorded at.",
"items": {
"type": "string"
},
"type": "array"
},
"runs": {
"description": "Runs recorded.",
"format": "uint",
"minimum": 0,
"type": "integer"
},
"title": {
"description": "A name.",
"type": "string"
},
"version": {
"description": "Its version.",
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"kind",
"version",
"title",
"freshness",
"runs",
"pairs"
],
"type": "object"
},
"EconomicsTokenizer": {
"description": "The identity of a tokenizer, recorded with every count it produced.\n\n```\nuse majordomus_cli::economics::model::EconomicsTokenizer;\nlet t: EconomicsTokenizer = serde_json::from_value(serde_json::json!({\n \"encoding\": \"o200k_base\", \"implementation\": \"tiktoken-rs 0.12.0\"\n}))\n.unwrap();\nlet other = EconomicsTokenizer { implementation: \"tiktoken-rs 0.13.0\".into(), ..t.clone() };\nassert_ne!(t, other, \"the same encoding from another implementation is another tokenizer\");\n```",
"properties": {
"encoding": {
"description": "The encoding: `o200k_base`.",
"type": "string"
},
"implementation": {
"description": "The implementation and its pinned version.",
"type": "string"
}
},
"required": [
"encoding",
"implementation"
],
"type": "object"
},
"EconomicsUsage": {
"description": "A run's totals, derived from its raw usage.\n\n```\nuse majordomus_cli::economics::model::EconomicsUsage;\nlet u = EconomicsUsage {\n input_uncached: 10, cache_write: 90, cache_read: 900, output: 50,\n total_input: 1000, total: 1050, requests: 3, ..EconomicsUsage::default()\n};\nassert_eq!(u.total_input, u.input_uncached + u.cache_write + u.cache_read);\nlet json = serde_json::to_value(&u).unwrap();\nassert!(json.get(\"reasoning\").is_none() && json.get(\"cost_microusd\").is_none());\n```",
"properties": {
"cache_read": {
"description": "Input read from the cache.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"cache_write": {
"description": "Input written to the cache.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"cost_microusd": {
"description": "The harness's cost projection, in millionths of a dollar, when every session had one.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"input_uncached": {
"description": "Input neither read from nor written to the cache.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"output": {
"description": "Output, reasoning included.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"reasoning": {
"description": "Reasoning tokens, when every request reported them.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"requests": {
"description": "Provider requests.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"tool_calls": {
"description": "Tool calls.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"total": {
"description": "All input and output.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"total_input": {
"description": "All input.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"input_uncached",
"cache_write",
"cache_read",
"output",
"total_input",
"total",
"requests",
"tool_calls"
],
"type": "object"
},
"EconomicsVariant": {
"description": "One arm of the experiment: what the session runs with, and what it runs without.\n\n```\nuse majordomus_cli::economics::model::EconomicsVariant;\nlet v: EconomicsVariant = serde_json::from_value(serde_json::json!({\n \"id\": \"baseline\", \"role\": \"control\", \"title\": \"Without Majordomus\",\n \"description\": \"The fixture repository as a competent team keeps it.\"\n}))\n.unwrap();\nassert_eq!(v.role, \"control\");\nassert!(v.excludes.is_empty(), \"an arm that names no exclusion excludes nothing\");\n```",
"properties": {
"description": {
"description": "Exactly what the arm consists of.",
"type": "string"
},
"excludes": {
"default": [],
"description": "What the arm deliberately does not include.",
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"description": "`baseline`, `majordomus`, ...",
"type": "string"
},
"role": {
"description": "`control` or `treatment`.",
"type": "string"
},
"title": {
"description": "A short name.",
"type": "string"
}
},
"required": [
"id",
"role",
"title",
"description"
],
"type": "object"
},
"EconomicsVerdict": {
"description": "The one sentence the evidence allows, and why it allows no stronger one.\n\n```\nuse majordomus_cli::economics::model::EconomicsVerdict;\nlet v: EconomicsVerdict = serde_json::from_value(serde_json::json!({\n \"publishable\": false,\n \"statement\": \"No quantitative token claim: 9 valid pairs of the 30 the rule requires.\",\n \"unmet\": [\"min_valid_pairs: 9 < 30\"]\n}))\n.unwrap();\nassert!(!v.publishable);\nassert_eq!(v.unmet.len(), 1, \"every threshold not met is named\");\n```",
"properties": {
"publishable": {
"description": "Whether a quantitative total-token claim meets the publication rule.",
"type": "boolean"
},
"statement": {
"description": "The strongest statement the evidence supports, generated, never typed.",
"type": "string"
},
"unmet": {
"default": [],
"description": "Every publication threshold not met.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"publishable",
"statement"
],
"type": "object"
}
},
"description": "The answer of `economics.summary`: every metric, where each comes from, and the one\nstatement the evidence allows.\n\n```\nuse majordomus_cli::economics::model::EconomicsSummary;\nlet s: EconomicsSummary = serde_json::from_value(serde_json::json!({\n \"present\": false,\n \"verdict\": { \"publishable\": false, \"statement\": \"No methodology is declared.\" },\n \"metrics\": [], \"suites\": [], \"pairs\": [], \"segments\": [], \"history\": [],\n \"variants\": [], \"hypotheses\": []\n}))\n.unwrap();\nassert!(s.methodology.is_none() && s.publication.is_none() && s.diagnostics.is_empty());\nlet json = serde_json::to_value(&s).unwrap();\nassert!(json.get(\"methodology\").is_none(), \"an absent methodology is absent, not null\");\n```",
"properties": {
"context": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsContextView"
},
{
"type": "null"
}
],
"description": "The context suite in brief."
},
"diagnostics": {
"default": [],
"description": "Problems reading the declarations or the evidence.",
"items": {
"type": "string"
},
"type": "array"
},
"history": {
"description": "The metric over recorded measurements.",
"items": {
"$ref": "#/$defs/EconomicsHistoryPoint"
},
"type": "array"
},
"hypotheses": {
"description": "The hypotheses, as hypotheses.",
"items": {
"$ref": "#/$defs/EconomicsHypothesis"
},
"type": "array"
},
"methodology": {
"description": "The methodology version.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"metrics": {
"description": "Every metric.",
"items": {
"$ref": "#/$defs/EconomicsMetric"
},
"type": "array"
},
"pairs": {
"description": "Every pair, valid or not.",
"items": {
"$ref": "#/$defs/EconomicsPair"
},
"type": "array"
},
"present": {
"description": "Whether a methodology is declared here at all.",
"type": "boolean"
},
"primary_metric": {
"description": "The primary metric's id.",
"type": [
"string",
"null"
]
},
"publication": {
"anyOf": [
{
"$ref": "#/$defs/EconomicsPublicationPolicy"
},
{
"type": "null"
}
],
"description": "The publication rule."
},
"question": {
"description": "The question.",
"type": [
"string",
"null"
]
},
"segments": {
"description": "Token reduction by segment.",
"items": {
"$ref": "#/$defs/EconomicsSegment"
},
"type": "array"
},
"suites": {
"description": "Every suite and its evidence.",
"items": {
"$ref": "#/$defs/EconomicsSuiteView"
},
"type": "array"
},
"variants": {
"description": "The arms of the experiment.",
"items": {
"$ref": "#/$defs/EconomicsVariant"
},
"type": "array"
},
"verdict": {
"$ref": "#/$defs/EconomicsVerdict",
"description": "What the total-token evidence allows to be said."
}
},
"required": [
"present",
"verdict",
"metrics",
"suites",
"pairs",
"segments",
"history",
"variants",
"hypotheses"
],
"title": "EconomicsSummary",
"type": "object"
}Politiky
- benchmark
- povinné — cíl na každém transportu, který expozice deklaruje; případy jsou ty ze vstupního typu
- cache
- process — až 8 položek v procesu, každá 5 s, v rozsahu otisku registru
Cíle benchmarku
Odvozeno z registru pro tento repozitář: jeden požadavek na transport a případy, které poskytuje typ vstupu. Celá matice je na stránce měření výkonu.