Expand description
The typed records of the economics subsystem: the declarations it reads (methodology, suites, tasks), the raw facts it records (runs), and the answer it derives (summary).
Raw facts and derived answers are different types on purpose. A run holds what the provider reported and what the success gates found, and nothing computed from them; a pair, a metric and a summary are recomputed from runs on every read, so a change to how a number is derived never needs the runs rewritten.
Every public type is prefixed Economics: the OpenAPI document keeps one flat namespace
of schema names, and Token* and Evidence* already belong to other subsystems.
use majordomus_cli::economics::model::EconomicsClass;
assert!(EconomicsClass::Observed.is_measurement());
assert!(!EconomicsClass::Counterfactual.is_measurement());
assert_eq!(EconomicsClass::Derived.weakest(EconomicsClass::Estimated), EconomicsClass::Estimated);Structs§
- Economics
Check - One success gate’s verdict on a run.
- Economics
Claim Binding - A published claim bound to the metric that must support it.
- Economics
Class Definition - One measurement class as the methodology defines it.
- Economics
Context Run - One recorded measurement of the context suite: every seed the compiler compiled, counted, and every seed it refused, named with its reason rather than dropped.
- Economics
Context Seed - Token counts of one context-compilation seed.
- Economics
Context View - The context suite’s latest measurement, in brief.
- Economics
Distribution - The shape of a set of per-pair values.
- Economics
Exclusion - One run excluded under the outlier rule.
- Economics
Explain Input - The input of
economics.explain: the id of the one metric to explain. Nothing else is accepted, so a misspelt field is refused rather than silently ignored. - Economics
Explanation - The answer of
economics.explain: one metric, and everything it rests on. - Economics
Gate Definition - One gate a run must pass to count as a success.
- Economics
Harness - The harness that ran a session, as it identified itself.
- Economics
History Point - A recorded measurement over time: one entry per recorded context run.
- Economics
Hypothesis - A statement the benchmark exists to test. Never a result.
- Economics
Interval - A bootstrap interval, with everything needed to reproduce it.
- Economics
Methodology .ai/repo/benchmarks/economics/methodology.yaml: what is measured, how, and when a result may be published.- Economics
Methodology Change - One entry of the methodology’s change log.
- Economics
Metric - One metric: its value, where it comes from, and how much it can bear.
- Economics
Model Total - A model’s usage over a whole session, as the harness totalled it (
modelUsage), including models the harness called on its own account. - Economics
Orientation - What a session did before its first edit: the cost of getting oriented.
- Economics
Outcome - Whether the run did the task.
- Economics
Outlier Policy - The rule for excluding runs, and the runs it excluded.
- Economics
Pair - A control run and its treatment run, and what follows from them.
- Economics
Pair Counts - Pair counts of a suite, so that nothing is dropped out of sight.
- Economics
Pairing - How runs are paired and when a pair is comparable and valid.
- Economics
Publication Policy - When a quantitative total-token claim may be published. Every threshold must hold.
- Economics
Query - The input of
economics.summary: optional narrowing of the pairs and metrics. The verdict is always about all the evidence, whatever is narrowed, and a narrowed metric is never verified: the publication rule was evaluated over all of it, not over a slice. - Economics
Request - The token usage of one provider request, provider-native field names kept.
- Economics
Revision - The repository revision a run was recorded at.
- Economics
Run - One recorded run: a task, one variant, one repetition, every session of it. Raw facts only; nothing here is derived from another field.
- Economics
RunList - The answer of
economics.runs: the raw facts every metric is computed from. - Economics
RunView - One run in brief, with its derived totals.
- Economics
Runs Query - The input of
economics.runs: optional filters on suite, task and variant, all of which must match. An empty query lists every recorded run, and an unknown field is refused. - Economics
Segment - A metric’s value within one segment of the pairs.
- Economics
Session - One session of a run: one harness invocation on one prompt, with every provider request it made, the harness’s own totals and the tools it called. A count the harness did not report is absent rather than zero.
- Economics
Statistics Policy - How per-pair results are aggregated into a headline value and its interval. Every parameter of the bootstrap is declared here, its generator seed included, so the same runs always give the same interval.
- Economics
Suite - A suite, as
.ai/repo/benchmarks/economics/suites/<id>.yamldeclares it: what is run, with which model and arms, and which tracked paths decide whether its evidence is still current. Alivesuite fills the session fields and acontextsuite the seed and tokenizer fields; the other kind’s fields stay absent and are not written back. - Economics
Suite View - A suite and the state of its evidence.
- Economics
Summary - The answer of
economics.summary: every metric, where each comes from, and the one statement the evidence allows. - Economics
Task - A task, as
.ai/repo/benchmarks/economics/tasks/<id>/task.yamldeclares it: the fixture a run starts from, the prompt of each session in order, and the hidden acceptance tests and verify command that decide whether the run succeeded. - Economics
Task Session - One session of a task: the prompt file it starts from.
- Economics
Tokenizer - The identity of a tokenizer, recorded with every count it produced.
- Economics
Usage - A run’s totals, derived from its raw usage.
- Economics
Variant - One arm of the experiment: what the session runs with, and what it runs without.
- Economics
Verdict - The one sentence the evidence allows, and why it allows no stronger one.
Enums§
- Economics
Class - How a number came to be. The classes are ordered from strongest to weakest, and a value computed from several inputs is never stronger than the weakest of them.
- Economics
Freshness - How recorded evidence stands against the repository as it is now.
- Economics
Metric Status - Where a metric stands against the publication rule: whether its value may back a claim, is a result too thin to claim anything, or was never measured at all.
- Economics
Pair Status - Why a pair does or does not count.