Skip to content

A result nobody can reproduce, including its author

The result is recorded and the conditions that produced it are not, so it is an anecdote rather than a measurement.

→ could not reproduce a result because nothing recorded the state it came from

cost when it happens
high
how often
occasional
area
observability
verification
recognised by
research-team
enterprise
platform-team
engineering-lead

The moment

The figure has been quoted in three places for a year. Somebody finally tries to reproduce it and gets something 40% different. Nobody can say whether the system changed, the machine changed, or the original measurement was taken differently — because none of that was recorded with the number.

Why it happens

A number is easy to write down and its conditions are not, so the number travels and the conditions stay behind. The moment it appears in a second document it has lost the last of its context, and from then on it is quoted rather than measured.

Why a better model does not fix it

Nothing about the measurement was a reasoning task. The conditions existed and were not captured; no later analysis can recover them.

What it costs

Decisions made on a number that means something different from what everyone thinks. And the specific waste of re-measuring, which is only discovered to be necessary when a result finally contradicts itself in public.

What Majordomus does

A performance claim names a measurement, and the measurement is compared against a baseline that is a tracked file a run wrote — one per platform, because a figure from a laptop cannot refute a figure from a runner. The sampling and the regression thresholds are data beside it, so a threshold is changed deliberately rather than tuned until a gate passes. Evidence elsewhere follows the same discipline: plan evidence refuses narrative and needs a command or an artifact, and records the result with the commit. An execution episode records what it ran under.

Before and after

before   "about 40ms"        (which machine? which build? which input?)

after    .ai/repo/benchmarks/rust/baseline.macos-aarch64-debug.json
         written by: majordomus bench baseline update  (refuses a dirty tree)
         compared by: majordomus bench --check         (policy.yaml thresholds)

What it does not do

It does not make a measurement meaningful, and a baseline recorded on a noisy machine is a noisy baseline. It refuses a number that no run produced, and refuses to record one from a dirty tree.

What this looks like

Concrete situations, one per audience. Each is declared in the moment's front matter, so the before and the after are data rather than prose a page could drift from.

  1. The number in the slide platform-team

    before
    A latency figure is quoted for a year; nobody can say which machine, which build or which input produced it.
    after
    A baseline is a tracked file a run wrote, per platform, with the sampling policy beside it; a threshold is set from a run rather than guessed.

  2. The conditions were the experiment research-team

    before
    A promising result is reported; the prompt, the model and the repository state that produced it were not captured.
    after
    The session records what it ran under, and evidence names the command that produced it and the commit it applied to.

  3. A measurement used in a decision enterprise

    before
    A capacity decision rests on a figure whose provenance is a message.
    after
    Evidence is refused unless it names a command or an artifact, and the recorded result carries the exit code and the commit.

How you would know

The observable symptoms this moment declares. They are the questionnaire on the index and the input of majordomus why diagnose; nothing else defines them.

  • A measurement is quoted without the conditions that produced it. number-without-conditions
  • A result could not be repeated and nobody can say what differed. cannot-repeat
  • A performance baseline was written by hand rather than produced by a run. baseline-by-hand

Where this lives in the tool

Everything below is read out of this moment's own front matter and resolved against the repository. A name here that did not exist would fail validation.

the commands that answer it

the capabilities of the executable that answer it

what it supervises — derived from the claims below

the claims that back this page, and the evidence behind each

the rules that govern it

  • project.performance-evidence
  • project.benchmarkable-commands
  • majordomus.session-records
  • majordomus.verify-outcomes

the use cases that show the way out

If this one is familiar, so is the next

What this moment names, what names it, and what shares its area, audience or tags. The second and third are derived; only the first is written down.

All 38, and how they connect to the tool →