Skip to content

A failure that was seen once and never again

A failure observed inside a session is described in that session and nowhere else, so the next one starts from the report rather than the evidence.

→ lost a failure a worker had already reproduced, because the session ended

cost when it happens
medium
how often
common

The moment

At three in the morning a worker reproduced the intermittent failure and said so clearly. By nine the session is gone. What survives is a sentence describing a failure that nobody can now make happen.

Why it happens

Reproduction is the expensive part of debugging and it lives in a running context: an environment, a sequence, a piece of state. None of that is written down at the moment it exists, because the worker is busy solving the problem, and by the time anyone wants it the context has been discarded.

Why a better model does not fix it

The failure was already found. What is missing is a durable artefact created at the moment of the observation. A better worker finds it faster and loses it just as completely.

What it costs

The investigation is repeated from the top, usually more than once, and each repetition is paid at the cost of the original. Intermittent failures are the worst case: the second investigation may not reproduce at all, so the bug is filed as unreproducible and returns in production.

What Majordomus does

Checkpointing is a first-class, cheap act with a hard length cap: majordomus checkpoint records what was true a moment ago, short enough that the next context can quote it whole, refusing an over-long body rather than truncating it. The profile sets an interval, and check and watch report checkpoint age, so a long-running investigation that has recorded nothing is visible. A session that ends writes a handover with objective, current state and next action, each required, and the next one resolves it with a divergence label computed from git.

Before and after

before   3am: "reproduced it — it's the retry loop under load"   (session, gone)

after    $ majordomus history --task t-…a4f1
         checkpoint  head=8c31f0e  "reproduced under 50 concurrent enqueues;
                                    the retry loop re-enters before the ack"
         handover    advanced      # Next Action: add the load case to test/queue

What it does not do

It does not capture the environment, and it does not record a session's output. The worker writes the checkpoint; the tool guarantees it is short, attached to a head, and readable by whatever comes next.

What this looks like

Concrete situations, one per audience. Each is declared in the moment's front matter, so the before and the after are data rather than prose a page could drift from.

  1. The overnight flake ai-native-team

    before
    A worker hits an intermittent failure at 3am, describes it, and the session ends; the morning has a paragraph and no reproduction.
    after
    A checkpoint records what was true at that moment, capped so the next context can quote it whole, with the head it applied to.
  2. A probe that showed something research-team

    before
    An experiment produces an interesting negative result; the branch is discarded and so is the observation.
    after
    The observation is a knowledge record with class `observed` and the evidence it was read off.
  3. Two days later solo-builder

    before
    Work resumes after a gap and the state of the investigation has to be rebuilt from the diff.
    after
    The handover names the objective, the current state and the next action, and says how far git has moved since.

How you would know

The observable symptoms this moment declares. They are the questionnaire on the index and the input of majordomus why diagnose; nothing else defines them.

  • A failure was reproduced once and could not be reproduced afterwards. cannot-reproduce-again
  • The evidence for a bug exists only in a session window. evidence-in-the-window
  • Debugging restarts from somebody''s description rather than from a recorded observation. starts-from-a-report

Where this lives in the tool

Everything below is read out of this moment's own front matter and resolved against the repository. A name here that did not exist would fail validation.

the commands that answer it

the capabilities of the executable that answer it

what it supervises — derived from the claims below

the claims that back this page, and the evidence behind each

the rules that govern it

  • majordomus.checkpoint-freshness
  • majordomus.handover-integrity
  • majordomus.handovers-carry-state
  • majordomus.ledger-integrity

the use cases that show the way out

If this one is familiar, so is the next

What this moment names, what names it, and what shares its area, audience or tags. The second and third are derived; only the first is written down.

All 38, and how they connect to the tool →