Skip to content

Catch an agent that says it is done when it is not

One task, start to finish: a worker strays outside its scope, claims it is finished, hands over, and is accepted only once the repository's own verification passes.

guaranteed Completion and quality gatesbasicagentoperatorwrites state

Situation

A coding agent is asked for one change. It makes the change, edits a file nobody asked it to touch, and reports that it is done. Another agent is already working in the same paths. The session ends before anyone reads the diff, and the next session starts from whatever the chat remembered. Every one of those is ordinary, and none of them is a model problem.

What you run

  • start: claims the paths the task may touch, and reports another worker already claiming them
  • check: refuses a touched file outside the claimed paths
  • finish: evaluates the finish contract line by line and writes nothing while a line fails
  • handover: writes the state the next session starts from, into the repository
  • context: gives the next worker that state back, without the conversation that produced it

Scenario

setup: challenge
given:
  - 'a repository with Majordomus installed, and a second worktree whose worker already claims lib'
  - 'a test, test/parse.sh, that passes only when the parser message names the column'
steps:
  - id: claim-the-work
    note: 'The task claims lib. Another worker already claims lib in a second worktree, and start says so before any file is touched.'
    run: ['start', 'refuse tabs in the parser', '--scope', 'lib']
    expect:
      exit: 0
      stdout_contains: ['^started t-', 'INFO overlap', 'claims lib']
  - id: the-worker-strays
    note: 'The worker makes the change it was asked for, and also edits a note under docs, which the task never claimed.'
    worker: 'echo "tabs are refused" >> lib/a && echo "parser notes" >> docs/d'
    expect:
      exit: 0
  - id: check-refuses-the-stray-file
    note: 'check names the file outside the claimed scope and exits non-zero. It is a failure, not a warning.'
    run: ['check']
    expect:
      exit: 10
      stdout_contains: ['FAIL scope', 'docs/d']
  - id: the-worker-says-it-is-done
    note: 'The worker reports the task finished. finish evaluates the contract instead of believing it, and refuses.'
    run: ['finish', '--outcome', 'completed', '--verify-command', 'sh test/parse.sh']
    expect:
      exit: 10
      stdout_contains: ['FAIL scope', 'FAIL verification', 'refused']
  - id: the-worker-reverts-the-stray-file
    note: 'The stray edit is taken back.'
    worker: 'git checkout -- docs/d'
    expect:
      exit: 0
  - id: hand-over
    note: 'The session ends. What the next one needs is written into the repository, not left in a transcript.'
    run: ['handover']
    stdin: challenge-handover.md
    expect:
      exit: 0
      files_exist: ['.ai/local/state/handovers']
  - id: the-next-session-resumes
    note: 'A new session asks the repository what it needs to know, and gets the task, its scope and the handover back.'
    run: ['context']
    expect:
      exit: 0
      stdout_contains: ['## TASK', 'LATEST COMPATIBLE HANDOVER', 'name the column']
  - id: finish-refuses-a-failing-verification
    note: 'Scope is clean now, but the repository''s own test still fails. finish runs it and refuses again.'
    run: ['finish', '--outcome', 'completed', '--verify-command', 'sh test/parse.sh']
    expect:
      exit: 10
      stdout_contains: ['OK +scope', 'FAIL verification', 'refused']
      stdout_not_contains: ['FAIL scope']
  - id: the-worker-finishes-the-work
    note: 'The worker does what the handover said was left.'
    worker: 'echo "the message names the column" >> lib/a'
    expect:
      exit: 0
  - id: finish-accepts
    note: 'Every line of the contract passes, so the outcome is recorded, with the verification that proved it.'
    run: ['finish', '--outcome', 'completed', '--verify-command', 'sh test/parse.sh']
    expect:
      exit: 0
      stdout_contains: ['OK +scope', 'OK +verification', 'completed']
      stdout_not_contains: ['FAIL']
then:
  - 'a file outside the claimed scope is refused by check and by finish'
  - 'a worker saying it is done is a claim finish evaluates, and a failing verification keeps it refused'
  - 'the next session resumes from what the repository recorded, not from the conversation'
  - 'the work is accepted only when every line of the contract passes'

Outcome

The work is accepted exactly once, when the repository's own test passes and nothing outside the claimed paths changed. The two earlier claims of being done were refused with the reason named and the command that reproduces it. The next session started from the handover the repository held, and the other worker in the same paths was named before anything was edited.

The scenario

Executed against the real tool, in a repository prepared by challenge, every time this site is generated. What follows is not a description of what should happen; it is what happened.

given

  • a repository with Majordomus installed, and a second worktree whose worker already claims lib
  • a test, test/parse.sh, that passes only when the parser message names the column

when

  1. 1majordomus start refuse tabs in the parser --scope lib
  2. 2worker: echo "tabs are refused" >> lib/a && echo "parser notes" >> docs/d
  3. 3majordomus check
  4. 4majordomus finish --outcome completed --verify-command sh test/parse.sh
  5. 5worker: git checkout -- docs/d
  6. 6majordomus handover
  7. 7majordomus context
  8. 8majordomus finish --outcome completed --verify-command sh test/parse.sh
  9. 9worker: echo "the message names the column" >> lib/a
  10. 10majordomus finish --outcome completed --verify-command sh test/parse.sh

then

  • a file outside the claimed scope is refused by check and by finish
  • a worker saying it is done is a claim finish evaluates, and a failing verification keeps it refused
  • the next session resumes from what the repository recorded, not from the conversation
  • the work is accepted only when every line of the contract passes

What the tool did

Evidence of the run this page was generated from: pass, 10 step(s). Output is normalised: the repository's path, timestamps, ids and durations are placeholders; nothing else is edited.

Tested by test/cases/94_use_cases.sh, which runs every scenario and refuses a step whose recorded output is not what it produced.

  1. The task claims lib. Another worker already claims lib in a second worktree, and start says so before any file is touched.

    started t-<id>  profile=implementation  scope=lib
    INFO overlap     repo-other — claims lib — contains your lib  [reproduce: majordomus check --overlap]
    next: majordomus context; checkpoint every 15m; majordomus check before claiming anything
    • exit 0
    • stdout_contains /^started t-/
    • stdout_contains /INFO overlap/
    • stdout_contains /claims lib/
  2. 2worker$ echo "tabs are refused" >> lib/a && echo "parser notes" >> docs/d exit 0

    The worker makes the change it was asked for, and also edits a note under docs, which the task never claimed.

    • exit 0
  3. check names the file outside the claimed scope and exits non-zero. It is a failure, not a warning.

    OK   knowledge   records — 0 record(s) valid against knowledge/v1; every reference resolves  [reproduce: majordomus knowledge check]
    ·    checkpoint  t-<id> — <timed against the checkpoint interval>
    OK   records     ledger.jsonl — every line is a well-formed event
    OK   blockers    t-<id> — none open
    OK   records     decisions.md — every entry is attributable
    FAIL scope       docs/d — outside claimed scope (lib)  [reproduce: git status --porcelain; git log --first-parent --no-merges --name-only --format= <sha>..HEAD]
    OK   state       t-<id> — exact (head <head>)
    INFO mesh        t-<id> — the repository declares no enabled mesh
    INFO use-case    30 advisory gap(s) — not covered by an active use case, and the policy does not require it: command:init command:update command:doctor command:watch command:start command:context command:checkpoint command:handover command:check command:finish command:history command:decision command:question command:prompt command:evidence command:skills command:capture command:search command:doctrine command:plan command:version command:session command:knowledge command:adr command:usecase command:rules command:migrate command:recover command:bench command:archive  [reproduce: majordomus usecase coverage]
    OK   use-case    0 of 30 target(s) covered — every public command the policy requires is named and run by an active use case (policy use_cases.coverage)
    INFO obligation  t-<id> — the task declares no obligations
    INFO gate        t-<id> — this repository declares no CI model at .ai/repo/ci/gates.yaml; nothing can be planned
    check: 12 finding(s), 1 failing
    • exit 10
    • stdout_contains /FAIL scope/
    • stdout_contains /docs/d/
  4. The worker reports the task finished. finish evaluates the contract instead of believing it, and refuses.

    OK   blockers    t-<id> — none open
    FAIL note        t-<id> — no --note file and no handover for this task  [reproduce: majordomus handover < note.md]
    INFO regression  t-<id> — profile implementation adds no requirement beyond the shared contract
    FAIL scope       docs/d — outside claimed scope (lib)  [reproduce: git status --porcelain; git log --first-parent --no-merges --name-only --format= <sha>..HEAD]
    OK   state       t-<id> — exact (head <head>)
    INFO continuity  t-<id> — applies to partial and blocked
    INFO mesh        t-<id> — the repository declares no enabled mesh
    INFO use-case    30 advisory gap(s) — not covered by an active use case, and the policy does not require it: command:init command:update command:doctor command:watch command:start command:context command:checkpoint command:handover command:check command:finish command:history command:decision command:question command:prompt command:evidence command:skills command:capture command:search command:doctrine command:plan command:version command:session command:knowledge command:adr command:usecase command:rules command:migrate command:recover command:bench command:archive  [reproduce: majordomus usecase coverage]
    OK   use-case    0 of 30 target(s) covered — every public command the policy requires is named and run by an active use case (policy use_cases.coverage)
    FAIL verification t-<id> — sh test/parse.sh — exit 1, <s>s  [reproduce: sh test/parse.sh]
    INFO obligation  t-<id> — the task declares no obligations
    INFO gate        t-<id> — this repository declares no CI model at .ai/repo/ci/gates.yaml; nothing can be planned
    INFO gate        t-<id> — not in verification.finish_requires
    finish: refused, 3 unmet
    blocking doctrines:
    - majordomus.note-integrity
    - majordomus.scope-integrity
    - majordomus.verification-integrity
    • exit 10
    • stdout_contains /FAIL scope/
    • stdout_contains /FAIL verification/
    • stdout_contains /refused/
  5. 5worker$ git checkout -- docs/d exit 0

    The stray edit is taken back.

    • exit 0
  6. 6$ majordomus handover exit 0< challenge-handover.md

    The session ends. What the next one needs is written into the repository, not left in a transcript.

    .ai/local/state/handovers/<time>--main--<head>--<hash>.md
    • exit 0
    • file_exists .ai/local/state/handovers
  7. A new session asks the repository what it needs to know, and gets the task, its scope and the handover back.

    # Majordomus context — <time>
    # a projection of durable state, not a source of truth: validate every line against git
    
    ## GIT
    repository   <repo>
    branch       main
    head         <sha>
    working_tree dirty
    task_record  exact (recorded head <head>)
    
    ## TASK
    id           t-<id>
    task         refuse tabs in the parser
    profile      implementation
    owner        <owner>
    outcome      active
    scope        lib
    started      <time> (<age> ago)
    checkpoint   <time> (<age> ago, interval 15m)
    
    ## PROFILE implementation
    capability   standard
    effort       medium
    verbosity    concise
    presentation engineering
    verification verify command
    output       outcome, changes, verification, blockers, next_action
    
    ## CONTEXT DOCUMENTS (for the scope, root first; majordomus context explain <path>)
    lib  ai.layer
    
    ## KNOWLEDGE (candidates awaiting review on this branch; majordomus knowledge candidates)
    No knowledge candidates await review on this branch. That is an answer, not a gap: `majordomus knowledge candidates` lists every branch.
    
    ## DECISIONS (5 most recent for this task)
    (none)
    
    ## LATEST COMPATIBLE HANDOVER (.ai/local/state/handovers/<time>--main--<head>--<hash>.md)
    match        same_worktree_same_branch
    git state    exact
    created      <time> (<age> ago)
    task         t-<id>
    # Objective
    refuse tabs in the parser
    # Current State
    tabs are refused; the stray note under docs is reverted; the parser's test still fails
    # Next Action
    make the message name the column, then run the parser's test again
    
    ## FILES TOUCHED IN SCOPE (1)
    lib/a
    
    ## EXCLUDED
    - history — profile implementation sets context.recent_history_depth: 0
    
    ## BUDGET
    57 of 300 lines
    • exit 0
    • stdout_contains /## TASK/
    • stdout_contains /LATEST COMPATIBLE HANDOVER/
    • stdout_contains /name the column/
  8. Scope is clean now, but the repository's own test still fails. finish runs it and refuses again.

    OK   blockers    t-<id> — none open
    OK   note        t-<id> — <time>--main--<head>--<hash>.md
    INFO regression  t-<id> — profile implementation adds no requirement beyond the shared contract
    OK   scope       t-<id> — 1 touched file(s), all within scope
    OK   state       t-<id> — exact (head <head>)
    INFO continuity  t-<id> — applies to partial and blocked
    INFO mesh        t-<id> — the repository declares no enabled mesh
    INFO use-case    30 advisory gap(s) — not covered by an active use case, and the policy does not require it: command:init command:update command:doctor command:watch command:start command:context command:checkpoint command:handover command:check command:finish command:history command:decision command:question command:prompt command:evidence command:skills command:capture command:search command:doctrine command:plan command:version command:session command:knowledge command:adr command:usecase command:rules command:migrate command:recover command:bench command:archive  [reproduce: majordomus usecase coverage]
    OK   use-case    0 of 30 target(s) covered — every public command the policy requires is named and run by an active use case (policy use_cases.coverage)
    FAIL verification t-<id> — sh test/parse.sh — exit 1, <s>s  [reproduce: sh test/parse.sh]
    INFO obligation  t-<id> — the task declares no obligations
    INFO gate        t-<id> — this repository declares no CI model at .ai/repo/ci/gates.yaml; nothing can be planned
    INFO gate        t-<id> — not in verification.finish_requires
    finish: refused, 1 unmet
    blocking doctrines:
    - majordomus.verification-integrity
    • exit 10
    • stdout_contains /OK +scope/
    • stdout_contains /FAIL verification/
    • stdout_contains /refused/
    • stdout_not_contains /FAIL scope/
  9. 9worker$ echo "the message names the column" >> lib/a exit 0

    The worker does what the handover said was left.

    • exit 0
  10. Every line of the contract passes, so the outcome is recorded, with the verification that proved it.

    OK   blockers    t-<id> — none open
    OK   note        t-<id> — <time>--main--<head>--<hash>.md
    INFO regression  t-<id> — profile implementation adds no requirement beyond the shared contract
    OK   scope       t-<id> — 1 touched file(s), all within scope
    OK   state       t-<id> — exact (head <head>)
    INFO continuity  t-<id> — applies to partial and blocked
    INFO mesh        t-<id> — the repository declares no enabled mesh
    INFO use-case    30 advisory gap(s) — not covered by an active use case, and the policy does not require it: command:init command:update command:doctor command:watch command:start command:context command:checkpoint command:handover command:check command:finish command:history command:decision command:question command:prompt command:evidence command:skills command:capture command:search command:doctrine command:plan command:version command:session command:knowledge command:adr command:usecase command:rules command:migrate command:recover command:bench command:archive  [reproduce: majordomus usecase coverage]
    OK   use-case    0 of 30 target(s) covered — every public command the policy requires is named and run by an active use case (policy use_cases.coverage)
    OK   verification t-<id> — sh test/parse.sh — exit 0, <s>s, tree <tree>
    INFO obligation  t-<id> — the task declares no obligations
    INFO gate        t-<id> — this repository declares no CI model at .ai/repo/ci/gates.yaml; nothing can be planned
    INFO gate        t-<id> — not in verification.finish_requires
    finish: t-<id> completed
    • exit 0
    • stdout_contains /OK +scope/
    • stdout_contains /OK +verification/
    • stdout_contains /completed/
    • stdout_not_contains /FAIL/

Why the tool knows this

The chain from this page to the code: the use case names commands, rules and claims; each rule is dispatched from the registry and each claim names the test that proves it; the scenario above executed the commands. Nothing in this chain is prose about intent.

The rules that make it hold

Guarantees exercised

Related use cases

Computed from what they share: claims, rules, commands, category, applications. Nobody maintains this list.

Searches the titles and descriptions of every page this site publishes.