Skip to content

recovery-from-injected-faults-is-proven — A killed worker, a cut connection and a conflict are injected on purpose, and recovery is proven without lost work or duplicate completion

A conservation reading counts completions per issue, unreachable commits, stranded claims and multiply-consumed handovers; the T5 protocol registers three fault kinds; mesh lab scenarios and live single-host trials inject them; the T5 row publishes every trial.

BLOCKED 0 of 5 issues done

ready
0
blocked
5
active
0
verify
0
done
0
cancelled
0

Problem

T5 of the owner's report of 2026-10-19 asks to kill a worker, cut a connection and cause a conflict, then prove recovery without loss or duplicate completion of work. The mesh lab proves the mesh layer recovers (test/mesh-lab/run scenarios claim_conflict, network_partition, process_crash_recovery), but no scenario holds a task, and nothing counts completions per issue or checks that a task's commits survived. Recovery of the record stores is covered by case 135, but not as a work-level trial.

Outcome

A conservation reading counts completions per issue, unreachable commits, stranded claims and multiply-consumed handovers; the T5 protocol registers three fault kinds; mesh lab scenarios and live single-host trials inject them; the T5 row publishes every trial.

Current state

test/mesh-lab/run:187-245 runs scenarios on claims, handovers, partition and crash, each judged by scripts/ci/mesh-lab-evidence and case 414. apps/majordomus-cli/src/mesh/state.rs:274 and :313 say consumed_by keeps two workers from both resuming a handover; mesh/handover.rs:15 says materializing the same handover twice writes one file. share/events.yaml:152-158 declares session.recovered. I3198 (lease-based assignment that returns when its lease is lost) is BLOCKED at p3 and is not on master.

Desired state

economics tasks carries a conservation section; T5 trials are committed for each fault kind; the mesh lab's evidence includes the work-level scenarios.

Scope

  • apps/majordomus-cli/src/economics
  • apps/majordomus-cli/src/economics/mod.rs
  • apps/majordomus-cli/tests/economics.rs
  • test/mesh-lab/run
  • test/mesh-lab/node-entrypoint
  • scripts/ci/mesh-lab-evidence
  • test/cases/414_mesh_lab_evidence.sh
  • docs/CLAIMS.yaml
  • docs/MESH.md
  • .ai/repo/benchmarks
  • docs/generated/economics.json
  • docs/generated/economics.md
  • docs/generated/economics.yaml
  • site/data/generated/economics.json

Out of scope

  • Lease-based assignment and its return (I3198, I3199)
  • Supervision repairs (I3183)
  • Running faults in this repository's own checkouts; trials run in throwaway clones
  • Killing any process the trial did not start

Acceptance criteria

  • The conservation reading names every violation with its records, and a clean run reads zero violations
  • Each registered fault kind has its registered number of trials committed, each with its conservation reading
  • The mesh lab's required scenario set includes the work-level scenarios, and case 414 holds it

Validation

  • majordomus plan validate

Risks

  • The mesh lab needs docker; it runs in its CI job, and a local run is optional
  • A killed session's uncommitted edits are lost by construction; the trial counts committed work, and the protocol says so
  • Faults are injected by exact pid, never by name (rule project.reclaim-only-what-you-own)

Issues

Each one is an execution contract with its own acceptance criteria, validation command and required evidence. The wave column is the layer of the dependency graph it sits in.

issuestatuswavedepends ontitle
I3390 BLOCKED 1 I3376 One conservation reading counts completions per issue, unreachable task commits, stranded claims and multiply-consumed handovers, each violation with its records
I3391 BLOCKED 2 I3352, I3353, I3390 The T5 protocol registers three fault kinds, the trials each needs and what a trial must show, before any fault is injected
I3392 BLOCKED 3 I3390, I3391 The mesh lab injects the three faults into work in progress, and its evidence holds the conservation reading of each scenario
I3393 BLOCKED 3 I3390, I3391 Each fault is injected into real coding-agent sessions on one host, and each trial's recovery and conservation reading are committed
I3394 BLOCKED 4 I3392, I3393, I3354 The mesh lab's work-fault evidence from CI is committed as T5 trials and the T5 verdict is published with every failed trial

The graph

recovery-from-injected-faults-is-proven dependency graphfrom .ai/repo/project
flowchart LR
    I3390["I3390<br/>One conservation reading counts completions per issue, unreachable task commits, stranded claims and multiply-consumed handovers, each violation with its records"]:::blocked
    I3391["I3391<br/>The T5 protocol registers three fault kinds, the trials each needs and what a trial must show, before any fault is injected"]:::blocked
    I3392["I3392<br/>The mesh lab injects the three faults into work in progress, and its evidence holds the conservation reading of each scenario"]:::blocked
    I3393["I3393<br/>Each fault is injected into real coding-agent sessions on one host, and each trial's recovery and conservation reading are committed"]:::blocked
    I3394["I3394<br/>The mesh lab's work-fault evidence from CI is committed as T5 trials and the T5 verdict is published with every failed trial"]:::blocked
    I3390 --> I3391
    I3390 --> I3392
    I3390 --> I3393
    I3391 --> I3392
    I3391 --> I3393
    I3392 --> I3394
    I3393 --> I3394
    classDef done stroke:#16a34a,fill:#dcfce7,color:#14532d,stroke-width:2px
    classDef active stroke:#2563eb,fill:#eff6ff,color:#1e3a8a,stroke-width:2px
    classDef verify stroke:#7c3aed,fill:#f5f3ff,color:#4c1d95,stroke-width:2px
    classDef ready stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
    classDef planned stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
    classDef blocked stroke:#b45309,fill:#fffbeb,color:#78350f,stroke-width:2px
    classDef cancelled stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2px
    classDef superseded stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2px

Canonical record: .ai/repo/project/milestones/recovery-from-injected-faults-is-proven.yaml. Read it back with majordomus plan show recovery-from-injected-faults-is-proven.

Searches the titles and descriptions of every page this site publishes.