recovery-from-injected-faults-is-proven — A killed worker, a cut connection and a conflict are injected on purpose, and recovery is proven without lost work or duplicate completion
A conservation reading counts completions per issue, unreachable commits, stranded claims and multiply-consumed handovers; the T5 protocol registers three fault kinds; mesh lab scenarios and live single-host trials inject them; the T5 row publishes every trial.
BLOCKED 0 of 5 issues done
- ready
- 0
- blocked
- 5
- active
- 0
- verify
- 0
- done
- 0
- cancelled
- 0
Problem
T5 of the owner's report of 2026-10-19 asks to kill a worker, cut a connection and cause a conflict, then prove recovery without loss or duplicate completion of work. The mesh lab proves the mesh layer recovers (test/mesh-lab/run scenarios claim_conflict, network_partition, process_crash_recovery), but no scenario holds a task, and nothing counts completions per issue or checks that a task's commits survived. Recovery of the record stores is covered by case 135, but not as a work-level trial.
Outcome
A conservation reading counts completions per issue, unreachable commits, stranded claims and multiply-consumed handovers; the T5 protocol registers three fault kinds; mesh lab scenarios and live single-host trials inject them; the T5 row publishes every trial.
Current state
test/mesh-lab/run:187-245 runs scenarios on claims, handovers, partition and crash, each judged by scripts/ci/mesh-lab-evidence and case 414. apps/majordomus-cli/src/mesh/state.rs:274 and :313 say consumed_by keeps two workers from both resuming a handover; mesh/handover.rs:15 says materializing the same handover twice writes one file. share/events.yaml:152-158 declares session.recovered. I3198 (lease-based assignment that returns when its lease is lost) is BLOCKED at p3 and is not on master.
Desired state
economics tasks carries a conservation section; T5 trials are committed for each fault kind; the mesh lab's evidence includes the work-level scenarios.
Scope
- apps/majordomus-cli/src/economics
- apps/majordomus-cli/src/economics/mod.rs
- apps/majordomus-cli/tests/economics.rs
- test/mesh-lab/run
- test/mesh-lab/node-entrypoint
- scripts/ci/mesh-lab-evidence
- test/cases/414_mesh_lab_evidence.sh
- docs/CLAIMS.yaml
- docs/MESH.md
- .ai/repo/benchmarks
- docs/generated/economics.json
- docs/generated/economics.md
- docs/generated/economics.yaml
- site/data/generated/economics.json
Out of scope
- Lease-based assignment and its return (I3198, I3199)
- Supervision repairs (I3183)
- Running faults in this repository's own checkouts; trials run in throwaway clones
- Killing any process the trial did not start
Acceptance criteria
- The conservation reading names every violation with its records, and a clean run reads zero violations
- Each registered fault kind has its registered number of trials committed, each with its conservation reading
- The mesh lab's required scenario set includes the work-level scenarios, and case 414 holds it
Validation
- majordomus plan validate
Risks
- The mesh lab needs docker; it runs in its CI job, and a local run is optional
- A killed session's uncommitted edits are lost by construction; the trial counts committed work, and the protocol says so
- Faults are injected by exact pid, never by name (rule project.reclaim-only-what-you-own)
Issues
Each one is an execution contract with its own acceptance criteria, validation command and required evidence. The wave column is the layer of the dependency graph it sits in.
| issue | status | wave | depends on | title |
|---|---|---|---|---|
| I3390 | BLOCKED | 1 | I3376 | One conservation reading counts completions per issue, unreachable task commits, stranded claims and multiply-consumed handovers, each violation with its records |
| I3391 | BLOCKED | 2 | I3352, I3353, I3390 | The T5 protocol registers three fault kinds, the trials each needs and what a trial must show, before any fault is injected |
| I3392 | BLOCKED | 3 | I3390, I3391 | The mesh lab injects the three faults into work in progress, and its evidence holds the conservation reading of each scenario |
| I3393 | BLOCKED | 3 | I3390, I3391 | Each fault is injected into real coding-agent sessions on one host, and each trial's recovery and conservation reading are committed |
| I3394 | BLOCKED | 4 | I3392, I3393, I3354 | The mesh lab's work-fault evidence from CI is committed as T5 trials and the T5 verdict is published with every failed trial |
The graph
flowchart LR
I3390["I3390<br/>One conservation reading counts completions per issue, unreachable task commits, stranded claims and multiply-consumed handovers, each violation with its records"]:::blocked
I3391["I3391<br/>The T5 protocol registers three fault kinds, the trials each needs and what a trial must show, before any fault is injected"]:::blocked
I3392["I3392<br/>The mesh lab injects the three faults into work in progress, and its evidence holds the conservation reading of each scenario"]:::blocked
I3393["I3393<br/>Each fault is injected into real coding-agent sessions on one host, and each trial's recovery and conservation reading are committed"]:::blocked
I3394["I3394<br/>The mesh lab's work-fault evidence from CI is committed as T5 trials and the T5 verdict is published with every failed trial"]:::blocked
I3390 --> I3391
I3390 --> I3392
I3390 --> I3393
I3391 --> I3392
I3391 --> I3393
I3392 --> I3394
I3393 --> I3394
classDef done stroke:#16a34a,fill:#dcfce7,color:#14532d,stroke-width:2px
classDef active stroke:#2563eb,fill:#eff6ff,color:#1e3a8a,stroke-width:2px
classDef verify stroke:#7c3aed,fill:#f5f3ff,color:#4c1d95,stroke-width:2px
classDef ready stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
classDef planned stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
classDef blocked stroke:#b45309,fill:#fffbeb,color:#78350f,stroke-width:2px
classDef cancelled stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2px
classDef superseded stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2pxLinked from this diagram (5)
- I3390 One conservation reading counts completions per issue, unreachable task commits, stranded claims and multiply-consumed handovers, each violation with its records
- I3391 The T5 protocol registers three fault kinds, the trials each needs and what a trial must show, before any fault is injected
- I3392 The mesh lab injects the three faults into work in progress, and its evidence holds the conservation reading of each scenario
- I3393 Each fault is injected into real coding-agent sessions on one host, and each trial's recovery and conservation reading are committed
- I3394 The mesh lab's work-fault evidence from CI is committed as T5 trials and the T5 verdict is published with every failed trial
Canonical record: .ai/repo/project/milestones/recovery-from-injected-faults-is-proven.yaml. Read it back with majordomus plan show recovery-from-injected-faults-is-proven.