Mesh cooperation is authenticated links between runtimes and one replicated journal folded the same way everywhere

67. Mesh cooperation is authenticated links between runtimes and one replicated journal folded the same way everywhere

Context

ADR 0050 gave the mesh discovery: signed advertisements, one registry, trust as a label. It deliberately stopped there — "whoever later builds remote operations must bring their own authorization decision" — and so did the code. Measured on 2026-09-15, the mesh could not support the product it was presented as:

1. Nothing cooperative crossed a machine. Claims lived on one server's peer board (ADR 0044 gathers the boards of one machine's checkouts through their lease files); handovers were files under one checkout's .ai/local/; no event, session or review travelled anywhere.
2. The same repository on two machines was two repositories. Advertisements carried sha256(git common directory path), and a macOS checkout path never matched the Linux checkout path of the same clone.
3. Two worktrees of one machine could not see each other. Both servers hold the same node key; the manager skipped the second one's datagrams as "self-heard", and its multicast socket could not bind the port the first held.
4. Presence was a sighting, not a relationship. A node was "present" for sixty seconds after an advertisement, whether or not anything could be exchanged with it.

Decision

Cooperation is a layer above discovery inside apps/majordomus-cli/src/mesh/, with these load-bearing rules.

1. Identities are typed and distinct. A node is a machine's key (ADR 0050). A runtime is one checkout's server on that node: runtime = digest(checkout id), so two worktrees are two runtimes and a restarted server is the same runtime. An instance is one process run. A stream is node-runtime-instance: the unit that writes events. A session is named within its stream. The discovery protocol goes to version 2 to carry the runtime; a version-2 reader reads version 1, a version-1 reader refuses version 2 as version.
2. The repository identity is content, not a path. digest(root commits), or a declared cooperation.repository (a shallow clone must declare one; a fork that should not cooperate with its origin may). Runtimes of different repository identities never link.
3. A link is admitted by a handshake, never by discovery. Hello → welcome, each an Ed25519-signed message under its own domain, checking size, shape, signature, freshness, nonce replay, protocol range (link protocol 1..1), self, repository identity and trust — in both directions — with a typed, counted, listed refusal for each. Candidates come from trusted registry records of the same repository and from declared seeds.
4. Transport is an adapter. LinkTransport::post(endpoint, path, message); the shared server answers HTTP through the crate's existing minimal client (ADR 0032: no HTTP client dependency, no TLS, no async). Unit tests put runtimes behind an in-process transport running the same protocol; integration tests and the lab use sockets.
5. Everything cooperative is an event of one journal. Session opened/closed, claim acquired/released, handover published/consumed, review requested/answered — each signed by its origin, identified by (stream, seq), stamped with a Lamport clock, persisted to .ai/local/state/mesh/journal.jsonl. A relay forwards bytes and cannot forge. An event of an unknown kind is stored and relayed but not interpreted, so a newer peer never breaks an older one's replication.
6. Replication is marks, not flooding. A sync round (every heartbeat, signed, under the link id with a rising counter) carries the sender's high-water mark per stream and the events the receiver lacks; the answer carries the same the other way. Delivery is at-least-once and application idempotent; within a stream events apply in sequence order (an early arrival waits, bounded, for its gap); across streams the fold orders by (lamport, stream, seq). The first sync after a handshake is a full state synchronisation by the same mechanism: a newcomer receives what its marks lack, which after compaction is the live state, not an infinite history.
7. Liveness is a beat, judged on the local clock. Each runtime raises its own beat every heartbeat; marks relay beats with how long ago the sender saw them rise. A stream whose beat has not risen within the expiry is expired on that runtime: its sessions are expired and its claims stop excluding. No two machines' wall clocks are compared. Links have their own standing (connecting, connected, degraded, unreachable, expired) with exponential backoff capped at the expiry, and a restarted peer re-handshakes into the same table row.
8. State is one pure fold. mesh::state::fold(events, liveness) yields sessions, claims, overlaps, handovers, reviews and a digest. A claim holds while unreleased, its session open and its stream alive. Two live exclusive claims of different sessions whose paths meet (peers::claims_meet, the board's own predicate) are a conflict won by the lowest (lamport, stream, seq); admission refuses a claim that would conflict with what this runtime already knows. Under a partition both sides may claim; after healing every runtime names the same winner. Never "last packet wins".
9. The board is projected, not replaced. Every heartbeat, attached board sessions become mesh sessions and their announcements advisory claims; ADR 0044's local gathering stays the machine-local view.
10. Every surface is a capability. mesh.cooperation, mesh.peers, mesh.peer, mesh.state, mesh.events, mesh.verify and the commands (sessions, claims, handovers, reviews, the two link routes) are declarations of the mesh module; the CLI, HTTP, OpenAPI, MCP and the Cockpit are their projections. scripts/ci/mesh-check refuses a link handler, a transport or a journal outside the subsystem and a Cockpit page that reads the runtime directly.

Alternatives rejected

• Gossiping the peer board. The board is positional (p1 is reused), in memory and unsigned; replicating it would replicate its ambiguities. The journal is signed and typed; the board is projected into it.
• A consensus protocol (Raft) for claims. It would buy strong consistency with a majority requirement a two-laptop fleet cannot meet, and an async runtime ADR 0032 refuses. Deterministic conflict resolution with named conflicts fits sessions that must keep working offline.
• A central coordinator. A single point of failure and a canonical store elsewhere — refused for the same reasons ADR 0050 refused a mandatory rendezvous.
• Wall-clock lease expiry. Machines' clocks disagree; relayed beats aged on the local monotonic clock do not need them to agree.
• mTLS links. A TLS stack is the dependency ADR 0032 keeps out; signed messages give authenticity, and confidentiality is the private network's or the overlay's.

Consequences

• The mesh now carries metadata about work between trusted runtimes of one repository. The SECURITY.md posture holds by default — nothing opens without an enabled declaration, and a link needs a trusted key — but an enabled mesh with trusted peers sends session intents, claim scopes, review subjects and published handover bodies to those peers, unencrypted unless the network encrypts. docs/MESH.md states it.
• ADR 0050's "a trusted node gains nothing" is refined, not reversed: a linked peer's events can refuse a local exclusive claim and can offer a handover a local command may write into this checkout. Nothing a peer sends executes anything.
• Discovery protocol 2 is not readable by an executable older than this change: a mixed fleet upgrades together, and the refusal is counted as version rather than silent.
• A per-checkout journal file exists under .ai/local/state/mesh/. It is bounded by compaction of long-expired streams (handovers are kept) and never committed.
• The acceptance of these guarantees is automated at three levels: unit and property tests, multi-process integration tests over TCP (tests/mesh_cooperation.rs), and a container lab on a real bridge that partitions and kills nodes (test/mesh-lab/run, CI job mesh-lab).

Addendum, 2026-09-29: compaction keeps what is live

The consequence above said the journal is bounded by compaction of long-expired streams and that handovers are kept. Three defects recorded on PR #588 showed what that left out, and the answers are recorded here rather than in a new ADR because they carry the same decision through: a compaction must not change what the fold says about anything live.

• A handover keeps its stream for a bounded time. A dead stream holding a handover nobody has taken stays for at most a day past the retention (HANDOVER_RETENTION); a handover somebody has taken keeps nothing. Kept for ever, a machine whose runs each handed over something filled its 64-stream quota with the dead, and every later run was refused.
• A taking or an answer stays with what it refers to. A dead stream holding the taking of a handover or the answer to a review stays while the handover's or the review's own stream does, so a handover still offered never looks untaken and an answered review never opens again. Who took and who answered is read by the fold itself.
• A compacted stream that beats again is heard again, whole. A fresh beat, verified and above the one its tombstone holds, lifts the tombstone, and the stream is fetched again from its first event rather than from the tombstone's mark: what it said before the compaction — a claim still held — is live again with it, and a suffix would admit a conflicting claim.

docs/MESH.md ("Compaction") states the semantics, including why two runtimes can print different digests while each compacts on its own schedule; the journal's unit and property tests, tests/mesh_compaction.rs and test/cases/493_mesh_compaction_keeps_what_is_live.sh hold them.

Addendum, 2026-10-15: a node key is rotated by a succession its predecessor signs (I2705)

I2134 made a key withdrawn from trust.allow lose its links without a restart (Cooperation::retrust, called from the live reload). It left two things undone: a key cannot be replaced, only cut off (threat T11), and a withdrawn key's events stay in the running journal until the next restart's reload applies the trust in force (I2124). Both follow from this record's rules and are decided here rather than in a new ADR. Two facts shape the answer: the node key is machine-wide (node.json under $XDG_STATE_HOME/majordomus or ~/.local/state/majordomus, shared by every worktree and repository on the machine), and the node id is the key's digest, so a new key is a new node.

A rotation is a succession statement. {schema: majordomus-mesh-succession/v1, old key, new key, at}, signed under its own domain by the old key and countersigned by the new one: the old signature says the holder of the trusted key hands over, the new one that the successor's holder exists and agreed. The statement names no repository, because the key it succeeds is the machine's. Rotating writes the new node.json private from its first byte, keeps the statement beside it (public halves only) and destroys the old secret.

It travels as the closing event of every old-key stream. Each runtime of the machine that is running holds the old key in memory; it writes key_succeeded carrying the statement as the last event of its stream, signed by the old key, and continues under the new key on a new stream, exactly as after a restart: sessions and claims are re-taken under the new stream and the old stream's expire with its beat. Nothing new is decided about claims. The event replicates like any other, so every runtime of every repository the machine cooperates in learns the succession, including runtimes it never linked to directly.

A peer follows a succession, and only through the trust in force. Each runtime folds the successions it holds into its trust. A successor is trusted while its predecessor is trusted by the declaration in force, and only then. Withdrawing a stolen key from the trunk's trust.allow therefore also withdraws any successor its thief named, and a rotation cannot be used to escape a revocation. The depth is one: a key trusted only through a succession cannot name a successor until the trunk lists it. Two different successors of one key are a fork. A fork trusts neither and leaves the old key rejected, because only a stolen key can produce one, and mesh doctor names it.

The succeeded key is retired, not withdrawn. From the moment a runtime folds the succession, the old key's hello is refused untrusted with a detail naming the successor. The refusal code stays untrusted because an older peer would read a new code as malformed. An old-key stream accepts nothing after its closing event, and an old-key stream the runtime had not seen before is refused. Streams that are still open without a closing event are accepted for one expiry and refused after it. The retired key's history is legitimate and is kept.

A withdrawn key's events leave the running journal at the retrust that withdrew it. A key that is no longer trusted and has no trusted successor is treated as compromised. At that retrust the running journal drops its events with the predicate the reload applies (Journal::open_trusting), counts them by key in the cooperation reading, and tombstones its streams so that peers do not send them again. Only a beat from a trusted key lifts such a tombstone. The key's succession statements stay, inert, as the audit of what it said. The journal file is rewritten at the next compaction as it is today, and the next reload already refuses those lines.

The trust root stays the trunk's trust.allow. A succession bridges the time until a person commits the replacement. mesh rotate prints the change to commit on the default branch (the new key in, the old key out), and mesh doctor warns while a key is trusted only through a succession. Once the commit lands, the new key is trusted directly and the old one is retired, not purged, because it has a trusted successor. A working tree still cannot widen the trunk's trust: the successor's trust comes from the statement and the trunk's listing of the predecessor, and the working tree's copy of the declaration plays no part.

Rejected: trusting a successor on the statement alone (a stolen key escapes withdrawal); rotating only by committing a new trust.allow (every peer refuses the machine until the commit lands everywhere, which is the outage a rotation exists to avoid); a new refusal code (see above); and purging a retired key's history (an orderly rotation would drop unconsumed handovers and the audit trail). The advisors the reasoning plan selected were both rate-limited when this was decided. The decision rests on the local evidence recorded with the task.

The work is five slices of at most an hour each, one pull request each, in this order:

1. The statement and the trust it confers (mesh/identity.rs, mesh/trust.rs; no shared file, so it can start now, beside the open #1179, #1188, #1197, #1207 and #1208). NodeIdentity::rotate returns the signed Succession and writes the new key with write_private. trust::evaluate takes the successions and answers Trusted("succession"), Retired(successor) or Rejected("succession fork"). Tests: mesh::identity a_rotation_is_signed_by_the_old_key_and_countersigned_by_the_new, and mesh::trust a_successor_is_trusted_only_while_its_predecessor_is and a_fork_trusts_neither.
2. The running journal drops a withdrawn key (mesh/journal.rs Journal::retain_trusted, the tombstone that only a trusted beat lifts; mesh/cooperation.rs retrust calls it and reports purged). This slice goes after #1179 and #1207 land, because both touch journal.rs. Test: mesh::cooperation a_withdrawn_keys_events_leave_the_running_journal_without_a_restart, plus its tests/mesh_cooperation.rs twin over TCP. It closes I2705's second acceptance criterion.
3. The succession is followed (mesh/journal.rs EventBody::KeySucceeded, listed in .ai/repo/ci/mesh-event-fields.yaml; mesh/cooperation.rs folds successions into the trust, retrusts on ingest, refuses the retired key's hello and new streams, closes an old stream at its closing event, and exempts a retired key from slice 2's purge). This slice goes after #1188 (the event field list), #1197 (the hello check order) and slices 1 and 2. Test: mesh::cooperation a_peer_that_folds_a_succession_admits_the_new_key_and_refuses_the_old.
4. A machine rotates (capability/builtin/mesh.rs declares mesh.rotate and is followed by majordomus generate; Cooperation::rotate writes the closing event and moves to the new stream; live.rs switches every other server of the machine when node.json changes). This slice goes after slice 3 and #1208 (manager.rs). Test: tests/mesh_cooperation.rs a_rotated_key_is_followed_and_the_old_key_is_refused, two processes over TCP. Its result is I2705's key_rotation_test evidence.
5. A peer that missed the event, the doctor and the words (mesh/link.rs and mesh/cooperation.rs: the new key's hello may carry the statement; mesh/doctor.rs warns about trust held only through a succession and about a fork; docs/MESH.md gains "Rotating a node key"). This slice goes last. Tests: mesh::cooperation a_hello_carrying_a_succession_is_admitted_by_a_peer_that_missed_the_event, and a mesh::doctor unit test for the warning.