runtime-lifecycle — One usable server per checkout starts, attaches, stops and reports its state without races or hidden kills
Entering a checkout, opening the Cockpit, or starting any number of MCP clients converges on one server for that checkout, which is taken over only when it is dead, stops in order when asked, removes only its own lease, reports a typed state every surface reads, and stays up while any surface uses it.
ACTIVE 7 of 10 issues done
- ready
- 2
- blocked
- 1
- active
- 0
- verify
- 0
- done
- 7
- cancelled
- 0
Problem
The shared server's election is sound at the create step (O_CREAT|O_EXCL at apps/majordomus-cli/src/lease.rs:516-520, proved by apps/majordomus-cli/tests/mcp_shared.rs::a_storm_of_clients_converges_on_one_server_and_one_board and case 191), but what follows it is not. The lease probe is GET / on the same four workers that run tool calls (apps/majordomus-cli/src/lease.rs:896-925, apps/majordomus-cli/src/http/server.rs:23), so four long calls make a healthy server look dead and a new client takes it over. A server that lost its lease keeps its bridged sessions forever, because bridges ping every 20 s (apps/majordomus-cli/src/mcp/bridge.rs:19) and the serve loop counts them as attached (apps/majordomus-cli/src/commands/serve.rs:136-146). `serve stop` is a SIGTERM whose handler unlinks the lease unconditionally and re-raises (apps/majordomus-cli/src/lease.rs:1164-1212), so it can delete a successor's lease and never runs the ordered stop (apps/majordomus-cli/src/shared.rs:282-289). The lease timings the policy declares never reach the mcp and serve elections (apps/majordomus-cli/src/lease.rs:89-191 OnceLock filled before apps/majordomus-cli/src/policy.rs:364 runs). Lifecycle state is an observer's reading of a file (apps/majordomus-cli/src/capability/builtin/server.rs:163-177) with no degraded, stopping or failed, and the live server of 2026-10-09 had logged nothing for eight hours because it inherited MAJORDOMUS_LOG=error (apps/majordomus-cli/src/commands/serve.rs:727-729). A UI-first server stops after 900 s while being viewed, because only MCP sessions count as use (apps/majordomus-cli/src/shared.rs:250-253). Measured in the runtime baseline audit of 2026-10-09 at master 1fb8d6be44 (plan/RUNTIME_BASELINE.md).
Outcome
Entering a checkout, opening the Cockpit, or starting any number of MCP clients converges on one server for that checkout, which is taken over only when it is dead, stops in order when asked, removes only its own lease, reports a typed state every surface reads, and stays up while any surface uses it.
Current state
Implemented and tested: atomic create, a published URL only after the socket serves (apps/majordomus-cli/src/shared.rs:294-305), stale, corrupt, abandoned, busy and superseded leases, port fallback, stdio hygiene. Partial: the probe, takeover and publish are compare-then-act (apps/majordomus-cli/src/lease.rs:778-798, 1082-1107); a lost server's existing sessions are served indefinitely (apps/majordomus-cli/src/http/mcp.rs:113-118); SIGTERM is not graceful (apps/majordomus-cli/tests/mcp_shared.rs::sigterm_removes_the_lease_before_the_process_dies asserts signal 15); declared timings are dead in mcp and serve. Absent: degraded, stopping and failed states, and liveness from non-MCP use. Refuted premises of the pack: there is no `majordomus ui` command, `serve ensure` and `env enter` are the UI-first entries (apps/majordomus-cli/src/commands/serve.rs:481-576, apps/majordomus-cli/src/commands/env.rs:701-760); `serve ensure` and `serve stop` are CLI-only by stated design and not capabilities; apps/majordomus-cli/src/served.rs is deployment observation, not server lifecycle; the top-level `majordomus doctor` is a governance doctor (lib/doctor.sh:53-77) and checks nothing about the server. The launcher rebuilding the executable in the primary checkout is I2059.
Desired state
Every lifecycle fault in the audit's matrix has a test: N concurrent first clients, a busy owner, a killed owner, a lost lease with live bridges, a SIGTERM racing a takeover, a declared timing, a UI-only user. server.status answers one of absent, starting, ready, degraded, outdated, stale, stopping or failed, decided in one function. docs/MCP.md and README.md describe the server that ships: one per checkout, what it writes, when it ends.
Scope
- apps/majordomus-cli/src/lease.rs
- apps/majordomus-cli/src/shared.rs
- apps/majordomus-cli/src/commands/serve.rs
- apps/majordomus-cli/src/commands/mcp.rs
- apps/majordomus-cli/src/commands/env.rs
- apps/majordomus-cli/src/http
- apps/majordomus-cli/src/capability/builtin/server.rs
- apps/majordomus-cli/src/capability/builtin/health.rs
- apps/majordomus-cli/src/policy.rs
- .ai/repo/policy.yaml
- apps/majordomus-cli/tests/
- test/cases
- docs/MCP.md
- README.md
Out of scope
- A `majordomus ui` command, unless I2131's record decides one is needed beyond `serve ensure`.
- One server for all repositories. ADR 0044 decided one per checkout and the board is gathered across them.
- Graceful termination inside the production container, which is I0906 (fly-deployment).
- The bridge's resend of a tool call, which is I2145 (mcp-transports-are-reliable).
Acceptance criteria
- A server whose four workers are busy with tool calls is never taken over by a client that started meanwhile
- A server that lost its lease has no session left after a declared grace and exits by itself
- No process removes a lease file whose token is not its own, proved under a SIGTERM racing a takeover
- `serve stop` runs the ordered stop and leaves no open episode
- A lease timing declared in policy.yaml is the one the mcp and serve elections wait
- server.status reports stopping, failed and degraded, each from a test that produces the state
- A server used only through the Cockpit or /events is not stopped as idle
Validation
- bin/majordomus plan validate
- cargo test --test mcp_shared --test serve_lifecycle --test lease_busy --test lease_lost --test lease_declared --test server_status
- MJ_TEST_JOBS=4 bash test/run.sh 90_mcp_shared_server 113_shared_server_survives_a_crash 191_entry_races 192_entry_across_checkouts_and_clients
Evidence required
- lifecycle_fault_tests
- typed_server_standing
- docs_describe_the_shipped_server
Risks
- Every issue here edits apps/majordomus-cli/src/lease.rs, apps/majordomus-cli/src/shared.rs or apps/majordomus-cli/src/commands/serve.rs, so they are not parallel-safe with each other; one worker should carry them in dependency order.
- A wrong takeover rule turns one healthy server into two. Each change needs the storm test (apps/majordomus-cli/tests/mcp_shared.rs) green before and after.
- The live servers on the owner's machine run whatever was last built; a change to the lease schema reaches them only on restart, so a mixed fleet must be tolerated during the rollout.
Issues
Each one is an execution contract with its own acceptance criteria, validation command and required evidence. The wave column is the layer of the dependency graph it sits in.
| issue | status | wave | depends on | title |
|---|---|---|---|---|
| I2126 | DONE | 0 | — | A server busy with tool calls is never taken over as if it were dead |
| I2127 | DONE | 0 | — | A server that lost its lease hands its sessions on and ends |
| I2128 | DONE | 0 | — | A lease is removed only by the process that holds it, and serve stop stops the server in order |
| I2129 | DONE | 0 | — | The lease timings the policy declares are the ones the mcp and serve elections use |
| I2131 | DONE | 0 | — | A server in use through the Cockpit or HTTP is not stopped as idle, and a checkout keeps one address |
| I2700 | READY | 1 | I2126 | A lease probe asks a cheap liveness route, and a server holding another lease token is not taken for the leaseholder |
| I2701 | DONE | 1 | I2129 | The idle timeout is declared beside the other server timings, and serve --idle overrides the declaration |
| I2702 | DONE | 1 | I2131 | A checkout keeps one address across restarts, because its server binds a port derived from the checkout |
| I3179 | READY | 0 | — | The runtime reports one typed state, with how it got there, and every surface shows it |
| I3180 | BLOCKED | 1 | I3179 | Entering a checkout whose executable is missing or stale starts one background build and says building |
The graph
flowchart LR
I2126["I2126<br/>A server busy with tool calls is never taken over as if it were dead"]:::done
I2127["I2127<br/>A server that lost its lease hands its sessions on and ends"]:::done
I2128["I2128<br/>A lease is removed only by the process that holds it, and serve stop stops the server in order"]:::done
I2129["I2129<br/>The lease timings the policy declares are the ones the mcp and serve elections use"]:::done
I2131["I2131<br/>A server in use through the Cockpit or HTTP is not stopped as idle, and a checkout keeps one address"]:::done
I2700["I2700<br/>A lease probe asks a cheap liveness route, and a server holding another lease token is not taken for the leaseholder"]:::ready
I2701["I2701<br/>The idle timeout is declared beside the other server timings, and serve --idle overrides the declaration"]:::done
I2702["I2702<br/>A checkout keeps one address across restarts, because its server binds a port derived from the checkout"]:::done
I3179["I3179<br/>The runtime reports one typed state, with how it got there, and every surface shows it"]:::ready
I3180["I3180<br/>Entering a checkout whose executable is missing or stale starts one background build and says building"]:::blocked
I2126 --> I2700
I2129 --> I2701
I2131 --> I2702
I3179 --> I3180
classDef done stroke:#16a34a,fill:#dcfce7,color:#14532d,stroke-width:2px
classDef active stroke:#2563eb,fill:#eff6ff,color:#1e3a8a,stroke-width:2px
classDef verify stroke:#7c3aed,fill:#f5f3ff,color:#4c1d95,stroke-width:2px
classDef ready stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
classDef planned stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
classDef blocked stroke:#b45309,fill:#fffbeb,color:#78350f,stroke-width:2px
classDef cancelled stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2px
classDef superseded stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2pxLinked from this diagram (10)
- I2126 A server busy with tool calls is never taken over as if it were dead
- I2127 A server that lost its lease hands its sessions on and ends
- I2128 A lease is removed only by the process that holds it, and serve stop stops the server in order
- I2129 The lease timings the policy declares are the ones the mcp and serve elections use
- I2131 A server in use through the Cockpit or HTTP is not stopped as idle, and a checkout keeps one address
- I2700 A lease probe asks a cheap liveness route, and a server holding another lease token is not taken for the leaseholder
- I2701 The idle timeout is declared beside the other server timings, and serve --idle overrides the declaration
- I2702 A checkout keeps one address across restarts, because its server binds a port derived from the checkout
- I3179 The runtime reports one typed state, with how it got there, and every surface shows it
- I3180 Entering a checkout whose executable is missing or stale starts one background build and says building
Canonical record: .ai/repo/project/milestones/runtime-lifecycle.yaml. Read it back with majordomus plan show runtime-lifecycle.