failure-is-visible — Every failure the runtime can suffer is bounded, tested or stated, and changes the health it reports

A client can tell running from ready from mesh-healthy from degraded on one endpoint; no client can hold a worker past a declared deadline; each failure class in a published table names its test or its stated limitation.

ACTIVE 5 of 6 issues done

ready: 1
blocked: 0
active: 0
verify: 0
done: 5
cancelled: 0

Problem

Readiness stays true for a degraded index and knows nothing of a lost lease or the mesh (apps/majordomus-cli/src/capability/builtin/health.rs:214; was_lost reaches only GET / at apps/majordomus-cli/src/http/router.rs:710); health.report has no mesh dimension (apps/majordomus-cli/src/capability/builtin/health.rs:299); 'degraded' has three unrelated meanings. The HTTP server reads request bodies with no deadline on four workers (apps/majordomus-cli/src/http/server.rs:23,208-222), so four slow clients hold the server. Of the failure classes the pack names, the audit found tests for SIGKILL, port collision, an unwritable lease directory and a frozen owner, and none for disk full, a read-only journal directory, a slow client, a frozen worker or a kill between lease creation and its first write; lease keep-alive write errors are swallowed (apps/majordomus-cli/src/lease.rs:1058-1060). The pack's premise of structured logging is refuted: logging is human tracing fmt on stderr with no JSON, correlation id or redaction (apps/majordomus-cli/src/logging.rs:29-37). The top-level doctor is a governance doctor (lib/doctor.sh:53-77). Measured in the runtime baseline audit of 2026-10-09 at master 1fb8d6be44 (plan/RUNTIME_BASELINE.md).

Outcome

Current state

Implemented: /api/v1/live, the lease's client-side bounded waits, link timeouts, clock-skew bounds, the mesh doctor's typed checks (apps/majordomus-cli/src/mesh/doctor.rs:93-372). Partial: readiness, health, failure-class coverage. Absent: server-side request deadlines.

Desired state

The failure table in docs/DEVELOPMENT_RUNTIME.md has no row without a test or a limitation, a case refuses a row with neither, and readiness and health reflect every state the runtime already knows about itself.

Scope

apps/majordomus-cli/src/capability/builtin/health.rs
apps/majordomus-cli/src/http
apps/majordomus-cli/src/lease.rs
apps/majordomus-cli/src/logging.rs
apps/majordomus-cli/src/mesh/doctor.rs
apps/majordomus-cli/src/policy.rs
.ai/repo/policy.yaml
lib/doctor.sh
lib/common.sh
apps/majordomus-cli/tests/
test/cases
docs/DEVELOPMENT_RUNTIME.md
docs/MCP.md

Out of scope

Circuit breakers; the audit found no evidence one is justified.
A metrics stack or log shipping; the runtime reports through its own capabilities.
Journal durability, which is I2137 (mesh-journal-converges).

Acceptance criteria

/api/v1/ready reports false with a named reason after a lease loss
health.report carries a mesh check when the mesh is enabled
Slow clients hold no worker past the declared deadline
Every failure class in the published table names a CI test or a stated limitation

Validation

bin/majordomus plan validate
cargo test --test health_server --test http_serve --test mcp_shared --test lease_busy
MJ_TEST_JOBS=4 bash test/run.sh <the cases these issues add>

Evidence required

readiness_reflects_state
request_deadline_test
failure_class_table

Risks

Making readiness stricter can make deploy health checks fail where they passed; the fly-deployment health block reads /api/v1/live and must keep doing so.
A deadline on request reads that is too short breaks slow links; it must be declared and measured.

Evidence

A milestone is never done because its issues are; the last step is its own acceptance, and it is gated on the rows below.

covers	type	command	result
readiness_reflects_state	ci	gh run view 38635619021 (validate on master c0e3585dc8, push): job rust (3) 116098148455 runs cargo test --test lease_lost --test health_server; job rust (2) 116098148534 runs the lib test capability::builtin::health::tests	success. lease_lost: a_server_that_lost_its_lease_says_so_and_takes_on_nobody_new ok (1 passed; /api/v1/ready false with reason lease_lost); health_server: 7 passed, including an_enabled_mesh_that_runs_is_a_healthy_dimension_as_the_doctor_says, an_enabled_mesh_whose_every_transport_failed_fails_the_dimension_as_the_doctor_does, the_mesh_is_a_dimension_of_health_and_unknown_where_no_server_decided_it; lib: readiness_names_a_degraded_layer_and_is_still_ready, readiness_reports_what_this_process_already_holds, readiness_refuses_a_process_whose_executable_was_removed_under_it ok. Delivered by I2157 (DONE); the run's head is origin/master, so the tested code is the code on master
request_deadline_test	ci	gh run view 38635619021 (validate on master c0e3585dc8, push): job rust (3) 116098148455 runs cargo test --test http_slow_client	success. http_slow_client: slow_client_released_test ok (slow clients closed at the declared deadline and counted in connections.closed_by_deadline, a fifth client answered), past_the_handler_bound_a_request_is_refused_and_the_probe_is_answered ok (2 passed, 16.88 s). The deadline is declared as server.read_deadline_seconds: 30 in .ai/repo/policy.yaml and documented in docs/MCP.md. Delivered by I2156 (DONE); the run's head is origin/master

Issues

Each one is an execution contract with its own acceptance criteria, validation command and required evidence. The wave column is the layer of the dependency graph it sits in.

issue	status	wave	depends on	title
I2156	DONE	0	—	A slow or silent HTTP client cannot hold a server worker
I2157	DONE	0	—	Ready means ready: a lost lease, a degraded layer or a failing mesh is not reported ready
I2707	DONE	1	I2157	majordomus doctor reports a stale server, or says plainly that it checks governance only
I2731	DONE	0	—	Every failure class in the published failure table names its test or its stated limitation
I2827	DONE	0	—	A tool written a moment ago is run once its writer lets go
I3310	READY	0	—	majordomus start refuses a task, owner or reason that the task record cannot hold, and writes nothing

The graph

failure-is-visible dependency graphfrom .ai/repo/project

flowchart LR
    I2156["I2156<br/>A slow or silent HTTP client cannot hold a server worker"]:::done
    I2157["I2157<br/>Ready means ready: a lost lease, a degraded layer or a failing mesh is not reported ready"]:::done
    I2707["I2707<br/>majordomus doctor reports a stale server, or says plainly that it checks governance only"]:::done
    I2731["I2731<br/>Every failure class in the published failure table names its test or its stated limitation"]:::done
    I2827["I2827<br/>A tool written a moment ago is run once its writer lets go"]:::done
    I3310["I3310<br/>majordomus start refuses a task, owner or reason that the task record cannot hold, and writes nothing"]:::ready
    I2157 --> I2707
    classDef done stroke:#16a34a,fill:#dcfce7,color:#14532d,stroke-width:2px
    classDef active stroke:#2563eb,fill:#eff6ff,color:#1e3a8a,stroke-width:2px
    classDef verify stroke:#7c3aed,fill:#f5f3ff,color:#4c1d95,stroke-width:2px
    classDef ready stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
    classDef planned stroke:#0891b2,fill:#ecfeff,color:#164e63,stroke-width:2px
    classDef blocked stroke:#b45309,fill:#fffbeb,color:#78350f,stroke-width:2px
    classDef cancelled stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2px
    classDef superseded stroke:#6b7280,fill:#f9fafb,color:#374151,stroke-width:2px

Linked from this diagram (6)

Canonical record: .ai/repo/project/milestones/failure-is-visible.yaml. Read it back with majordomus plan show failure-is-visible.