Skip to content

The behavioural suite runs a bounded number of cases at a time with the serial runner's semantics, a case that writes into the checkout declares itself exclusive and runs alone, and the parallel phase fails naming the paths when the checkout changed under it

MJ_TEST_JOBS=4 bash test/run.sh runs the cases four at a time and means exactly what bash test/run.sh means: every case runs, in a disposable repository each, a failing case turns the run red with its own output in front of its verdict, the summary is one deterministic block in name order, a filter that matches nothing is a usage error, and so is an empty case directory. Each case runs in its own repository already, which is what makes them independent; the one thing they share is the checkout they read from, so a case that must write into it says so with # majordomus-exclusive: <reason> and runs after the pool, alone.

declared statusdeclared status: guaranteed Declared deterministic and blocking, implemented, and naming the behavioural test that is to prove it. Whether a recorded run supports it is the evidence verdict, shown beside it.

recorded evidencerecorded evidence: not run Declared guaranteed, and not supported by a recorded run: the test is named, and no passing run of the current inputs is recorded. how evidence is recorded

MJ_TEST_JOBS bounds the pool (xargs -P, never a bare ampersand); each case has its own log; the verdicts are rendered in name order with a failing case's whole log before its line; a filter that matches nothing and an empty case directory are usage errors in both modes; MJ_TEST_REPORT writes one row per case for the summary; test/lib.sh refuses a case that does not stand in the fixture test/run.sh made for it, so a case started by hand cannot write its fixture into a checkout. test/cases/26_ci_wiring.sh proves the failure semantics through both modes on a private harness.

What it means

MJ_TEST_JOBS=4 bash test/run.sh runs the cases four at a time and means exactly what bash test/run.sh means: every case runs, in a disposable repository each, a failing case turns the run red with its own output in front of its verdict, the summary is one deterministic block in name order, a filter that matches nothing is a usage error, and so is an empty case directory. Each case runs in its own repository already, which is what makes them independent; the one thing they share is the checkout they read from, so a case that must write into it says so with # majordomus-exclusive: <reason> and runs after the pool, alone.

How it works

The runner classifies the cases by their header, feeds the parallel ones to xargs -P (bounded; never a bare &), each worker writing the case's log, status and duration to files, then runs the exclusive ones one at a time, then renders the verdicts in name order from the files. Before and after the parallel phase it takes git status of the checkout and fails naming the paths that differ. A case started anywhere but the fixture the runner made for it never reaches its first line: test/lib.sh refuses it with run this case through test/run.sh unless $T and $ROOT are set, the case stands in $T, and $T is neither the checkout, nor inside it, nor another checkout of this repository. MJ_TEST_REPORT writes one row per case (name, result, seconds, phase) for the job summary. Without MJ_TEST_JOBS the runner is the serial one it always was, streaming. test/cases/94_ci_plan.sh proves the semantics on a private harness of throwaway cases: the red run with the log rendered, the exclusive phase, the report, the bounded pool being faster than serial, the checkout guard naming the path, the usage errors; test/cases/26_ci_wiring.sh proves the failure semantics through both modes.

How to see it

MJ_TEST_JOBS=4 MJ_TEST_REPORT=/tmp/suite.tsv bash test/run.sh
sort -t "$(printf '\t')" -k3,3nr /tmp/suite.tsv | head       # the slowest cases
bash test/run.sh                                            # serially, streaming
grep -l '^# majordomus-exclusive:' test/cases/*.sh          # the cases that run alone
just test-shell 94_ci_plan

What it does not cover

The guard sees the checkout's tracked and untracked files, not ignored ones: a case writing under site/public/ or apps/majordomus-cli/target/ is not caught by it, and the two site cases declare themselves exclusive for that reason. A case that depends on the machine being idle (a wall-time assertion) is not made deterministic by the runner; case 65 was normalised for that. The worker count is a measured choice per runner, recorded in the CI model and the baseline, not a promise about any other machine.

Why it exists

The suite ran one case after the other and took half an hour on a runner where its cases sum to a few minutes of work each; the long tail is wide and flat, so a bounded pool cuts the wall time without changing what is proved. The exclusive header keeps the invariant project.tests-run-in-disposable-repos states, cases never write into this checkout while others read it, explicit and checked instead of assumed.

Detail rendered from docs/claims/suite-parallel.md.

From the statement to the proof

  1. 1 · the public statement

    The behavioural suite runs a bounded number of cases at a time with the serial runner's semantics, a case that writes into the checkout declares itself exclusive and runs alone, and the parallel phase fails naming the paths when the checkout changed under it

    Made on CI.

  2. 2 · the canonical claim

    suite-paralleldeclared status: guaranteed in docs/CLAIMS.yaml

    Defined in read it on this site · docs/CI.md

  3. 3 · the implementation

    test/run.sh

  4. 4 · the test and the scenarios

    Test: test/cases/94_ci_plan.sh

  5. 5 · the recorded runs

    recorded evidence: not run how evidence is recorded

Verify it yourself

The test runs in a disposable temporary repository and asserts the behaviour, not a string in the source.

from a clone of the repository
bash test/run.sh 94_ci_plan

Related claims same implementation

Searches the titles and descriptions of every page this site publishes.