HTTP API: evidence

One claim with everything the repository can say about its proof, and the way back out to the test that carries it.

The proof is flattened in, so a client reads one claim rather than an evidence wrapper around one; the executions and the other claims the same test proves sit beside it.

use majordomus_cli::capability::builtin::evidence::ClaimEvidence;
use majordomus_cli::evidence::ClaimProof;

let proof: ClaimProof = serde_json::from_value(serde_json::json!({
    "id": "evidence-both-directions",
    "claim": "a claim and its test answer for each other",
    "status": "guaranteed",
    "source": null,
    "implementation": null,
    "test_path": "test/cases/84_distribution_model.sh",
    "test": "suite:84_distribution_model",
    "state": "not_run",
    "meaning": "no run of this test has been recorded",
    "execution": null,
    "changed": [],
    "reproduce": "bash test/run.sh 84_distribution_model"
}))
.unwrap();

let answer = ClaimEvidence {
    proof,
    executions: Vec::new(),
    also_proves: vec!["evidence-record-is-not-networked".to_string()],
};
let json = serde_json::to_value(&answer).unwrap();
assert_eq!(json["id"], "evidence-both-directions");
assert!(json.get("proof").is_none(), "the claim is flattened in, not nested");
assert_eq!(json["also_proves"], serde_json::json!(["evidence-record-is-not-networked"]));
// a list even where the ledger keeps one: retention is a decision that may change,
// and a client that read a bare object would break when it did
assert_eq!(json["executions"], serde_json::json!([]));

One claim, joined to whatever the repository actually recorded about it.

The claim's own fields are carried verbatim beside the derived ones, so that a reader who disbelieves the state can check the join rather than take it.

use majordomus_cli::evidence::{ClaimProof, ProofState};

let proof = ClaimProof {
    id: "scope-integrity".into(),
    claim: "a commit outside the task's scope is refused".into(),
    status: "guaranteed".into(),
    source: Some("docs/SCOPE.md".into()),
    implementation: Some("apps/majordomus-cli/src/scope.rs".into()),
    test_path: Some("test/cases/07_scope.sh".into()),
    test: Some("suite:07_scope".into()),
    state: ProofState::NotRun,
    meaning: ProofState::NotRun.meaning().to_string(),
    execution: None,
    changed: vec![],
    detail: None,
    reproduce: Some("bash test/run.sh 07_scope".into()),
};

let json = serde_json::to_value(&proof).unwrap();
// `changed` is emitted empty rather than omitted: a client must not have to tell
// "no file changed" from "the server did not say"
assert_eq!(json["changed"], serde_json::json!([]));
// an execution that does not exist is absent, not a null standing in for one
assert!(json.get("execution").is_none());
// and so is a detail the state did not need
assert!(json.get("detail").is_none());
assert_eq!(json["meaning"], serde_json::json!(ProofState::NotRun.meaning()));

Example

A ClaimProof is produced by report; its state is the whole answer and its meaning is that answer in a sentence, worded once for every surface.

use majordomus_cli::evidence::{report, ClaimProof, Ledger};
use majordomus_cli::synthetic::SyntheticRepository;
let repo = SyntheticRepository::small().unwrap();
let ledger = Ledger::load(repo.root()).unwrap();
let r = report(&repo.index().unwrap(), &ledger);
let proofs: &[ClaimProof] = &r.claims;
assert!(proofs.is_empty(), "a synthetic tree declares no claims");

A claim whose declared status the evidence does not support.

Only a guaranteed claim can produce one: the other statuses already say that a current proof is not what they are claiming. A finding says what was claimed, what the evidence supports instead, and — where running something would settle it — what to run.

use majordomus_cli::evidence::{Finding, ProofState};

let finding = Finding {
    claim: "scope-integrity".into(),
    status: "guaranteed".into(),
    state: ProofState::NotRun,
    reason: "the claim guarantees a behaviour and names a test, and no run of that \
             test has ever been recorded"
        .into(),
    reproduce: Some("bash test/run.sh 07_scope".into()),
};

// the gap is the finding: what the matrix declares against what the ledger holds
assert_ne!(finding.status, finding.state.label());
assert!(finding.reproduce.unwrap().starts_with("bash test/run.sh"));

Example

use majordomus_cli::evidence::{Finding, ProofState};
let f = Finding {
    claim: "some-claim".into(),
    status: "guaranteed".into(),
    state: ProofState::NotRun,
    reason: "the claim names a test nobody has run".into(),
    reproduce: Some("bash test/run.sh 07_scope".into()),
};
assert_eq!(f.state, ProofState::NotRun);

What the ledger is and whether it is there at all.

A report that did not say where its evidence came from would be asking to be believed. This is the provenance of the whole report: the file, how much it holds, how old the newest entry is, and which commits anything was ever recorded against.

use majordomus_cli::evidence::{Ledger, LedgerSummary, LEDGER_PATH};

let summary: LedgerSummary = Ledger::empty().summary();
assert_eq!(summary.path, LEDGER_PATH);
assert!(!summary.present, "a ledger holding nothing has recorded nothing");
assert_eq!(summary.executions, 0);
assert!(summary.newest.is_none());
assert!(summary.commits.is_empty());

Example

use majordomus_cli::evidence::LedgerSummary;
let s = LedgerSummary {
    path: ".ai/repo/evidence/ledger.json".into(),
    present: false,
    executions: 0,
    newest: None,
    commits: vec![],
};
assert!(!s.present, "a repository that recorded nothing says so");

What a report was judged at: the revision, its tree, and the runs the checkout holds that the revision's own ledger does not.

A verdict is always a verdict at something. evidence show judges the working tree; evidence show --presented HEAD judges the checked-out commit as committed, from the ledger committed in it — which is the reading a site built from that commit needs.

use majordomus_cli::evidence::{PresentedRevision, TreeState};

let here = PresentedRevision {
    revision: "working_tree".into(),
    tree: TreeState::Dirty,
    uncommitted: vec![],
};
let json = serde_json::to_value(&here).unwrap();
assert_eq!(json["tree"], "dirty");
// nothing held uncommitted is absent, not an empty list to read past
assert!(json.get("uncommitted").is_none());

The whole joined picture: every claim of the matrix against every execution recorded.

What the capability answers, what the gate judges and what the site renders are this one value; the totals and the findings are derived from the claims, never counted twice.

use majordomus_cli::evidence::{
    EvidenceReport, Finding, Ledger, PresentedRevision, ProofState, Subject, TreeState,
};

let mut report = EvidenceReport {
    head: None,
    working_tree: "unknown".into(),
    presented: PresentedRevision {
        revision: "working_tree".into(),
        tree: TreeState::Unknown,
        uncommitted: vec![],
    },
    ledger: Ledger::empty().summary(),
    subject: Subject { examined: 0, complete: true, excluded: vec![] },
    claims: vec![],
    totals: Default::default(),
    findings: vec![],
};
assert!(report.satisfied(), "nothing claimed, so nothing unsupported");

// and the same report over a subject it could not wholly read is not a pass
let mut partial = report.clone();
partial.subject.complete = false;
assert!(!partial.satisfied(), "an absent answer is not a clean one");

report.findings.push(Finding {
    claim: "scope-integrity".into(),
    status: "guaranteed".into(),
    state: ProofState::Failing,
    reason: "the claim guarantees a behaviour whose test most recently failed".into(),
    reproduce: None,
});
// one unsupported guarantee is enough: the gate is not a proportion
assert!(!report.satisfied());

Example

The whole matrix joined to the ledger. satisfied is true when no claim declares a guarantee the recorded evidence does not support.

use majordomus_cli::evidence::{report, EvidenceReport, Ledger};
use majordomus_cli::synthetic::SyntheticRepository;
let repo = SyntheticRepository::small().unwrap();
let ledger = Ledger::load(repo.root()).unwrap();
let r: EvidenceReport = report(&repo.index().unwrap(), &ledger);
assert!(r.satisfied(), "no claim, so no unsupported guarantee");

Whether the report could see its whole subject.

A verdict over a matrix the index silently shrank is the failure this repository met four times in one day: an object is excluded, every projection of the smaller index is a faithful projection, and each check prints a true statement about the half it can see. A freshness check answers "was this generated from this tree" and cannot answer "did the generator see everything in it", so the answer has to carry its own denominator.

use majordomus_cli::evidence::Subject;
let whole = Subject { examined: 149, complete: true, excluded: vec![] };
assert!(whole.complete);

let partial = Subject {
    examined: 148,
    complete: false,
    excluded: vec!["docs/CLAIMS.yaml: unknown_key".into()],
};
assert!(!partial.complete, "a shrunken index must not report a clean verdict");

Whether a tree was the commit it sits on: the three words the ledger and git already use, typed.

unknown is not a softer clean. It is what is recorded when git could not be asked, and not knowing is not proof, so every rule here treats it as not clean.

use majordomus_cli::evidence::freshness::TreeState;

assert_eq!(TreeState::parse("clean"), TreeState::Clean);
assert_eq!(TreeState::parse("dirty"), TreeState::Dirty);
// a word nobody wrote down is not a clean tree
assert_eq!(TreeState::parse("spotless"), TreeState::Unknown);
assert_eq!(TreeState::Dirty.as_str(), "dirty");
assert_eq!(serde_json::to_value(TreeState::Unknown).unwrap(), "unknown");

One test, and what it proves.

The other direction of ClaimEvidence, read from the test's end: its identity, its source and whether that source is still the one that ran, the command that runs it again, and every claim that names it.

use majordomus_cli::capability::builtin::evidence::TestEvidence;
use majordomus_cli::evidence::TestId;

let id = TestId::of("apps/majordomus-cli/tests/why.rs").unwrap();
let answer = TestEvidence {
    test: id.as_string(),
    runner: id.runner,
    source: id.source(),
    present: true,
    reproduce: id.reproduce(),
    execution: None,
    digest_matches: None,
    proves: Vec::new(),
};
assert_eq!(answer.test, "crate:why");
assert_eq!(answer.reproduce, "cargo test --test why");

// nothing recorded: the two provenance keys are absent rather than null, so "no
// execution" cannot be read as an execution that said nothing
let json = serde_json::to_value(&answer).unwrap();
assert!(json.get("execution").is_none() && json.get("digest_matches").is_none());
assert_eq!(json["proves"], serde_json::json!([]));