[ffroliva]

Summary

Makes e2e evidence a contributor requirement instead of an opt-in maintainer activity.

E2E is the layer that proves a change works against the real Flow blackbox. Offline-green only proves our code does what we think it does. Nothing in the contributor path actually asked for it:

• the CONTRIBUTING lifecycle table routed only generation-path changes to live-verify
• "Test categories" documented how to run e2e, framing it as opt-in maintainer regression
• the PR template's only live-Flow line was scoped to generation paths too

External PR #671 followed all of that to the letter and arrived with eight offline test files and zero e2e — and the HTTP fast path it added failed on the first real run.

This also records what /gflow:live-verify is not: it drives CLI commands by hand into a gitignored tmp/live-verify/ note and never runs pytest -m e2e. An e2e test is a re-runnable regression; a live-verify ledger is a narrative record. The two are complementary, and the docs now say so.

Changes

• CONTRIBUTING.md — new lifecycle row (any behavior change → the e2e test you ran or wrote); the generation-path row now reads as layered on top of it; "Three rules" becomes four; new "E2E is required, not optional" block explaining that CI cannot run these, which is precisely why it falls to the contributor.
• AGENTS.md — the same rule where agents read it, the live-verify clarification, and the full cost sub-marker list.
• .github/PULL_REQUEST_TEMPLATE.md — an e2e line that has to be answered: name the test you ran, point at the new one, or say why you could not.
• docs/DEVELOPMENT.md (+ website mirror) — no longer says the maintainer runs e2e on the author's behalf.
• tests/test_documentation_gate.py — two tests pinning the rule, so deleting any of its three statements fails CI instead of passing quietly.

Council fixes (65a431e)

Review returned YELLOW with five must-fix items, all applied:

1. Exemption was stated three incompatible ways — CONTRIBUTING said "not reviewable" (absolute), the template said "say why and a maintainer will run it" (exempt), AGENTS.md offered no exemption. All three now carry the template's wording.
2. Scoped to "a behavior change that touches a Flow surface" — as written the rule demanded an e2e run from docs-only, help-text and exit-code changes, which cannot produce one. This PR is itself such a change.
3. CONTRIBUTING still called e2e "opt-in" fourteen lines under "required, not optional" — now "excluded from the default run; select with -m e2e".
4. Dropped the "~25 of them" count (actually 32 files) — a hardcoded count drifts.
5. Untouched files still taught the old rule — docs/DEVELOPMENT.md, AGENTS.md's "All 7 quality gates" over a nine-command block, and a 10-phase table with no phase that runs the suite (Phase 8 is now "E2E + Live Verification").

Plus nice-to-haves: GFLOW_CLI_E2E_RUN_CHARACTER added to the marker list, and a redaction hint wherever the docs ask for pasted output.

Test plan

Docs and tests only; no src/ changes.

• uv run python scripts/ci/check_repo_hygiene.py — 944 files, no violations
• uv run python scripts/ci/check_doc_links.py — all links resolved across 29 files
• uv run python scripts/ci/check_website_docs_pii.py — clean across 25 published files
• uv run python scripts/ci/generate_website_docs.py --check — mirror in sync (20 files)
• uv run python scripts/ci/check_council_memory.py — 47 memory files, all cited and resolving
• uv run ruff check src tests / ruff format --check — clean
• uv run pyright src — 0 errors, 0 warnings
• pytest tests/test_documentation_gate.py tests/mcp tests/scripts tests/api/transports/test_overlay_precondition.py — 412 passed

The earlier body said "293 passed"; that run predated tests/test_documentation_gate.py — the one test file that reads AGENTS.md and the PR template, and therefore the one most able to catch a mistake here. It is now included, and the count above is from the current SHA.

E2E evidence: not applicable — this PR touches no Flow surface. Stated rather than silently skipped, per the exemption it introduces.

[ffroliva]

PR #675 — Council Review Verdict

Reviewed at 0adec44 (pinned, no drift). Protocol: skills/pr-council-review/SKILL.md v2.1, docs-only mode (D4 reframed to docs-verification). Seven dimensions dispatched in parallel; every negative claim below was re-verified by the synthesiser with git show 0adec44:<path>.

Consensus: 🟡 YELLOW — the rule is right, its wording contradicts itself and five untouched files

Dim	Verdict	Headline
D0 CI-mechanical	🟢	hygiene (944 files), doc-links (29), website mirror, council-memory (47) all green at the SHA; tests/mcp tests/scripts tests/api/transports/test_overlay_precondition.py → 403 passed; tests/test_documentation_gate.py → 7 passed. 14/14 GitHub checks green.
D1 Correctness	🟡	The load-bearing claim — /gflow:live-verify never runs pytest -m e2e — is TRUE (grep of the skill and wrapper: zero hits). Marker list, addopts exclusion, env vars, fixtures all verified. One number is wrong.
D2 Quality	🟡	Same rule written four times in three wordings; the repo's own "one canonical place, others cite" rule is violated inside the PR.
D3 Security	🟢	No blocker. "Paste the result" has no redaction hint, and e2e skip/failure output carries local paths, profile names and UUIDs by construction.
D4 Docs-verification	🟡	Gates reproduce; the PR body's test count does not, and the only test that reads these files was missed.
D5 Memory	🟢	No contradiction; maintainer memory says exactly what the PR says. Two memory actions on merge.
D9 Docs drift	🟡	Five untouched files still state the old cadence in the present tense.
D14 YAGNI	🟡	Exemption path is stated three different ways; "every behavior change" has no carve-out for changes with no Flow surface — including this PR.

Must-fix (5)

1. Exemption contradicts itself. CONTRIBUTING.md:42-43 says a behavior change without e2e "is not reviewable"; CONTRIBUTING.md:123-124 and PULL_REQUEST_TEMPLATE.md:15 say "if you cannot, say why and a maintainer will run it"; AGENTS.md:99 has no exemption at all. Pick one (the template's) and make the other copies cite it. Add the four words "that touches a Flow surface" to the canonical statement so a help-text or exit-code change is not asked for an e2e it cannot have. (D14, D1)
2. "e2e tests … are opt-in" survives 14 lines under "E2E is required, not optional". CONTRIBUTING.md:126, same file, same PR. Reword to "excluded from the default run; select with -m e2e". (D9, D2, D5)
3. "tests/e2e/ has ~25 of them" — CONTRIBUTING.md:117. git ls-tree gives 32 files, collection gives 75 tests. Drop the number; nothing checks it. (D1, D2, D4)
4. Untouched files still state the old rule as fact. docs/DEVELOPMENT.md:131 "The maintainer reviews, runs e2e if needed, and merges" (regenerate website/docs/DEVELOPMENT.md); AGENTS.md:179 Pipeline Continuation "All 7 quality gates green" (the PR fixed six→nine one section up); AGENTS.md 10-phase table has no phase that runs pytest -m e2e — widen Phase 8 to "E2E + Live Verification". (D9)
5. PR body test plan. Claims "293 passed"; actual at the SHA is 403. The set omits tests/test_documentation_gate.py, the one test that reads AGENTS.md and the PR template (it passes, 7/7). Correct the body. (D4)

Nice-to-have (5)

1. Redaction hint next to every "paste the result" (template :15, CONTRIBUTING :27 and :118): "trim to the summary line; scrub paths, profile names, project/asset UUIDs". (D3)
2. AGENTS.md:100 marker list: names GFLOW_CLI_E2E_RUN_VIDEO but not GFLOW_CLI_E2E_RUN_CHARACTER (pyproject.toml:161). Either both or replace the list with a pointer to pyproject.toml / docs/E2E_TESTING.md. (D1, D2, D14)
3. Wire the rule: two assertions in tests/test_documentation_gate.py pinning "E2E evidence provided" in the template and "| Any behavior change |" in CONTRIBUTING. (D4)
4. skills/pr-council-review/SKILL.md: D4 never asks for e2e evidence and § 9 says e2e "runs deliberately last" — opposite ordering to the new rule. Owned by the PR author as a follow-up after merge. (D9)
5. skills/live-verify/SKILL.md:156 "does not replace check or doc-review" — add "nor pytest -m e2e". (D9)

False positives

None. Spot-checked CONTRIBUTING:126, CONTRIBUTING:42-43, DEVELOPMENT.md:131, AGENTS.md:179 and the e2e file count (32) against the SHA — all as reported.

Confirmed-good

• The central claim about live-verify is true and the memory store agrees with it.
• Marker/cost attributions match pyproject.toml one-for-one; addopts excludes e2e; fixtures exist.
• "six gates → nine gates" drive-by is correct.
• Docs-only, no secret-shaped strings, no captured data; all doc gates green.

Memory actions (D5)

• UPDATE maintainer feedback-e2e-is-the-required-evidence-layer.md + MEMORY.md: "CONTRIBUTING does not require it" becomes false on merge.
• ADD in-repo docs/superpowers/memory/e2e-is-the-required-evidence-layer.md and cite it from the D4 row of the slug table in the same commit (else check_council_memory.py fails).

Verdict handling

YELLOW = soft block. Must-fix 1-4 are small edits in the PR's own files plus two lines in docs/DEVELOPMENT.md; 5 is the PR body. Returned to the author (gflow-cli-5d). Re-review the fixup commit, then merge.