Repository navigation
Wave 2: six evidence fixes, verified in parallel, plus the records they falsified - #686
Merged
Merged
Conversation
…r it The Readiness-v1 sheets (dated 2026-05-19) and ReleaseSignOff-v1 (bec2a6e, 2026-05-22) mark tiers PASS. An audit claimed several of those PASSes rested on nothing. This change checks each one against its OWN date and SHA, not against today. A filter that broke in June does not falsify a PASS recorded in May. Each PASS row, and each tick in the release sign-off, now carries a dated "[corrected 2026-09-30: ...]" note in place, with one of four labels: SUPPORTED (with the run id), UNSUPPORTED-AT-THE-TIME, LATER-DECAYED (with date and cause), or UNEVIDENCED (a local run nobody can now inspect, not disproven). Nothing is removed or re-ticked: a word diff against 876cc52 shows only insertions in all 9 files. The production-readiness README and each corrected sheet state the rule from now on. A PASS names a run on the header SHA; otherwise the row reads NOT RUN or CARRIED FORWARD with both SHAs. A readiness PASS cites a Readiness summary run whose verdict is `verified`. The labels follow that rule. A run on any commit other than the sheet's is "carried forward tested -> sheet", even when the trees are identical (a PR head and its merge), and the note names the tree hash or says what differs. A run that finished after the release sign-off says "after sign-off". What the evidence shows: - No commit held these gates on 2026-05-19. The scripts first appear in 43d5fca (authored 2026-05-22 00:45 UTC), and 7457e6c's Makefile has none of the targets. The sheets were committed in 0ecb5bf, whose tree f1e22df equals PR #110's head ee8befa. - SUPPORTED on 0ecb5bf itself: kernel A (26262716833). Carried forward ee8befa -> 0ecb5bf (identical tree f1e22df): application A (26262666898), composition A-C (26262666838; static counts reproduce the sheet's 44/6/18 exactly), security A-B (26262666843), and the portal and agent-server dry runs (26262666791). Carried forward 7457e6c -> 0ecb5bf (2026-05-19): kernel B and ship A (26129468793), mesh-lab-e2e and composition D (26129468841), and kernel C's gRPC leg and D's pack-list check. - UNSUPPORTED-AT-THE-TIME: security C's unit step and 3 of the 4 suites named by application B are UnitTestBase suites that FullyQualifiedName~ cannot select. CI run 31982500018 prints "No test matches ... TrustCommandTests" and then PASS. ReleaseSignOff's "all automated gates green for target SHA": 8 of the 19 runs on bec2a6e failed, including Full Platform Readiness Gate and its Readiness summary. When the box was ticked, none had succeeded and 4 had already shown a failure (mapbox-tile-helpers-ci 26262737725 and onboarding-quickstart-gate 26262737929 at creation, then Cross-Platform Tests and Persistence Tests jobs at 01:36). - LATER-DECAYED: security B since e6b5579 (2026-08-17); application C since 9579c81 and e6b5579; compat A's migration step since 678d071 (2026-06-08). CI logs and a container run at 876cc52 show each one printing "No test matches" and then PASS. - Release sign-off, all after the 01:38:36 sign-off commit: security Tiers A-B carried forward 2a1b4d1 -> bec2a6e (PR #114 head, identical tree b2996d4, run 26262735412, done 01:57:29); compat 26262738367 (01:56:14); DR Tiers A-B 26262738375 (01:50:21). waterproofing-gate.yml has never run (API total_count 0). dr-gate's own tier C took its placeholder path in CI, but "mesh director persistence verified" is SUPPORTED after sign-off by Mesh virtual lab gate run 26262738372 on bec2a6e, which runs the same persistence script with the key and LiteDb provider it needs. - The Operations tick ("rollback drill recorded"): the drill record exists, filled in by the sign-off commit fb54476. Its checks passed CI only after sign-off. It records a restore in place ("To version: N/A"), not a rollback to an earlier version, and its reports went to gitignored .nexo/. The mesh task id it cites (20260522001559314-...) is NOT a timestamp of the drill: the persistence script always posts idempotency key mesh-lab-persist-idem, POST /api/mesh/tasks returns the existing task for a known key, and the task lives on a named volume that mesh-lab-up and dr-gate-tier-c never remove. The id dates the first seeding of that task (00:15:59 UTC, before bec2a6e existed), so it cannot show which tree the local drill ran on. What this does NOT fix: the gates themselves. Security B, application C and compat A still print PASS on zero tests, because nothing in those scripts rejects a filter that matches nothing. The new evidence rule is prose only: no check reads these tables, so a wrong run id in a note stays green, and scripts/rc-gate-tier-e.sh (run by waterproofing-gate) still reports "sign-off: checked items found" from the ticked lines, including the Engineering tick now annotated UNSUPPORTED-AT-THE-TIME; the README says so. RollbackDrill-v1.md's own "Result PASS" is not annotated; the note sits on the sign-off tick that links to it. May test counts cannot be recovered (logs return HTTP 410); counts given are static counts at the historical SHA, checked against 2026-08-17 CI logs. CompatReadiness, DRReadiness and PerfReadiness have no PASS rows and are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt pass cert-gate is a required check, and `dotnet test` exits 0 when a test is skipped. The zero-test guard floored the TRX `total` against `dotnet test --list-tests`, and both of those count a skipped test, so a skip moved both sides of the floor together. Measured on 876cc52 by putting [Fact(Skip = "...")] on WorkflowFilesParseTests.EveryWorkflowFileParsesAsYaml (Certification namespace): run-cert-gate.sh exited 0, the guard printed "cert-gate reported 1288 tests (expected>=1283, ...)", and the TRX read total=1288 executed=1287. The audit claim holds. What this does: - scripts/cert-gate-skip-guard.sh, run by run-cert-gate.sh straight after the zero-test guard, requires `total - executed` to EQUAL the integer in the new committed file scripts/cert-gate-skipped.baseline. More skipped: red, naming the skipped tests and saying to raise the baseline in the same change if the skip is deliberate. Fewer skipped: red, saying to lower it, so slack cannot absorb a later skip. Missing TRX, missing or malformed baseline, or no single readable <Counters> element: red. It uses total - executed, not the TRX `notExecuted` attribute, which read 0 beside the skipped result in the same measured run. - The baseline is 0: 1288 total / 1288 executed in the devtest container and in the cert-gate logs of ten CI runs (master and pull_request, 2026-09-29/30); no test in the cert-gate selection declares a Skip, and xunit 2.9 has no runtime skip. It is its own file, not derived from the run or a generated artifact. - Counters are read from the <Counters> element only (cert_gate_trx_counter, added to scripts/cert-gate-config.sh). The filter line is untouched and is still line 6, where ci/cert-gate-assertions.md and the closing plan cite it. The zero-test guard used to take the first `total="N"` anywhere in the file, which can be text a test printed; it now uses the same reader. Its fail-closed branch (no count derivable from --list-tests -> exit 1) is unchanged. The reader requires whitespace before an attribute name; no TRX counter name is the tail of another today (grep is case-sensitive, so `notExecuted` never matches `executed=`), so that is defence against a future attribute, and the comment says so rather than claiming a collision. - scripts/cert-gate-summary.sh exits 1 when there is no TRX, or a TRX with no readable <Counters>, instead of 0 (both cases asserted), and reports executed / reported / skipped. Every run that reaches it without a TRX or readable counters has already failed in run-cert-gate.sh, whose guards refuse both, so no verdict changes; the step just stops contradicting it. run-cert-gate.sh no longer also writes the summary: the workflow's `if: always()` step already did, so CI showed the table twice. - tests/scripts/cert-gate-guards.test.sh (29 assertions, discovered by scripts/ci/run-repo-gates.sh inside required shell-lint) copies the real scripts into a sandbox root and drives them with fixture TRX files shaped like the measured one (BOM, results before the summary, notExecuted="0" beside a skip) and a stand-in dotnet that exits 0 on a skip. Verified in the devtest container (net8.0, exactly `bash scripts/run-cert-gate.sh`): the clean run exits 0 with "0 skipped (total=1288 executed=1288)"; the same Skip mutation now exits 1 at the skip guard, naming the test; with the TRX deleted, both guards and the summary exit 1. Ten script mutations (unwiring the guard, one-directional compare, reading notExecuted, first-match parsing in either guard, summary exit 0 on a missing TRX, summary `|| echo 0` on an unreadable <Counters>, dead-probe floor, no trailing-whitespace trim on the baseline, no whitespace before the attribute name in the reader) each redden only the assertions naming that behaviour. The baseline-trim one discriminates on Linux, where shell-lint runs it; Git-for-Windows bash strips a trailing CR in command substitution, so locally it cannot. What it does NOT fix: - The count is enforced, not the identity: un-skipping one cert-gate test and skipping another in one change keeps the count equal (both edits are still in that change's C# diff). - A Theory-level Skip collapses all its data rows into one skipped result, so the guard goes red but reports 1 where N cases were lost; the rows also drop out of --list-tests, so the floor cannot see them either. - Deleting a test, or removing its [Fact], lowers both the TRX total and --list-tests together; neither guard sees that. - A test that returns early without asserting still counts as executed. - An environment-conditional skip ([NotOnCiFact], [OptInFact]) placed in the cert-gate selection would make the count differ between CI and a local run; none is there today. - The summary still hardcodes four GenerationSafetyTests names and prints "not found" if they are renamed; it is informational, not a gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts lane VSTest exits 0 when --filter matches no test. Three lanes printed PASS having run nothing, measured in the devtest container at 876cc52 and visible in CI logs: - security-gate Tier B (every PR on its paths; run 36642632648 prints "No test matches" then "security-gate-tier-b: PASS"). It ran -f net8.0, and e6b5579 compiled Tests/API for net10.0 only. Its fifth name, SecurityAnalysisRuleTests, is a UnitTestBase suite that no FullyQualifiedName filter has ever selected. net8.0 selects 0; net10.0 selects 44. - compat-gate Tier A, step 1 (run 34687917953). MeshTaskExecutionServiceTests moved to commercial/tests/...Fleet in #151 and the step kept pointing at Tests.Infrastructure. Infrastructure selects 0; Commercial.Tests.Fleet selects 1. - application-gate Tier C. Both classes link Ashlar.API (net10.0 only). net8.0 selects 0; net10.0 selects 4. What this does: - scripts/ci/zero-test-guard.sh is modelled on cert-gate-zero-test-guard.sh and fails closed. It reads the TRX the run wrote, and it fails when: - the TRX is missing; - its Counters cannot be read; - executed == 0; - a RunInfo <Text> carries VSTest's "No test matches the given testcase filter". Skipped tests count as not run: VSTest 18.x counts a skip in total and not in executed, and leaves notExecuted at 0. So an all-skipped selection fails. A partly skipped one passes and prints the skip count. --solution covers a single solution-wide invocation. - The three repairs, each proven with --list-tests and a real run in the container: - security Tier B is two invocations. The API classes run on net10.0 (44), and the SecurityAnalysisRuleTests bridge row runs as its own invocation (1). Folded into one filter, that row also exists on net8.0. The framework-revert mutation then left the lane at 1 executed test, and the guard stayed green. - compat Tier A step 1 runs on Commercial.Tests.Fleet (1). - application Tier C runs on net10.0 (4). The window is 720s wherever a HostTouching (480s) fact is selected. - The guard follows every pinned filter in the workflows, the gate scripts, the Make targets a workflow reaches, and the compose service compose-gate brings up: 91 invocations in all. Each keeps its own TRX, and set -e / make still exit on the test's own failure first. Invocations that ran two target frameworks under one LogFileName now run one per framework: cross-platform-tests, dogfood-* and products-gate. VSTest overwrites a shared name. - compose-gate: the docker-compose.test.yml service runs a pinned filter (BaseFrameworkSmokeTests, 9 tests) and then writes ok:true whether or not a test ran; the step only checked that file existed. Measured in the container with the filter broken: the service exited 0 and wrote ok:true, and the guard the step now runs failed it. - perf-certification moves to LogFilePrefix with per-project TestResults (two projects in one run both wrote perf-certification_net8.0_20260930163958.trx), and its metrics step sums every TRX. The gather now runs BEFORE the test's exit code is re-raised: under set -e a failing run used to exit before the copy, so the report said 0 tests / 0 failed and the artifact carried no results. - scripts/ci/verify-zero-test-guard-wiring.py is discovered by run-repo-gates, so it runs in required shell-lint. It derives every pinned dotnet test from .github/workflows, scripts/**/*.sh, tests/**/*.sh, the Makefile and deploy/compose/*.yml, and requires: - a TRX of its own, -f on a multi-target project, LogFilePrefix and --solution for a .sln; - a later guard naming that TRX in the SAME unit (Make target, workflow run block, script; for a compose service, after the line that brings it up). "Naming" is read from the guard's arguments, not its line: quotes removed, a trailing # comment dropped, GitHub expressions compared as written, and one argument's basename must EQUAL the LogFileName (--solution: match <prefix>_* or *__<prefix>_*). A substring test let a guard over kernel-gate-hosting.trx pass for a run writing hosting.trx, and a comment naming the TRX pass for a guard over another file; - a guard that can fail: a command of its own (not echo/if/!), not text in a heredoc body, no ||, &&, ;, | or & on its line, no make `-` prefix, set -e in force (no set +e before it); - no `exit`/`return` with no status or a literal 0 between the test and its guard in the same script or run block (Make lines are shells of their own). It has a literal floor of 100 (106 found; fixture text is not counted), five required lanes (one per kind of file scanned) and stale-exemption checks. - Tests: zero-test-guard.test.sh (20) against TRX files VSTest wrote, including one whose StdOut prints the no-match phrase (the <Text> anchor); zero-test-guard-wiring.test.sh (46) against throwaway trees, including a TRX name that only contains the run's (barfoo.trx), a name only in a comment, a --solution prefix that contains the run's, a guard only in a heredoc, and exit 0 / a conditional bare exit before the guard; perf-certification-trx.test.sh (8) drives the real workflow steps with a stand-in dotnet. Every behavioural change was mutation-checked. The mutation was applied, its original string was counted to 0, the named assertion went red, the tree was restored clean, and the test went green again. What it does NOT fix: - This is not a floor. A lane shrinking from 40 tests to 1 still passes. So does an OR-filter that loses one term while another term keeps it above zero. - The wiring check does not see continue-on-error, a step's `if:`, or a non-bash step shell. It reads a unit as a flat list of lines, not control flow: it accepts a guard in a DIFFERENT if/else branch (or case arm) from its test, and a guard inside a shell function, subshell or loop. It accepts an exit whose status is not a literal (`exit "$rc"`, `exit $?`) between a test and its guard - perf-certification re-raises `exit "$rc"` correctly and the check cannot tell it from `rc=0; exit "$rc"` - and it does not see exec, kill $$, or an exit inside a function called in between. A heredoc whose delimiter is not a plain word is not recognised as one. It does not reach Dockerfile CMD lines, C# that launches dotnet test, or PowerShell. - full-platform-readiness-gate.yml and the cert-gate scripts are owned elsewhere and exempted here. The readiness smoke lanes have no guard, and two of them run both frameworks under one LogFileName. cert-gate's own guard reads total=, and total counts skipped tests. uat-gate Tier 2 is exempt: it checks Total=16, which also counts skipped tests. - ops-gate Tier B now fails on CI. That is deliberate. dogfood-block8 selects one [NotOnCiFact] test, which is skipped under GITHUB_ACTIONS, so on CI the lane never asserted anything. - compat-gate stays red on master at Tier B for an unrelated reason: the CLI pipeline run returns state Failed. - The docs still describe security Tier B and application Tier C as PASS. docs/HowGatesGoQuiet.md section 12 still lists these three lanes as open. ci/test-ownership.tsv:54 and a help-text example in TestOwnershipRunnerConventionTests.cs:222 cite line numbers this moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re it before tagging #685 made `Readiness summary` compute verified / partial / not-verified / failed, but only as job outputs and a step summary. Job outputs are readable only inside the run, and the Checks API does not expose the step summary (output.summary is null on the check run), so a release sign-off still had nothing to read but the green tick - which also passes when no platform ran. The summary step now also emits ::notice title=Readiness verdict::verdict=<v> lanes_ran=<n> lanes_skipped=<m> before its exit, so a failing run publishes "failed" rather than nothing. Annotations ARE served by GET /repos/{owner}/{repo}/check-runs/{id}/annotations (measured on check run 109702461129), and a titled notice arrives with its title in the `title` field (measured with the one-word title VERSION on rc-gate check run 104216814783; a title with a space is inferred to survive). scripts/release/readiness-verdict-for-sha.sh <sha> reads that annotation for every `Readiness summary` check run of the SHA and exits 0 only on VERIFIED. It fails closed, naming the reason on its last line, when: no readiness run exists (on push the gate is path-filtered, so a docs-only commit has none); the annotation is missing on any run (a run from before this change, or a summary job cancelled early), even beside a verified run; runs disagree (any `failed` beside a pass is conflicting-verdicts); a run is still in progress, including a second run that has no summary check run yet and shows only in the workflow-runs listing; a check run of that name comes from another app or workflow file, or it or its Actions run names another commit; a verified annotation reports skipped lanes; or gh is unauthenticated or the API errors (exit 2). `partial` is refused with no override, and so is a `verified` reported only by a pull_request run, which tested refs/pull/N/merge rather than the SHA. Re-run attempts of one workflow run are read at their latest attempt, as branch protection reads them. What each check run's Actions run is - its workflow file, its commit, whether it has finished - comes from GET actions/runs/{id}, fetched per run, never from the workflow-runs listing, which is fetched first and can be stale. The fetching is bash + gh; the decision is scripts/release/readiness-verdict-for-sha.py, which also takes an offline fixture via READINESS_VERDICT_FIXTURE. A fixture is never evidence, so its output says so on the first and last lines, and a VERIFIED fixture exits 3, never 0: `READINESS_VERDICT_FIXTURE=<file> bash ... $SHA && git tag` does not tag, and neither would a future gate built on the exit code. A fixture that reads REFUSED or UNDETERMINED keeps 1 or 2. For a SHA with no run the script prints the honest route: push a throwaway branch pinned to the SHA and dispatch the gate on it. `gh workflow run --ref` takes a branch or tag, not a SHA, and evaluates whatever the ref points at when the run starts; a master dispatch also shares the push runs' concurrency group, so the next readiness push run on master cancels it. That route cannot clear a FAILED run, and the likeliest refusal on a release SHA is exactly that: a merge that touches a readiness path cancels the run still in progress for the commit before it, and that run's summary records failed (master push run 36637044333 on f34a9f6 was cancelled by the 0.2.0 release commit's push; its Readiness summary concluded failure). The failure stays attached to the SHA, so a verified dispatch beside it is conflicting-verdicts; only a re-run of that run supersedes it. A re-run keeps its ref, so a re-run of a master run joins master's concurrency group (cancel-in-progress: true): it cancels any master readiness run in flight and is cancelled by the next readiness-path merge. Every piece of re-run advice the script prints, and the runbook, says so and says to re-run only when `gh run list --workflow full-platform-readiness-gate.yml --branch master` shows nothing queued or in progress. docs/RELEASE_RUNBOOK.md "Before you tag" gains step 9, appended so no existing step number moves (nothing in docs/ cites one by number): run the script on the SHA being tagged, what each outcome means (only a VERIFIED read from GitHub exits 0; one ending "[offline fixture - not evidence]" exits 3), how to re-run a master run safely, and the path-filter route. Tests. readiness-verdict.test.sh (10 -> 18 assertions) checks the single annotation line for every verdict and round-trips the real step's line through the reader, so writer and reader cannot drift apart together; the round trip expects exit 3, since it feeds the reader a fixture. readiness-verdict-for-sha.test.sh (new, 43 assertions) drives every branch from fixtures, leads with non-vacuity controls (it can say VERIFIED, and it demonstrably read the fixture), expects exit 3 from every fixture VERIFIED, and exercises the live path against a stub gh, where a VERIFIED exits 0. The live cases include the in-progress run only the workflow-runs listing can show, and two in which the listing and GET actions/runs/{id} disagree (run absent from the listing; listing says completed while the run's own fetch says in_progress), so a reader that issues that GET but reads the listing fails both. Every decision branch, the per-run read, the fixture exit code and every piece of re-run advice has a mutation that was observed reddening its own assertion, then restored to a clean tree. Repo gates: 16 of 16 pass. NOT fixed here: - A commit that predates this change has no annotation, so the script cannot certify it; it refuses and says to read the run summary by eye. - It is a runbook step, not a gate: release.yml does not call it. - A run cancelled by a newer push is reported as failed, the same as a real failure; the operator tells them apart by opening the run. The workflow does not distinguish them in the annotation. - That a re-run cancels master's in-flight readiness run is inferred from GitHub's documented re-run semantics (same GITHUB_REF) and the workflow's concurrency key, not measured: measuring it would mean re-running on master. - The stub gh proves the plumbing, not GitHub's API; the API shapes were measured by hand on real check runs, and GitHub's annotation caps are not re-checked by anything. - master's own readiness run on 876cc52 is red on the macOS lane (SelfExtendAdmissionBridgeTests); unrelated to this change, but this PR edits the readiness workflow and so runs that lane. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and gate it The native readiness lanes run `ashlar ci verify`, which runs `ashlar validate`, which builds and runs every test project its recursive sweep discovers (22 on this tree). Those lanes are diff-conditional: they run only when a PR touches a glob in READINESS_PATHS. The transitive ProjectReference closure of the 22 suites is 70 projects, and 37 of their directories matched no glob. So editing src/Ashlar.Mcp.Server (built into six suites) skipped every lane that runs them, while editing their tests re-ran them. #684 closed this for the kernel suite alone. The same hole existed one level out: src/Ashlar.Compat is compiled into most of the closure through src/Directory.Build.props and Directory.Build.targets, three nested Directory.Build.props and the root .editorconfig are read implicitly, and the CLI embeds samples/templates/brick while Tests.Infrastructure copies samples/physical-atom-cert. None of those re-ran readiness either. What this does: - Adds those 44 entries to BOTH readiness lists (on.push.paths and READINESS_PATHS), at the same positions and in the same order: the 37 project directories as dir/** after the kernel test entry, and the 7 implicit and cross-directory inputs after the shared root build inputs. - Adds ReadinessClosureConventionTests (cert-gate). It calls validate's own discovery predicate (under a virtual root, because the predicate also reads the absolute path for "test"), walks ProjectReference with an XML parser so attribute order cannot hide an edge, follows analyzer-only (OutputItemType=Analyzer, ReferenceOutputAssembly=false) and conditional references on purpose, and fails naming the uncovered path, what names it, the list and a suite that needs it. - Reads references from every shared MSBuild file a closure project is evaluated from, not only from the project file: the implicit Directory.Build.props/.targets and Directory.Packages.props above it, and any file it reaches by an explicit <Import>, transitively. A ProjectReference declared in src/Directory.Build.props is a real edge of every src/ project, and a first version of this test dropped it silently: a verifier added a Condition'd reference to src/Ashlar.Lite there for the kernel suite and the gate stayed green. Such a reference is now an edge of every closure project that reads the file (conditions are not evaluated, the same over-approximation as for conditional references). A reference, or an item path with a directory, in a shared file that is not anchored by $(MSBuildThisFileDirectory) is refused, because MSBuild resolves it against each importing project and it names no single file; so is a computed or wildcard Import. - Also checks every implicit Directory.Build.*/Directory.Packages.props/.editorconfig/ .globalconfig/nuget.config/global.json above a closure project, and every path outside the project's own directory that a closure project or one of those shared MSBuild files names: in the Include of an item of ANY type except the identity types (PackageReference, PackageVersion, FrameworkReference, InternalsVisibleTo, Using and the like, and ProjectReference, which the walk covers) and a None item that is not copied, in any item's HintPath, or in an Import. The item types are a denylist on purpose. A second version scanned an allowlist of six types (Compile, EmbeddedResource, Content, AdditionalFiles, Analyzer, None), and a verifier showed the item type alone decided the verdict: a cross-directory <Protobuf> item, a Reference HintPath and an EditorConfigFiles item in src/Ashlar.Transport.Grpc (which already compiles a Protobuf item) left it green, while a Compile item at the same spot turned it red. Moving that project's .proto to a shared protos/ directory would have kept cert-gate green while later edits to the contract skipped the lanes that build its suites. Now a type nobody listed is read as naming files, and a computed path on one fails instead of being skipped. A bare file name in a shared file names a file in each importer's own directory, which that project's glob covers; a path under $(NuGetPackageRoot) is in the package cache, outside the repository. $(Property) paths and wildcard/absolute/escaping/missing references fail rather than being approximated. Floors are stated constants (20 suites, 60 projects, 6 implicit, 3 cross-directory; measured 22/70/10/6, unchanged by the denylist). - Adds its row to ci/cert-gate-assertions.md (after the runner-column row, so the "line 34" citation in TestOwnershipRunnerConventionTests stays true), updates the readiness core-path prose in docs/CiGateInventory.md, and regenerates the knowledge graph (+1 file, +11 facts). Cost, measured: of the last 40 merged PRs (#633-#685) none would newly run the heavy lanes. 21 ran them already, 19 did not, and a replication of the in-job bash matcher agreed with the observed Linux-lane conclusion on all 40. The only two that touched a newly covered path (#640, #661) were already heavy through another entry. Mutation-checked in the devtest container (net8.0): - Removing src/Ashlar.Mcp.Server/** and src/Ashlar.Compat/** from both lists reddens the two tree facts naming exactly those paths, while AiPipelineCiRoutingConventionTests stays green. - A walker that sees only Include-first references reddens the fixture fact (misses lib/B and lib/C). Reordering the attributes of Core.Application's analyzer reference leaves the logged closure identical. Breaking the discovery port fires the suite floor at 0. - The verifier's shared-file edge (src/Directory.Build.props referencing src/Ashlar.Lite for the kernel suite) plus a direct control edge (src/Ashlar.Hosting.Bundle) now reddens the tree fact naming both directories in both lists; on the previous walker only the control was named. - Reading only the project file, not following explicit imports, accepting an unanchored shared reference, and letting a wildcard Import through each redden their own fixture facts. - The verifier's allowlist probe, re-applied to src/Ashlar.Transport.Grpc.csproj (a never-true ItemGroup with a cross-directory Protobuf item, a Reference HintPath and an EditorConfigFiles item, plus a property-function read of ver-verifier/v.txt), now reddens the tree input fact with 6 problems: protos-verifier/probe.proto (Protobuf), lib-verifier/VerifierProbe.dll (Reference HintPath) and cfg-verifier/probe.editorconfig (EditorConfigFiles), in both lists. It went green on the allowlist. ver-verifier/v.txt is not named; that is the documented property-function limit below. - Restoring the six-type allowlist reddens the new fixture fact (it misses 6 of its 7 paths) and the computed-Protobuf row. Not reading HintPath reddens the fixture fact (misses lib/Probe.dll) and the computed-HintPath row. Dropping the $(NuGetPackageRoot) rule makes CopyAssemblies.targets' AssembliesToCopy list throw in the tree fact. Dropping the bare-name rule, or InternalsVisibleTo from the identity list, reddens only the fixture fact. Each mutation turned red only those facts (24 or 25 of 26 still pass). The clean tree passes 26/26 here, 70/70 together with AiPipelineCiRouting and CertGateFilterCoverage, and 1314/1314 over the whole cert-gate filter. What this does NOT fix: readiness-summary still counts a skipped lane as a pass; validate still passes a project that selects zero tests; the container lanes do not run the sweep at all; files tests read at runtime (ci/*.json, workflows, docs) are not build inputs and are not traced; files read through an MSBuild property function are not traced - $([System.IO.File]::ReadAllText) is how the root Directory.Build.props and Directory.Build.targets read VERSION, which is in both lists by hand and is not proven by this test; paths in item metadata other than HintPath (a Protobuf ProtoRoot or AdditionalImportDirs) and files a compiled input reaches on its own (a .proto import) are not traced; projects an <MSBuild> task builds inside a target are not traced; and projects `ci verify` builds that are not discovered suites are out of scope. ci/test-ownership.tsv rows 47-49, 55 and 58, docs/HowGatesGoQuiet.md, CertifierReferenceSetTests' summary and the readiness CLOSING-PLAN still describe the old routing gap; they belong to other owners and are listed for them. This proves routing, not that anything passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tests
WHY. `ashlar validate` passed on `totalTestsFailed == 0 && evidenceErrors.Count == 0`, so a
discovered test project that executed nothing contributed nothing and the sweep still reported
green. The readiness gate's native lanes run exactly that unfiltered sweep (`ci verify` ->
`validate`), and four rows of ci/test-ownership.tsv plus the SPEC-006 conformance paragraph had
to carry the caveat "a zero-selection project still passes (ValidationServiceAdapter.cs:248)".
A project that silently lost every test - all of them skipped, or all of them tagged into a
category validate excludes - read the same as one that ran 412.
WHAT. After a completed run (no abort, evidence readable), a project whose EXECUTED count is zero
is an EvidenceErrors entry when the caller did not narrow the sweep:
- Per project, keyed on executed (not skipped) tests, not on the NoTestsSelected arm. An
all-skipped project records rows, classifies as ResultsRecorded, and would slip past a check
on that arm; a dedicated fact covers it. A project with SOME executed tests beside skips still
passes ("a documented skip is not a red test" is unchanged).
- Per project means the project's OWN executed count, not the sweep's running total. Every
single-project fixture has total == project count, so a rule judged on the total, on the
result list, or only for the first project passed all of them - and would reopen the gap in
the real 22-project sweep for any empty project discovered after one that ran something. A
two-project E2E fact pins it: tests/PassTests.csproj (2 executing facts, `<Compile Remove=
"zz/**" />`) is discovered BEFORE tests/zz/EmptyTests.csproj (Stress-only), because file
enumeration yields a directory's own files before it descends; the arrange step asserts that
order through the adapter's own discovery predicate. It requires TestsRun == 2, Passed false,
and exactly one EvidenceErrors entry starting "EmptyTests.csproj: executed no tests in an
unfiltered sweep (it selected none".
- Independent of HOW the run was classified as empty. NoTestsSelected is reached two ways: a
zero-row TRX, or no TRX under the project beside the console's "No test matches the given
testcase filter" line. VSTest 17.12 writes a zero-row TRX even for an empty selection, so every
earlier fixture took the first way, and a rule tied to TRX presence passed all of them. A new
E2E fact sends its TRX out of the project through VSTestResultsDirectory (the shape a
repository-level Directory.Build.props commonly has), guards that no TRX landed where validate
looks, and pins both legs: unfiltered fails "it selected none", Category=Stress still passes
as "no tests selected".
- "No caller filter" is string.IsNullOrWhiteSpace(filter), through one helper
(CallerNarrowedTheSweep) that the argv builder now also uses, so `--filter ""` cannot run an
unfiltered sweep with the rule switched off.
- Routed through EvidenceErrors, which already owns "fail without inventing a test": TestsRun,
TestsPassed and TestsFailed stay what was observed. When nothing failed but a project gave no
evidence, the headline now reads "Validation failed (N/N tests passed, but 1 project produced
no passing evidence); <project>: <reason>" instead of "Validation failed (0/N tests failed)".
That headline also applies to the existing invalid-evidence failure; no script parses it. The
message is passed through unchanged by the CLI, POST /validate and the director's
RunValidation option, so all three show the new headline and the new failure.
- Invalid evidence is judged first and reported once, as itself - including under a caller's
filter, which a new leg pins (a filter must not turn an unreadable TRX into a pass).
- Unchanged: the zero-DISCOVERY early return ("No test projects found - validation skipped",
docs/IntegratorGuide.md), and an empty selection under a caller's filter, which still passes
and is still reported as "no tests selected".
- ValidationResultRenderingTests documented a passing "0/0; no tests selected" pairing the
unfiltered path can no longer produce. It now carries the three pairings the adapter actually
composes, and a new fact derives them from the adapter so they cannot drift again.
POLICY (Stress / DockerOptional, and all-skipped suites). validate always excludes
Category=Stress and Category=DockerOptional, so a project whose every test carries one of those
traits legitimately selects nothing. It FAILS anyway: this is the conservative choice. A sweep
that reports "passed" for a project it ran nothing in is a lane, not a receipt. A future
all-Stress or all-DockerOptional test project must carry at least one test outside those
categories (a fast structural or smoke fact) so the sweep executes something in it. The same
applies to a project whose every test is SKIPPED where the sweep runs it - a suite made only of
[OptInFact] tests without their switch, only of [NotOnCiFact] tests under CI=true, or skipped
wholesale on one platform: it fails on that lane and must carry one test that runs there without
the dependency. There is deliberately no opt-out marker. The fact that pinned the old behaviour,
An_empty_selection_passes_with_zero_observed_tests, is flipped on purpose into
An_unfiltered_sweep_fails_a_project_whose_every_test_is_excluded_by_default.
DAY ONE. Measured on readiness run 36713219284 (master 76395ff): all 22 discovered projects
executed > 0 tests on Linux, Windows and macOS (smallest: HelloBrick.Tests 1, Ingress.DynamoDb 2;
totals 5703/5703, 5694/5694, 5703/5703), so this is green on day one - which also means CI
history cannot validate it. It was mutation-checked instead, in the devtest container: each
mutant applied with its original string's count proven to drop to 0, then restored with the tree
proven clean and the suite green again (GapCoverage + ValidationFilter/Evidence certification
102/102 and Tests.CLI rendering 4/4 around the first revision's mutants; the whole Validation
namespace + those two certification classes 121/121 around the three no-TRX mutants, and
122/122 around the four multi-project mutants, each of which reddened 1/122).
- rule removed (`executed >= 0`): 12 facts red plus the CLI tie fact.
- rule keyed on the NoTestsSelected arm only: exactly one fact red, the all-skipped one,
which is therefore the only guard for that arm.
- rule applied only when a TRX exists (`trxFile is null ? null : ZeroExecutionError(...)`,
the mutant a verifier found surviving every earlier fact): 120/121, only the new no-TRX
fact red ("Expected result.Passed to be False ... but found True").
- the no-TRX fixture's redirect pointed back INSIDE the project: the new fact's own guard
red (a TRX found under tests/trx-in), while the adapter's verdict was unchanged - so the
guard is the only thing telling the two arms apart, and it works.
- the no-TRX console arm read as invalid evidence: the new fact red (EvidenceErrors[0] said
"test results are missing" instead of "it selected none").
- `filter is not null` inside the rule: the 4 blank-filter rows and the whitespace E2E leg.
- skips counted as executed: 6 red.
- any skip fails the project: 2 red, including "a documented skip is not a red test".
- invalid-evidence gate lost: the drop-rows fact red (misreported as zero-execution).
- the old "0/N tests failed" headline back: 2 red plus the CLI tie fact.
- rule applied under a caller's filter: 5 red, including the must-survive
Caller_filter_selects_observed_results_and_preserves_default_exclusions.
- shared blank-filter helper as `is not null`: the cert-gate argv facts
(An_absent_filter_retains_the_default_exclusions "" and whitespace) and the rule's rows.
- M-AGG, the rule judged on the running total (`ZeroExecutionError(testProject.Name, filter,
totalTestsRun, ...)` - the mutant a verifier found surviving 121/121): 121/122, only the
two-project fact red ("Expected result.Passed to be False ... Validation passed (2/2 tests);
no tests selected: EmptyTests.csproj, but found True").
- the rule judged on the result list (`allTestResults.Count`): 121/122, only that fact red,
same message.
- the rule fired only for the first discovered project (`currentProject == 1 ? ... : null`):
121/122, only that fact red, same message.
- the two-project fixture's empty project moved to the sweep root, so it is discovered FIRST
(where a running total is still 0 and M-AGG would survive): the fact's order guard red
before anything ran.
WHAT THIS DOES NOT FIX.
- The floor is ONE executed test per project, not a project's population. A suite that drops
from 412 to 1 still passes; the rows and SPEC-006 keep "a lane, not a receipt" for that reason.
- The sweep still prints no per-project count in its result.
- The heavy readiness lanes are still diff-conditional and readiness-summary still counts a
skipped lane; this rule only bites when the lane runs.
- No cert-gate fact pins it: the facts live in Tests.Infrastructure's Validation namespace (and
Tests.CLI), which run in the readiness sweep and the container, not in cert-gate.
- A runner that writes no TRX at all for an empty selection is exercised only through the
redirect fixture, which reaches the same no-TRX arm; no toolchain available here does that.
- The new E2E facts have run on Linux only (devtest container); their first Windows and macOS
runs are this PR's readiness lanes. The two-project fact's order guard rests on .NET's
breadth-first file enumeration, which is the same on every OS; if it ever differs, the guard
fails loudly rather than letting the fact go vacuous.
- docs/CiGateInventory.md still says a project that selects zero tests still passes, and
CHANGELOG [Unreleased] has no entry; both files are outside this change.
Verified in the devtest container, on this commit's tree: Validation namespace +
ValidationFilter/EvidenceCertification 122/122 and the cert-gate filter 1288/1288 at net8.0
(every convention test that reads a file touched here passed: TestOwnership*,
GateStoreAnchorProvenance, GateSignatureResidual, LaneBlameWindow, Timeout,
ValidationEvidence/FilterCertification). On the previous revision, whose production code,
Tests.CLI and Tests.Application sources are byte-identical to this one: Tests.Infrastructure
(net8.0 + net10.0), Tests.CLI (net10.0) and Tests.Application build; Tests.Infrastructure bridge
24/24 (incl. ValidationServiceAdapterTests), Tests.CLI 38/38 (rendering, ConsoleRenderer, bridge
incl. ValidateCommandTests), Tests.Application bridge 21/21 (incl. RunValidationValidatorTests).
The two-project fact adds one declared fact (knowledge graph 4387 -> 4388).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each of the six workstreams below refused to edit a file outside its ownership and listed the text its change made false instead. This applies those corrections in one place, after all six are in, so no record lags the code it describes. No behaviour changes. - ci/test-ownership.tsv rows 47, 48, 49, 55 and 58 said their suites were in neither readiness path list. The closure change put them in both; each row now says so, and row 58 records that the wider 41-directory gap is closed and what the closure test does not trace (files read through MSBuild property functions). - Row 54 and TestOwnershipRunnerConventionTests' help text cited line numbers (perf-gate-tier-a.sh:23-24, Makefile:194, full-platform-readiness-gate.yml:586) that were already wrong. They now cite the invocation, the target header and the job id, which do not move when lines are added above them. - HowGatesGoQuiet.md, CLOSING-PLAN.md and a CertifierReferenceSetTests doc comment described the analyzer / ingress routing gap as open; it is closed. - HowGatesGoQuiet.md section 12 listed three zero-test lanes as open and said MeshTaskExecutionServiceTests "exists in no project". All three are fixed, and the class exists: it moved to Ashlar.Commercial.Tests.Fleet in #151, which is why the step selected nothing. MeshPhase6LeasesAndCheckpoints.md placed it in Ashlar.Tests.Infrastructure; corrected. - CiGateInventory.md, certification-evidence.md and Testing.md described the sweep's zero-selection pass and cert-gate's guards as they were before. - The release checklist issue template told the operator to register nuget.org Trusted Publishing for release.yml (and release-nuget.yml). docs/PUBLISHING.md records, verified on the v0.1.1 publish, that nuget.org matches the REUSABLE workflow that runs NuGet/login - reusable-release-nuget.yml - and that a policy on a caller does not match. The 0.2.0-rc1 rehearsal failed on exactly "No matching trust policy". The template also omitted the runbook's pre-tag steps 4-9, including the rc rehearsal and the readiness verdict; it now carries them. - CHANGELOG [Unreleased] was "Nothing yet." It now records the user-visible changes: validate failing zero-execution projects, cert-gate's skip baseline, the zero-test guard, the readiness verdict, and the closed routing gap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
cert-gate on this exact tree ( 1288 → 1314: the additions include 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seven commits, meant to be rebase-merged so each lands on
masteras its own commit. Six areindependent fixes built in parallel, each in its own worktree; the seventh brings the records into line
with them. One PR instead of six because merges here are serial (
strict: true) — six PRs would be sixCI cycles, most of them heavy.
Each of the six went through the same loop: implement → adversarial verifier → repair → second verifier
→ repair → final verifier that could block only on an unaddressed finding, a fail-open defect or a
day-one red. All six ended LAND with every finding closed. Every behavioural change was
mutation-checked — broken on purpose and observed failing with real counts — before it counted.
docs(readiness)fix(ci)cert-gate skip[Fact(Skip=…)]on a cert-gate test leftrun-cert-gate.shat exit 0 reporting 1288 tests, because the TRXtotaland--list-testsboth count a skip. A skip guard pinstotal - executedto a committed baseline (0) that fails in both directions.cert-gate-summary.shno longer exits 0 on a missing TRX.fix(ci)zero-test guarddotnet test --filterselecting nothing now fails its lane — 91 invocations guarded, and a wiring check in requiredshell-lintrefuses an unguarded one. security Tier B, application Tier C and compat Tier A each passed on zero tests; each now selects its intended tests (44, 4, 1).feat(release)verdictReadiness verdictannotation — the only channel readable from outside a run.scripts/release/readiness-verdict-for-sha.sh <sha>exits 0 only for a VERIFIED read live from GitHub; every other outcome fails closed and names why, and an offline fixture can never exit 0. Runbook step 9.fix(ci)closureReadinessClosureConventionTests(cert-gate) recomputes the closure from the tree, reading every item type except a denylist of identity items, so a<Protobuf Include="..\..\protos\x.proto"/>cannot slip past.fix(validate)ashlar validatefails a project that executes zero tests — judged per project, including all-skipped ones the old code path could not see, routed throughEvidenceErrorsso no count is invented.--filter ""cannot turn it off. A caller filter selecting nothing still passes.docs(records)release.yml—docs/PUBLISHING.mdrecords, verified on v0.1.1, that the policy must target the reusablereusable-release-nuget.yml. The 0.2.0-rc1 rehearsal failed on exactly "No matching trust policy".Verification of the combined tree
regenerating it (current, 4,399 declared facts).
run-repo-gates.sh: 21 of 21 pass (15 before this PR).Known risk on this PR's own run
The heavy lanes run here. There is a ~1.1% chance per readiness run of an unrelated red from a
thread-safety bug in YamlDotNet 13.7.1, reached through a static deserializer in
PolicyLoader— root-caused and reproduced (10/6000 concurrent cold parses), and being fixed in aseparate PR. If a lane reds with "policy could not be parsed: Exception during deserialization", that
is the cause; it is not this change.
Not in this PR
rc-gate-tier-e.shstill accepts a ticked box.full-platform-readiness-gate.yml's own four pinned smoke filters are exempt from the zero-test guard(it is a required context; wiring them needs its own care).
🤖 Generated with Claude Code