ci: one gate was rebuilding the binary the other 360 were running

.github/workflows/ci.yml

@@ -2,12 +2,17 @@ name: CI
 
 # Guardrails as automated gates (CLAUDE.md "The guardrails as automated gates"):
 #   - release job:  configure (portable default, no RIPWIRE_NATIVE — CI hardware is not "this dev machine")
-#                    + build + the full gate suite (170+ gates incl. determinism/G4/quality) per
-#                    OS x build-flavour (Release / plain — see the job comment), run in parallel via
+#                    + build + the full gate suite (170+ gates incl. determinism/G4/quality) on SIX legs:
+#                    build-flavour (Release / plain) x front end (macOS AppleClang, Linux gcc, Linux
+#                    clang) — see the job comment for why each axis exists — run in parallel via
 #                    test/pargates.py -j 3 (same authoritative list as regression.sh).
 #                    ubuntu-24.04 here is also the FIRST real proof of L1's portability fix: real x86-64
 #                    Linux hardware, not the local RIPWIRE_PRETEND_LINUX proxy test/portablebuildcheck.sh
 #                    uses on the (Apple Silicon) dev machines this repo is normally built on.
+#   - rhel job:      the platform this project ships a Linux prebuilt for, checked at PR time instead of only
+#                    at release. RHEL 9 userland (ubi9) via `container:` on a hosted Ubuntu runner, built with
+#                    Red Hat's own gcc-toolset (RHEL 9's default gcc 11 cannot do C++23). Build + fixture
+#                    self-run + the two headline guarantees, NOT the full suite — see the job comment for why.
 #   - asan job:      configure -DRIPWIRE_ASAN=ON (G1: address,undefined,integer,float-divide-by-zero,
 #                    float-cast-overflow, fail-fast) + build + the fixture self-run + the 5 heaviest verbs
 #                    (--pack-task/--from-trace/--quality-delta/--merge-scout/--edit-check)
@@ -114,12 +119,29 @@ jobs:
     #             already-gated degrade path + --version's build-type token) and SKIP honestly on the
     #             Release leg, pointing at the plain leg as where the alert is proven; a missing alert
     #             on the plain flavour still FAILS. Neither flavour alone is enough.
-    name: release (${{ matrix.os }}, ${{ matrix.flavor }})
+    #
+    # And BOTH FRONT ENDS on Linux, not just the platform default. gcc is the one this job used to have,
+    # and it is a feature — a second front end over the same tree, which found three real portability bugs
+    # on its first run. But it is not the whole story either: RIPWIRE_OPT_REMARKS and RIPWIRE_PGO are
+    # Clang-only by construction (-Rpass=/-fsave-optimization-record, .profdata/llvm-profdata) and
+    # CMakeLists FATAL_ERRORs on anything else, so on a gcc-only Linux leg optremarkscheck's configure arms
+    # cannot be expressed and can only skip. macOS covers Clang but not Linux's libc, headers or linker.
+    # Six legs = {macos-14 AppleClang, ubuntu gcc, ubuntu clang} x {Release, plain}: every guardrail is
+    # observed by at least one leg on the OS it matters for, and no leg has to assert the impossible.
+    name: release (${{ matrix.os }}, ${{ matrix.flavor }}, ${{ matrix.cc }})
     strategy:
       fail-fast: false
+      # Explicit include list rather than three crossed axes: there is no gcc on the macOS runners (`g++`
+      # there is an AppleClang shim, so a `cc: gcc` macOS leg would silently be a second AppleClang leg
+      # claiming to be gcc), and a leg that lies about its toolchain is worse than a leg that is missing.
       matrix:
-        os: [macos-14, ubuntu-24.04]
-        flavor: [Release, plain]
+        include:
+          - { os: macos-14,     flavor: Release, cc: appleclang }
+          - { os: macos-14,     flavor: plain,   cc: appleclang }
+          - { os: ubuntu-24.04, flavor: Release, cc: gcc }
+          - { os: ubuntu-24.04, flavor: plain,   cc: gcc }
+          - { os: ubuntu-24.04, flavor: Release, cc: clang }
+          - { os: ubuntu-24.04, flavor: plain,   cc: clang }
     runs-on: ${{ matrix.os }}
     steps:
       # L4 (Linux probe): checkout@v4 defaults to a --depth 1 clone, which leaves ONE commit of history
@@ -130,20 +152,35 @@ jobs:
         with:
           fetch-depth: 0   # churn/co-change gates read real git history — a shallow clone reddens them
 
+      # clang is installed on BOTH Linux legs, not only the clang one. optremarkscheck's Clang-only
+      # configure arms pin `clang++` when the default front end is not Clang, so the gcc leg keeps that
+      # coverage instead of skipping it — the same remedy the asan job applies for G1's Clang-only
+      # sanitizer members. It is a package install, not a change of the leg's own compiler: the gcc leg
+      # still BUILDS ripwire with gcc, which is the point of having it.
       - name: Install tooling (Linux)
         if: runner.os == 'Linux'
         run: |
           sudo apt-get update
-          sudo apt-get install -y --no-install-recommends libxml2-utils ripgrep bc
+          sudo apt-get install -y --no-install-recommends libxml2-utils ripgrep bc clang
 
       - name: Install tooling (macOS)
         if: runner.os == 'macOS'
         run: brew install ripgrep
 
-
+      # CC/CXX are set for the clang leg ONLY. Leaving them unset elsewhere is deliberate: the gcc leg and
+      # the macOS leg must each get their platform's own default, which is what a stranger's clone gets.
       - name: Configure (portable — no RIPWIRE_NATIVE; empty build type = the plain flavour)
+        env:
+          CC:  ${{ matrix.cc == 'clang' && 'clang'   || '' }}
+          CXX: ${{ matrix.cc == 'clang' && 'clang++' || '' }}
         run: cmake -S . -B build ${{ matrix.flavor == 'Release' && '-DCMAKE_BUILD_TYPE=Release' || '' }}
 
+      # The front end is recorded per leg so a log reader never has to infer it from the job name alone —
+      # a `cc: clang` leg whose CMake cache says GNU is a broken matrix, not a passing run.
+      - name: Confirm the front end this leg actually configured with
+        run: |
+          grep -h 'CMAKE_CXX_COMPILER_ID\|CMAKE_CXX_COMPILER_VERSION' build/CMakeFiles/*/CMakeCXXCompiler.cmake
+
       - name: Build
         run: cmake --build build -j
 
@@ -165,6 +202,86 @@ jobs:
       - name: G4 — xmllint --noout
         run: ./build/ripwire test/fixture --no-cache | xmllint --noout -
 
+  # ─── rhel: the platform this project actually ships a prebuilt for, checked at PR time ───────────────
+  #
+  # GitHub hosts no RHEL runner image, but a job can run inside RHEL userland ON a hosted Ubuntu runner via
+  # `container:`. UBI (Universal Base Image) pulls from registry.access.redhat.com with no subscription,
+  # which is what makes this work on public CI — release.yml's `smoke-rhel` job already relies on it.
+  #
+  # WHY IT IS WORTH A LEG. release.yml builds the Linux artifact in a manylinux_2_28 container and smoke-runs
+  # it on ubi9 precisely because a binary built on stock ubuntu-24.04 needs GLIBCXX_3.4.31 and DIES on RHEL 9
+  # (verified, and recorded in that workflow's own comment). But that check only runs when a release is cut:
+  # every ordinary PR was tested on glibc 2.39 Ubuntu and macOS only, so a RHEL-incompatible change stayed
+  # invisible until publish time. This leg moves that signal to the pull request.
+  #
+  # SCOPE, STATED HONESTLY. It builds and runs the two headline product guarantees (determinism, well-formed
+  # XML) plus the fixture self-run — NOT the 361-gate suite. That is deliberate, not laziness: the suite needs
+  # ripgrep, which is not in the RHEL repositories (it lives in EPEL), and the RHEL-specific signal here is
+  # "does this toolchain compile it and does this userland run it", which the ubuntu legs cannot give and
+  # which these steps do give. The full suite stays on the four ubuntu legs and the two macOS ones.
+  rhel:
+    name: rhel (ubi9, ${{ matrix.flavor }})
+    runs-on: ubuntu-24.04
+    container: registry.access.redhat.com/ubi9/ubi
+    strategy:
+      fail-fast: false
+      matrix:
+        flavor: [Release, plain]
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      # RHEL 9's default gcc is 11, which does not implement C++23 — CMakeLists sets CXX_STANDARD 23 with
+      # STANDARD_REQUIRED ON, so the configure would fail outright. gcc-toolset-N is Red Hat's own supported
+      # answer (a parallel-installable newer GCC in /opt/rh). Probe newest-first rather than pinning one
+      # version: which toolsets a given UBI9 AppStream carries changes over the RHEL 9.x lifecycle, and a
+      # hard pin turns a routine repo refresh into a red leg. Whichever is found is ECHOED, so the log always
+      # records which compiler actually built this leg rather than leaving it to be inferred.
+      - name: Install the RHEL build toolchain
+        run: |
+          set -eu
+          dnf install -y cmake git libxml2 python3 make diffutils findutils
+          found=""
+          for v in 14 13 12; do
+            if dnf install -y "gcc-toolset-$v-gcc-c++"; then
+              found="$v"
+              break
+            fi
+          done
+          [ -n "$found" ] || { echo "no gcc-toolset-{14,13,12} in this UBI9 AppStream — C++23 is not buildable here"; exit 1; }
+          echo "TOOLSET=$found" >> "$GITHUB_ENV"
+
+      - name: Record the toolchain this leg is using
+        run: |
+          echo "gcc-toolset-$TOOLSET"
+          "/opt/rh/gcc-toolset-$TOOLSET/root/usr/bin/g++" --version
+          cmake --version | head -1
+          cat /etc/redhat-release
+
+      # CC/CXX point straight at the toolset's binaries rather than `source .../enable`, because every `run:`
+      # step is its own shell — a sourced environment would not survive to the next step.
+      - name: Configure (portable — no RIPWIRE_NATIVE; empty build type = the plain flavour)
+        env:
+          CC:  /opt/rh/gcc-toolset-${{ env.TOOLSET }}/root/usr/bin/gcc
+          CXX: /opt/rh/gcc-toolset-${{ env.TOOLSET }}/root/usr/bin/g++
+        run: cmake -S . -B build ${{ matrix.flavor == 'Release' && '-DCMAKE_BUILD_TYPE=Release' || '' }}
+
+      - name: Build
+        run: cmake --build build -j
+
+      - name: Self-run on the fixture (it parses and ranks under RHEL userland)
+        run: ./build/ripwire test/fixture --no-cache | head -c 400; echo
+
+      - name: det-gate — 2-run byte-identical diff
+        run: |
+          ./build/ripwire test/fixture --no-cache >/tmp/run_a.xml
+          ./build/ripwire test/fixture --no-cache >/tmp/run_b.xml
+          diff -q /tmp/run_a.xml /tmp/run_b.xml
+
+      - name: G4 — xmllint --noout
+        run: ./build/ripwire test/fixture --no-cache | xmllint --noout -
+
   asan:
     name: asan (${{ matrix.os }})
     strategy:

CLAUDE.md

@@ -34,6 +34,42 @@ which compiles `DEGRADED_PATH_ALERT` out; any gate that asserts a degrade path t
 CI builds *both* flavours on purpose — Release catches optimizer-only bugs, the plain build catches
 degrade paths. If you add a degrade path, the plain run is what proves it.
 
+**Never edit the tree while a build is running, and never background a build you then edit around.**
+This is not a style preference — it silently produces a binary that cannot exist from any single
+commit, and it costs hours to diagnose because every symptom points somewhere else.
+
+Make decides what to recompile by comparing mtimes. Header tracking is correct and complete (the
+compiler writes real depfiles: `build/CMakeFiles/ripwire.dir/src/ingest.cpp.o.d` lists `src/model.h`
+among ~1100 headers, and `pagerank.cpp.o.d` correctly lists none). But mtime ordering is only
+meaningful if the sources hold still. If you edit `src/model.h` — or `git checkout` a branch that
+does — while a build is in flight, that build writes `.o` files whose **mtime is newer than the
+header** but whose **content predates it**. Make then correctly concludes "up to date" and never
+recompiles them again. `cmake --build build -j` reports success, exit 0, no warnings, forever.
+
+Two ways this was hit in one session, both from backgrounding a build:
+
+- A `cmake --build asan` started on one branch and finished after a `git checkout` to another that
+  changed `Symbol`. Half the objects had `sizeof(Symbol)==96`, half `104`. ASan reported a
+  heap-buffer-overflow in `ingest` — a real report, of a fake bug. The tell: the overflowing buffer
+  was **1344 bytes = 14 × 96**, an exact multiple of the *previous* struct size. If a sanitizer
+  report's region size divides evenly by an old `sizeof`, stop debugging and rebuild.
+- `src/quality.h`'s `kIngestParserVerMirror` edited while a clean rebuild ran. The binary kept
+  emitting the old value through repeated successful rebuilds; `qextractionkeycheck` failed with
+  `parserVer=41, expected 12/42` and looked like a missed mirror update. `touch`ing the header fixed
+  it, which is the diagnosis: the object was newer than the source it disagreed with.
+
+If sources may have moved under a build — after any branch switch, or if you are unsure — do not
+trust an incremental rebuild:
+
+```bash
+cmake --build build --clean-first -j          # and the same for asan/ if that tree is in play
+```
+
+`test/g1freshcheck.sh` catches the ordinary stale binary (binary older than source) and is worth
+believing when it fires — it is not noise. It cannot catch this variant, because here the binary is
+*newer* than the source and only its contents are stale. Nothing in CMake can repair a source that
+changed mid-compile; the discipline is the fix.
+
 ## Verify
 
 ```bash

bench/headtohead/r4-2026-08-06/README.md

@@ -77,6 +77,18 @@ is not atomic, and a torn line is a silently lost instance.
   stay in output order and simply never match the universe. Filtering one tool's noise and not
   another's is how a comparison becomes an advertisement.
 
+## One recorded field was rewritten after the fact
+
+`results/r4_cbm.jsonl`'s `project` key. cbm mints its project id from the **absolute** path of the clone
+it indexes, so all 60 recorded values began with the run machine's home directory — which
+`test/ripwirepubliccheck.sh` arm 5 rejects, correctly, and which turned that gate red on all four CI legs.
+The host-specific prefix is now the literal `workdir-`; the `repos-` / `repos_c-` pool marker and the repo
+slug after it are untouched, because those are real provenance (which clone tree an instance ran in).
+
+Nothing derived from the numbers moved: `r4_score.py` never reads `project` — only `r4_worker.py` does,
+and only to hand cbm's own indexer's id back to its search call within a single run. A re-run regenerates
+the field from whatever path it runs at, so it was never reproducible across machines to begin with.
+
 ## Two harness bugs worth remembering
 
 Both were found because a run *failed loudly* rather than scoring a zero.