You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci(snippets): known-failures baseline and compile-fail tests #17
Follow-up to #5 / #13. tools/check_snippets.py compiles every ```cpp block, but docs/examples/ can't be held to a pass/fail bar yet because most of its blocks fail (349 of 464 on the first run), and BAD examples tagged `nocompile` are never checked at all.
Proposed
Known-failures baseline — a checked-in file (e.g. tools/snippet_baseline.txt) listing the blocks that fail today, keyed by a stable ID (file + heading + index within heading, or a hash of the normalised block). The checker exits non-zero on any new failure and on any baselined block that now passes (so the file only shrinks). That lets docs/examples/ be checked strictly right away.
Compile-fail tests — a fence tag such as ```cpp compile-fail (optionally compile-fail="regex") for BAD examples that are meant not to compile. The checker asserts they fail, and with the expected diagnostic when a regex is given. Examples that are wrong but still compile (UB, dangling) stay nocompile.
Later (not in this issue)
Link and run tiers (cpp run, expected-output checks).
A self-test for the checker (fixture Markdown with known pass/fail/skip counts).
Acceptance
Baseline file and stable IDs; python3 tools/check_snippets.py docs/examples exits 0 against the baseline. (No CI — Actions is disabled; the checker runs locally.)
compile-fail tag implemented and documented in CLAUDE.md; at least the guideline BAD examples that are type errors converted from nocompile.
Source: the issue #5 research report (.firecrawl/reports/issue-5-2026-10-31.md, "Rollout" and "Recommendations for this repo"), triaged in #13 (comment). #13 is merged.
tools/check_snippets.py: stable block IDs (<file>#<heading-slug>/<n>), --baseline / --update-baseline (new failures, fixed entries and stale entries fail the run), and a compile-fail[="regex"] fence tag.
tools/snippet_baseline/gcc.txt: the 349 docs/examples/ blocks failing on GCC 15.2 today.
Guidelines: ES.23 narrowing, the consteval example and the returning-references example are now compile-fail; safety.md corrected — those returns are ill-formed since C++23 (P2266R3) / C++26 (P2748R5).
CLAUDE.md documents the tag, IDs and baseline workflow; sources.md change-log row.
Why
Proposals 1 and 2 of this issue (from the issue Compile-check code snippets in CI #5 research report). IDs are heading-based, not hashes, so editing a still-failing snippet doesn't make it "new".
Tested: guidelines 25 checked / 0 failed; examples 349 known / 0 new; removed and fake entries are reported as new / stale; the safety example under -std=c++20 is caught as "compiled without errors". P2266R3 / P2748R5 checked on cppreference's compiler-support pages and with GCC 15 in C++20/23/26 modes.
No CI: the checker runs locally (Actions is disabled).
Still open
Only a GCC baseline; a Clang one needs a recent Clang + libc++ locally.
Activity
homer6 commentedon Oct 31, 2026
Delivered in #21.
What changed
tools/check_snippets.py: stable block IDs (<file>#<heading-slug>/<n>),--baseline/--update-baseline(new failures, fixed entries and stale entries fail the run), and acompile-fail[="regex"]fence tag.tools/snippet_baseline/gcc.txt: the 349docs/examples/blocks failing on GCC 15.2 today.constevalexample and the returning-references example are nowcompile-fail;safety.mdcorrected — those returns are ill-formed since C++23 (P2266R3) / C++26 (P2748R5).CLAUDE.mddocuments the tag, IDs and baseline workflow;sources.mdchange-log row.Why
-std=c++20is caught as "compiled without errors". P2266R3 / P2748R5 checked on cppreference's compiler-support pages and with GCC 15 in C++20/23/26 modes.Still open
homer6 commentedon Oct 31, 2026
Merged in #21 (a4fcc42).
What changed
Why
Still open
tools/snippet_baseline/gcc.txtto empty is tracked under docs: complete the reference — fill gaps against cppreference and fix the root README #20.