Skip to content

ci(snippets): known-failures baseline and compile-fail tests #17

Closed

Description

@homer6

Follow-up to #5 / #13. tools/check_snippets.py compiles every ```cpp block, but docs/examples/ can't be held to a pass/fail bar yet because most of its blocks fail (349 of 464 on the first run), and BAD examples tagged `nocompile` are never checked at all.

Proposed

  1. Known-failures baseline — a checked-in file (e.g. tools/snippet_baseline.txt) listing the blocks that fail today, keyed by a stable ID (file + heading + index within heading, or a hash of the normalised block). The checker exits non-zero on any new failure and on any baselined block that now passes (so the file only shrinks). That lets docs/examples/ be checked strictly right away.
  2. Compile-fail tests — a fence tag such as ```cpp compile-fail (optionally compile-fail="regex") for BAD examples that are meant not to compile. The checker asserts they fail, and with the expected diagnostic when a regex is given. Examples that are wrong but still compile (UB, dangling) stay nocompile.

Later (not in this issue)

  • Link and run tiers (cpp run, expected-output checks).
  • A self-test for the checker (fixture Markdown with known pass/fail/skip counts).

Acceptance

  • Baseline file and stable IDs; python3 tools/check_snippets.py docs/examples exits 0 against the baseline. (No CI — Actions is disabled; the checker runs locally.)
  • compile-fail tag implemented and documented in CLAUDE.md; at least the guideline BAD examples that are type errors converted from nocompile.
  • Checker summary shows baseline counts (still failing / newly fixed / new failures).

Source: the issue #5 research report (.firecrawl/reports/issue-5-2026-10-31.md, "Rollout" and "Recommendations for this repo"), triaged in #13 (comment). #13 is merged.

Activity

  1. homer6 commented on Oct 31, 2026

    @homer6
    OwnerAuthor

    Delivered in #21.

    What changed

    • tools/check_snippets.py: stable block IDs (<file>#<heading-slug>/<n>), --baseline / --update-baseline (new failures, fixed entries and stale entries fail the run), and a compile-fail[="regex"] fence tag.
    • tools/snippet_baseline/gcc.txt: the 349 docs/examples/ blocks failing on GCC 15.2 today.
    • Guidelines: ES.23 narrowing, the consteval example and the returning-references example are now compile-fail; safety.md corrected — those returns are ill-formed since C++23 (P2266R3) / C++26 (P2748R5).
    • CLAUDE.md documents the tag, IDs and baseline workflow; sources.md change-log row.

    Why

    • Proposals 1 and 2 of this issue (from the issue Compile-check code snippets in CI #5 research report). IDs are heading-based, not hashes, so editing a still-failing snippet doesn't make it "new".
    • Tested: guidelines 25 checked / 0 failed; examples 349 known / 0 new; removed and fake entries are reported as new / stale; the safety example under -std=c++20 is caught as "compiled without errors". P2266R3 / P2748R5 checked on cppreference's compiler-support pages and with GCC 15 in C++20/23/26 modes.
    • No CI: the checker runs locally (Actions is disabled).

    Still open

  2. homer6 commented on Oct 31, 2026

    @homer6
    OwnerAuthor

    Merged in #21 (a4fcc42).

    What changed

    • Nothing since the PR was opened; no rebase was needed.

    Why

    • Reviewed and approved by the repo owner.

    Still open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions