How the reference is checked: the snippet checker (does each block compile?), the GoogleTest suite (does the code do what the page says?), the toolchain presets, and the manual-only GitHub Actions platform workflows. The results per platform and per feature are in
platform-support.md(one page per toolchain inplatform-support/) and in the README's platform table.
Verified locally first: every release passes the full test suite and the
snippet checker on GCC 15 and Clang 22 (macOS, arm64) and GCC 16 (Linux,
Docker); that's the primary proof. The badges in the README are our own,
generated by tools/collect_results.py from the report's results (#230): one
per platform (Linux, macOS, Windows, FreeBSD) and one for the local runs, each
"✓ N toolchains" when every toolchain passes, or "✗ n of N failing". They come
from the consented GitHub Actions runs (manual only, with the owner's consent,
before each release) and the local runs, whichever branch the runs were on,
and they're SVG files in platform-support/badges/. The full platform × compiler ×
architecture matrix, with counts, skip reasons, run links and dates, is in
platform-support.md.
The goal is to check every language and library feature the reference documents, and every example, on every supported platform, architecture and compiler, so the results show how far each toolchain, and the industry as a whole, supports modern C++. That's why:
- the platform runs that are published use every test area and every snippet, never a sample;
- a feature a toolchain lacks shows up as a skip with a reason (a
feature-test macro, a
needs=tag, or a version gate naming the compiler bug or paper), never as a deleted test, so the skip reasons inplatform-support.mdread as a support map; - the workflows cover native Linux, macOS and Windows (MSVC, clang-cl, MinGW) and FreeBSD VMs, and grow as new toolchains become available. Emulated platforms (QEMU) are out for now (#207).
Keeping the results current (asking for runs, processing every log,
capturing findings as tests, vendor bugs, re-running what failed and
regenerating the report) follows the
platform-support skill.
Run the checker locally:
python3 tools/check_snippets.py docs/guidelines --cxx g++-15
python3 tools/check_snippets.py docs/examples --cxx g++-15 -q \
--baseline tools/snippet_baseline/gcc.txt
python3 tools/check_snippets.py docs --cxx /opt/homebrew/opt/llvm/bin/clang++ \
--flag=-stdlib=libc++ --flag=-fexperimental-library -q \
--baseline tools/snippet_baseline/clang.txtAll three must pass cleanly. The GCC list of known failures
(tools/snippet_baseline/gcc.txt) is empty and stays that way; the Clang one
(tools/snippet_baseline/clang.txt) lists only libc++ gaps and only shrinks. Fence tags (body,
bad, nocompile, compile-fail, needs=…, run) are explained in CLAUDE.md.
Code that compiles but is wrong, and run blocks (#262). A BAD example that
compiles is tagged bad, so the checker compiles it and it has to keep
compiling. bad warns="regex" also requires a matching -Wall -Wextra
warning. A block with a main can be tagged run: the checker builds it with
-fsanitize=address,undefined and runs it, and it must exit clean, or, for
bad run, a sanitizer must catch it. The summary has a line such as
364 bad (compile but wrong; 0 must warn); 84 run blocks: 84 run under ASan+UBSan.
Runs need the sanitizer runtimes (Clang 22 locally, GCC on Linux); elsewhere
they're reported as skipped with the reason, and --no-run skips them.
Other toolchains: --toolchain msvc drives cl /Zs (syntax only, the prelude
via /FI) and --toolchain clang-cl drives clang-cl /Zs, both from a VS
Developer shell; each has its own baseline (tools/snippet_baseline/msvc.txt,
clang-cl.txt). Module blocks are checked with GCC and Clang only for now; with
the cl-style toolchains they skip with that reason. --json FILE also writes
the results (platform, compiler, counts, failures, skips with reasons) as JSON.
The snippet checker proves the code compiles; the GoogleTest suite under
tests/ checks that it does what the pages say (printed output,
values in comments, thrown exceptions, type claims). Run it locally with CMake
3.25+ (presets) and Ninja:
tools/run_tests.sh # gcc-15: Homebrew g++-15, libstdc++
tools/run_tests.sh clang-22 # Homebrew Clang 22 with libc++
tools/run_tests.sh gcc-16 # GCC 16 in the gcc:16 Docker image
tools/run_tests.sh gcc-15 -L containers # one area (ctest arguments)tools/run_tests.sh <preset> runs cmake --workflow --preset <preset>
(configure → build → test into build/<preset>/) and then lists each skipped
test with its reason, which ctest itself doesn't show. You can also run the
workflow directly: cmake --workflow --preset gcc-15.
-
gcc-15 —
CMAKE_CXX_COMPILER=g++-15. -
clang-22 —
/opt/homebrew/opt/llvm/bin/clang++ -stdlib=libc++, linked with-L/opt/homebrew/opt/llvm/lib/c++ -L/opt/homebrew/opt/llvm/lib/unwind -lunwindplus matching-rpaths, so the binaries use Homebrew's libc++ (whose headers they were compiled against) rather than the older system one in/usr/lib. It also setsCMAKE_CXX_STDLIB_MODULES_JSONto Homebrew'slibc++.modules.json, whichclang++ -print-file-namedoesn't find, so CMake can build thestdmodule. -
gcc-16 — runs inside Docker. The
gcc:16image (arm64-native, Debian trixie) hasg++16.2 and git but no CMake or Ninja, so the script installs them from Debian (CMake 3.31) before running the workflow:docker run --rm -v "$PWD":/w -w /w gcc:16 bash -c \ 'apt-get update -qq && apt-get install -y -qq --no-install-recommends cmake ninja-build && cmake --workflow --preset gcc-16'
The script re-runs itself in Docker only for
gcc-16, and not when it's already in a container (/.dockerenv,/run/.containerenv) orCPPDOCS_NO_DOCKER=1is set (a host whoseg++is GCC 16).
The whole suite also runs under AddressSanitizer and UBSan:
tools/run_tests.sh clang-22-asan # Homebrew Clang 22 + libc++, -fsanitize=address,undefined
tools/run_tests.sh gcc-16-asan # GCC 16 in the gcc:16 image (LeakSanitizer on too)The presets set the CMake cache variable CPPDOCS_SANITIZE (address,undefined,
thread, or address for MSVC/clang-cl's /fsanitize=address), which
tests/CMakeLists.txt applies to every target, gtest and the module
libraries included. UB stops the test that hit it
(-fno-sanitize-recover=all). The test presets set ASAN_OPTIONS and
UBSAN_OPTIONS; tools/ci/run.sh sets the same when CPPDOCS_SANITIZE is in
the environment. The results JSON records the sanitizers
(compiler.sanitize), and the report gives a sanitized job its own toolchain
page (…-asan, …-tsan) and matrix column.
Where a test can't work under a sanitizer it skips with the reason, never
vanishes: under ASan or TSan a huge reserve can't throw bad_alloc,
because the sanitizer's operator new reports the request instead, whatever
its options (#284). Under TSan, a second call_once after the callable
threw deadlocks, so that one test skips too.
ThreadSanitizer runs on Linux against libstdc++ only (the clang-sanitizers
job). Packaged libc++ isn't built with TSan, so its call_once and unwinder
look like races and crash TSan: with Homebrew LLVM on macOS, every test that
throws died in libunwind, and call_once reported a race the standard rules
out. There's no local TSan preset for that reason.
GCC from Homebrew on macOS has no sanitizer runtimes, so the local GCC
sanitizer run uses the gcc:16 image.
The whole suite also runs the way a hardened release ships:
tools/run_tests.sh gcc-15-hardened # libstdc++ assertions, stack protection, PIE
tools/run_tests.sh clang-22-hardened # libc++ fast hardening mode, stack protection, PIE
tools/run_tests.sh gcc-16-hardened # in the gcc:16 image: also _FORTIFY_SOURCE=3 and full RELROCPPDOCS_HARDEN=ON (the *-hardened presets, which also build
RelWithDebInfo) turns on, in tests/CMakeLists.txt:
| What | GCC / Clang | MSVC / clang-cl |
|---|---|---|
| Library hardening, by the library in use | -D_GLIBCXX_ASSERTIONS (libstdc++), -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_FAST (libc++) |
/D_MSVC_STL_HARDENING=1 |
| Stack and initialization | -fstack-protector-strong, -ftrivial-auto-var-init=zero, -fstack-clash-protection (Linux) |
/sdl (cl) |
| Control flow | -fcf-protection=full (x86), -mbranch-protection=standard (arm64) |
/guard:cf, /CETCOMPAT (x64) |
| Fortification | -D_FORTIFY_SOURCE=3 (glibc, optimized builds only) |
— |
| Linking | PIE, -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack (ELF) |
— |
Each flag is probed with check_cxx_compiler_flag / check_linker_flag
first, so a toolchain that lacks one builds without it. The flags actually
used are cached as CPPDOCS_HARDEN_FLAGS, written to the results JSON
(compiler.harden, compiler.harden_flags) and shown on the toolchain page;
a hardened job gets its own results and page (…-hardened), like a sanitized
one. In CI, CPPDOCS_HARDEN=1 in a job's environment does the same (the
CMake option takes its default from it), and the job leaves snippets to the
plain job. The hardened jobs are GCC 16 and Clang 22 with libc++ on Linux,
Homebrew LLVM on macOS, and MSVC on Windows Server 2025. tests/security/hardening_test.cpp has death tests (#306) that
prove the library checks fire: an out-of-range operator[] on vector,
array, span, string and string_view, front() on an empty vector, and
* on an empty optional or an expected holding an error each stop the
program. They run whenever library hardening is active (including libstdc++
15+'s default in unoptimized builds) and skip with the reason otherwise. What each flag does,
and how to check a binary, is on docs/security/hardening.md.
fuzz/ has libFuzzer targets for the security pages' parsers, built with
CPPDOCS_FUZZ=ON and Clang, and run time-boxed by tools/ci/fuzz.sh (and
the fuzz job in linux.yml). Their saved corpus is replayed as ordinary
tests (tests/security/fuzz_corpus_test.cpp) on every toolchain. See
fuzz/README.md.
tools/run_clang_tidy.sh [build-dir] runs clang-tidy with the repo's profile
over tests/security/ and the fuzz/ headers they include. The profile
(.clang-tidy) enables cppcoreguidelines-*, cert-* and bugprone-*, and
turns every finding into an error. It disables two checks, each with its
reason in the file: unchecked operator[], because library hardening checks
it (hardening), and magic numbers.
tests/.clang-tidy also turns off two checks that misread GoogleTest's
macros.
cmake --preset clang-22 # writes compile_commands.json
cmake --build build/clang-22 --target gtest gtest_main
tools/run_clang_tidy.sh build/clang-22 # "no findings", or each findingA finding is fixed, or kept with // NOLINT(<check>): <why> that names the
check and the reason. The clang-tidy job in linux.yml runs the script
with clang-tidy-22.
The presets above are the owner's Mac. These take no paths: the compiler comes
from CXX (or -DCMAKE_CXX_COMPILER=…), and without either they use the
driver name in the table from PATH. They build into build/<preset>/ with
Ninja, and the configure step fails if the compiler isn't the preset's
toolchain (say AppleClang under gcc), so a job can't test the wrong one.
| Preset | Toolchain | Default CXX |
|---|---|---|
gcc |
GCC + libstdc++ (Linux, macOS, FreeBSD) | g++ |
clang |
Clang + the platform's library (libstdc++ on Linux, libc++ on macOS/FreeBSD) | clang++ |
clang-libcxx |
Clang + libc++ (-stdlib=libc++ via CMAKE_CXX_FLAGS_INIT, so CXXFLAGS/LDFLAGS still apply) |
clang++ |
msvc |
MSVC cl + MS STL (Windows) |
cl |
clang-cl |
clang-cl + MS STL (Windows) |
clang-cl |
mingw |
MinGW-w64 GCC + libstdc++ (MSYS2) | g++ |
CXX=g++-14 tools/run_tests.sh gcc
CXX=clang++-20 tools/run_tests.sh clang-libcxx
CXX=/opt/homebrew/opt/llvm/bin/clang++ \
LDFLAGS="-L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++" \
tools/run_tests.sh clang-libcxx # Homebrew LLVM on macOS (import std skips:
# configure first with -DCMAKE_CXX_STDLIB_MODULES_JSON=
# /opt/homebrew/opt/llvm/lib/c++/libc++.modules.json)
tools/run_tests.sh --json build/results-gcc.json gcc # also write JSON resultsExtra link libraries are found at configure time, not hard-coded:
tests/CMakeLists.txt try-links a small program without, then with, each
candidate and links the first that works into every test target (the
configure output says cpp-docs: linking the tests with -l…):
atomic, whenstd::atomic<T>for aTtoo big to be lock-free needs__atomic_is_lock_freeand friends (GCC 13–15, Clang on Linux with libstdc++ or libc++, GCC on FreeBSD, MinGW);stdc++exp(GCC 14+) orstdc++_libbacktrace(GCC 13), when<stacktrace>,<debugging>or, on Windows,std::print's terminal output (vprint_unicode) need libstdc++'s experimental library.
Where FindThreads uses -pthread (FreeBSD, for one), every target is
compiled with it, so the module BMIs match their importers (Clang rejects a
BMI built without it: -Wmodule-file-config-mismatch). Clang without
clang-scan-deps (apt.llvm.org's clang-N without clang-tools-N) skips
the named-module tests with that reason; the Linux workflow installs
clang-tools-N with every Clang (#205), so this only happens elsewhere.
MSVC and clang-cl use Ninja too, so they must run where cl, link and
the Windows SDK are on PATH: a Developer PowerShell / Command Prompt for
VS (or vcvarsall.bat x64; in GitHub Actions, a step that sets up the MSVC
developer environment). From there, tools/run_tests.ps1 msvc (PowerShell) or
tools/run_tests.sh msvc (Git Bash started from that shell) runs the
workflow. The two presets show up in cmake --list-presets only on Windows.
The tests are built with /std:c++latest /permissive- /EHsc /utf-8 /Zc:__cplusplus /bigobj /W4 (plus /Zc:preprocessor for cl), and
GoogleTest with gtest_force_shared_crt so both use the DLL runtime. mingw
runs from an MSYS2 MinGW shell (tools/run_tests.sh mingw).
tools/ctest_to_json.py --build-dir build/<preset> turns any ctest run
(LastTest.log, or a ctest --output-junit file with --junit) into JSON:
platform, compiler and version, preset, counts, failures and skips with their
reasons, and results: every test with its status (passed, failed, skipped).
tools/check_snippets.py --json writes the same shape for snippets, with
every block's ID and status in results. The text output and counts don't
change.
tools/collect_results.py turns those JSON files into
docs/platform-support.md: give it a GitHub
Actions run ID (it fetches the run's results-*.json artifacts with
gh run download) or a folder of results, plus --local <dir> for local
runs; --stdout prints the page instead. It also rewrites the platform
table in the README (the block between
<!-- platforms:start --> and <!-- platforms:end -->; --no-readme leaves
it alone, --check only reports whether either is stale, --self-test runs
its built-in samples). A file that doesn't match the schema fails with the
file and field named. The page and the table change only from runs the owner
consented to.
The report has an entry page and one page per toolchain:
docs/platform-support.md entry page: how to read it, the platform table, the feature
support matrix summary (rows = areas, columns = compilers),
known gaps and the compiler/platform bugs found
docs/platform-support/
README.md the folder's index: every toolchain page with its results,
the layout and how to regenerate (generated)
<slug>.md one per platform × arch × compiler (generated), e.g.
linux-ubuntu-24.04-x86_64-gcc-13.md, local-macos-arm64-gcc-15.md
runs/<run-id>/ archived consented runs: results-*.json and run.json
local/ archived local results: results-*.json and local.json
known-bugs.json compiler and platform bugs found (maintained by hand)
A toolchain page (built by tools/feature_support.py) has a header (OS or
image, compiler and version, standard library (detected by tests/CMakeLists.txt
from the library's own macros and cached as CPPDOCS_STDLIB), architecture and byte order,
run link, date, commit, areas covered, whether its snippets are
report-only), totals, then per area every page, every feature (each
GoogleTest case, labelled by the // "Section": claim comment above its
TEST) and every example section: ✅ / ❌ / ⏭ with the skip reason / — not run
/ ❔ unknown. Rows come from the checkout; a job's cells come from its
results lists, or, for older results without them, are inferred only when
the job's counts match (else ❔). In the entry page's summary matrix, jobs
with the same compiler, version, standard library and identical results
share a column; its header links the toolchain page (or the list of them).
The known-gaps section ends with the bugs from
platform-support/known-bugs.json
(toolchain, summary, upstream bug, repo issue, and the skip reason that
records it).
The results are archived in the repo, so the report can be rebuilt after
GitHub deletes the artifacts: the first time the collector sees a run ID it
saves each job's results-*.json and a run.json (URL, date, commit,
inputs, jobs and conclusions, a digest of the counts; no logs) under
platform-support/runs/<run-id>/ and reads the run
from there afterwards; --local <folder> copies the local results into
platform-support/local/ (with the date and
commit), which later refreshes reuse. Regenerate everything with every
platform's latest consented run in one command:
tools/collect_results.py <linux-run> <macos-run> <windows-msvc-run> \
<windows-gnu-run> <freebsd-run> [--local <folder>](or one all.yml run ID), then commit the entry page, the toolchain pages,
the README and the runs/ and local/ changes. The archive holds only what
the report uses: compact JSON, without job files a newer run superseded or
run folders not named on the command line (git history keeps them), so it
stays at about one full run. The collector removes
toolchain pages it no longer produces, and --check covers them all.
GoogleTest v1.18.0 is fetched at configure time with FetchContent. Tests
mirror the pages: docs/examples/<area>/.../<page>.md is checked by
tests/<area>/.../<page>_test.cpp, and each area folder builds into one
executable (examples_containers_tests, examples_text_processing_tests, …;
examples_tests builds them all). Helpers in tests/support/:
capture.hpp captures what std::cout, printf and std::print write, and
feature.hpp provides CPPDOCS_REQUIRE(__cpp_lib_x), which skips a test with
the missing feature's name when a toolchain doesn't define the macro.
Named modules: tests/language/modules/modules.cmake
builds the module units in tests/language/modules/units/ with
FILE_SET CXX_MODULES (CMake 3.28+ and Ninja) and scans the tests that import
them. import std is used where CMake provides it: CMake 4.3 and 4.4 with the
experimental gate the top-level CMakeLists.txt sets, or 4.5+ without one.
Elsewhere — the gcc-16 image's CMake 3.31, GCC 15 with CMake 4.5 on macOS —
those tests skip and say why.
GitHub Actions workflows here are manual only: workflow_dispatch is the
only trigger, nothing runs on push, pull request or a schedule, and a run
needs the owner's go-ahead. Every workflow is read-only (permissions: contents: read) with actions pinned to commit SHAs;
python3 tools/check_workflows.py checks that. How to run, watch and read a
workflow is in .github/workflows/README.md.
The platform workflows (linux.yml, macos.yml, windows-msvc.yml,
windows-gnu.yml, freebsd.yml) build and run the suite and the
snippet checker with the portable presets on each platform, through the
scripts in tools/ci/, and upload one
results-<platform>-<compiler>.json per job. A release needs a green
consented run of each on dev first, with every failure fixed (the release
checklist is in CLAUDE.md). The local checks above remain the
source of truth.
Each platform workflow and all.yml take a jobs input, a regex for the job
names to start (#376), so a fix-up run starts only the jobs that failed;
tools/ci/plan.py previews a pattern, and the jobs
are listed in .github/ci/. Details:
Starting only some jobs.