[joyful-ii-V-I]

Defect

--pin-census=FILE writes TSV rows (C rows have 9 fields: C mech pre post flags caller_id callee targets line). Ids were written raw. A C++ out-of-line member of a class template whose template-argument list spans source lines has a scope that holds the line break verbatim, for example SmallVec<T, Alloc, SizeType,\n GrowingPolicy, N>. That split one C row into a 6-field line plus a continuation line starting with neither C, S, O nor #, and the symbol's S row broke the same way. It was seen once on a large private C++ corpus. The map was never affected: escapeXml writes the same scope as 
.

Five more spellings of the defect reproduce on the pre-fix binary:

• a TAB inside the argument list (a 10-field row)
• a form feed inside the argument list (a line end to Python's splitlines())
• a backslash line splice
• CRLF source (a raw CR, which Python's text mode also treats as a line end)
• a | inside an id. | is the targets separator, so pipe|dir/far.hpp::far_helper#N read back as two targets.

On this repository the | case is real: Markdown heading symbols such as --token-budget=N[K|M|G] made a |-split read 18 single-target C rows as 2 to 6 targets.

Change

• src/pincensus.h: every id and callee field goes through appendCensusField, a Byteset256 + appendCleanRun escaper with the same shape as escapeXml. A backslash becomes \\, TAB/LF/CR become \t \n \r, and every other C0 byte and | become \xHH. The hex digits come from a constexpr table, not a format call into a char[], so fixedbufsweep has no new buffer to classify. | is written \x7c rather than \| so that a naive split stays exact. Ids are composed raw once per symbol and escaped into their slot, and callee names are escaped into one reused buffer per writer. The resolver's names and scopes are untouched. The columns are unchanged, and an id without those bytes is byte-identical to v2. The header's first line now says pin-census v3, and five new # lines document the escape.
• bench/scip_match_diag.py: parse_census decodes ids, callees and targets after splitting, because this harness opens files by an id's path and compares an id's name with SCIP descriptors.
• bench/scip_pin_precision.py: comment only. It joins ids as opaque keys, and both runs escape a given symbol the same way.

Measured

• A clean git archive a55b118e export of this repository, pre-fix vs fixed binary, --pin-census --no-cache: 99 of 51,821 rows change (18 C, 81 S). 98 of them hold | and 1 holds a backslash, all 99 decode back to their pre-fix bytes exactly, and every other row is byte-identical. All 18 of those C rows had target lists that a |-split misread (13 read as 2 targets, 4 as 3, 1 as 6); after the fix, none do. Misshapen rows on this tree: 0 before and 0 after, since the multi-line template shape does not occur here.
• Gate fixture (6 decided call sites, 16 symbols): before the fix, a reader splitting lines on LF and fields on TAB parsed 2 of 6 C rows and 12 of 16 S ids. After the fix it parses 6 of 6 and 16 of 16.

Gate: test/pincensuscheck.sh arm (L), plus (B) now expects v3

The fixture is generated in $TMP, so a pipe|dir directory, CRLF, TAB and form-feed bytes never enter the tree. The checker uses its own decoder and encoder for the documented rule. Arm (L) asserts:

• Shape: every non-comment line is a C/S/O row with its full field count. The same check also runs over the (B) and (G) censuses.
• Presence guards: the map's sc= spans each awkward byte, and p="pipe|dir/far.hpp" exists. These read an independent surface.
• Round trip: each of the 5 caller ids decodes to its exact source bytes, re-encodes byte-identically, appears verbatim as an S row id, and holds no raw separator. The form-feed case is the one that exercises the zero-padded \xHH path.
• Targets: the | target splits into one id, which decodes to pipe|dir/far.hpp::far_helper and is an S id.
• Reconciliation: the dispositions reconcile (calls equals the bucket sum, unaccounted=0, bound equals the non-external C rows), and the summary rows=/symbols= equal what a line reader parses.
• Control: un-escaping one \n inside a real C row makes the same shape check report misshapen lines.
• Determinism: two runs produce byte-identical censuses. A pinned verdict count (18) catches a checker that stops early.

Red first: against the pre-fix binary the gate exits 1 with 12 FAIL rows ((B) header, the (L) header, shape, the 5 round trips, the | target, dispositions, summary, control). After the fix: 52 PASS, exit 0. It also passes 52/52 under ./asan/ripwire, and ASan/UBSan report nothing on the escape fixture, test/pincensusfix or test/scipfix --scip. The ASan and dev censuses are byte-identical.

Gates run

• Full suite at 6bf2255e (this lane merged with main at 31e788ce), one uninterrupted run with no timeouts: gates=633 pass=631 skip=2 fail=0 wall=1585.4s jobs=6 tree_writes=0. The only skips are the environmental ones (argvdiffcheck, editchecknotecheck). An asan/ tree was present and freshly rebuilt, so g1freshcheck ran and passed.
• An earlier full run at c9531d40, before the hex-table change, had one real FAIL: fixedbufsweep refused the char hex[8] + formatTo site as an unclassified fixed buffer. The table replaced it with no change to the census bytes, and the next suite at eb438c26 was gates=632 pass=630 skip=2 fail=0.
• After merging harden: git file-system-monitor policy, exclusive temp files, confined edit-plan reads #250 (105666c1, a CHANGELOG-only conflict with no other overlap) at 6a436944, rebuilt and run as separate scripts: pincensuscheck (52 PASS), fixedbufsweep, declinecheck, scipjoincheck, externalvetocheck, stdqualcheck, tsimportprecisecheck, gatecountcheck, manifestcheck, limitstablecheck, gateexitcheck, printffmtparitycheck, versioncheck and xmlwellformed. All rc=0 with 0 FAIL. The full suite was not re-run on this last merge; CI covers it.
• Generators: docs/gatecount_build.py --check reports 620, from main's gates, since this lane adds none. docs/limits_build.py --check matches (213 caps).
• Quality-delta: ./build/ripwire . --quality-delta=$(git merge-base origin/main HEAD)..HEAD gives regressions="0" gating="0". The remaining rows are api-new-surface="2" (the new escaper and its byte set) and stale ack-ledger rows.
• Edit-check: --edit-check on pinCensusIdentities, writePinCensusDecisionRows, writePinCensusOracleRows and writePinCensus reports status="unchanged" incompatible="0".

Pins moved

None. The census is a side file: stdout is unchanged (arm (E) still passes), and no parity manifest, legend pin, kParserVer, qschemetrip hash, gate-count marker or LIMITS row moves.

Noticed, not fixed here

An out-of-line template member such as void Box<T>::grow() gets scope Box<T>, not Box, so --callers=Box::grow resolves to the in-class declaration and reports count="0". That is resolver/extraction work (kParserVer), and a separate session has picked it up.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Updated --pin-census output to format v3.
  • Added reversible escaping for backslashes, tabs, line breaks, carriage returns, control characters, and | characters in IDs and callee fields.
  • Added decoding support for tools that consume census data, including IDs used as paths.
  • Unescaped IDs remain unchanged.

• Bug Fixes

  • Census records now remain correctly structured when source data contains separators or control characters.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 684568ce-c283-47e9-af69-79d91a4a04aa

📥 Commits

Reviewing files that changed from the base of the PR and between 105666c and 6a43694.

📒 Files selected for processing (5)

• CHANGELOG.md
• bench/scip_match_diag.py
• bench/scip_pin_precision.py
• src/pincensus.h
• test/pincensuscheck.sh

Included review availability: Your plan provides up to 8 included reviews per hour; 2 remain after this review.

---

📝 Walkthrough

Walkthrough

Changes

Pin census v3 field escaping

Layer / File(s)	Summary
Field escape contract src/pincensus.h	Defines v3 escaping for backslashes, separators, control bytes, and `
Census field emission src/pincensus.h	Escapes identities and callee names before writing C and O rows.
Census reader integration bench/scip_match_diag.py, bench/scip_pin_precision.py	Decodes escaped fields in diagnostic parsing. Documents opaque-key joins in precision parsing.
Validation and format documentation test/pincensuscheck.sh, CHANGELOG.md	Adds v3 fixtures and checks for field shape, decoding, round trips, summaries, mutation detection, and deterministic output. Documents the format.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 6a436

The v3 escaping change has no identified merge-blocking risk and is ready to merge after normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 4 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main pin-census defect: row splitting caused by line breaks, TAB characters, and pipe characters inside identifiers. It is concise and directly related to the primary …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 4 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch lane/pin-census-multiline-id

---

_{Comment @coderabbitai help to get the list of available commands.}