🧪 sphinx-codelinks: snapshot the real extraction output, with marker scope association (moved from sphinx-codelinks#100) by ubmarco · Pull Request #2128 · useblocks/sphinx-needs

ubmarco · 2026-10-07T23:17:38Z

What

packages/sphinx-codelinks: moves useblocks/sphinx-codelinks#100 into the monorepo with the
recipe from #1871 (11 commits, authorship kept), and addresses that PR's review on top.

The declarative extraction fixtures (tests/data/extraction/*.yaml) now snapshot the real
production output: exactly what SourceAnalyse.dump_marked_content() writes, as the flat sorted
list of Metadata.to_dict() items. Before, they held a test-only projection. Warnings are a separate
[warnings].json artefact per case. Three deviations from production are documented in the harness
and the README:

filepath is relative to the case root.
One additive, test-only tagged_scope_type key is added.
The warnings are sorted (production reports them in tree-sitter capture order, which differs between runs).

New scope.yaml (25 cases) covers marker scope association: above a function, inside a body,
no scope, the Python docstring path, YAML's structure finder, and C++ class and struct bodies.

Why

Scope association (find_associated_scope) had no fixture coverage at all. Snapshotting the real
shape also covers filepath, source_map columns and end positions, type and marker, which the
projection dropped.

Review of useblocks/sphinx-codelinks#100, addressed here:

Warnings snapshots were order-unstable. _build_warnings sorts the records again
(file_path, lineno, type, sub_type, msg). Production reports them in tree-sitter capture order,
which differs between two runs of the same input once several extractors are on, so it cannot be
pinned.
SCOPE_NODE_TYPES[cpp] named class_definition, a kind tree-sitter-cpp never emits, so a
marker in a class body had no scope. It now names class_specifier and struct_specifier, pinned by
scope.yaml's cpp_class_body / cpp_struct_body (🐛, changelog entry).
The field-count wording was already fixed on master by 🐛 sphinx-codelinks: malformed one-line markers are reported in the build, at the source line #2092 ("minimum is" / "maximum is").

Also from the review:

test_extraction_is_crlf_insensitive was imperative. It becomes a declarative line_endings key
on the case schema, plus line_endings.yaml (CRLF and a lone CR); such a case also runs with LF
endings and must match.
The extraction corpus is shared with ubCode, which copies the YAML files it also has (today 11 of the 14) byte for byte and takes these snapshots as the expected output of its parity test; the README and the harness header now say so, and useblocks/ubcode#3928 tracks how ubCode's sync script takes up this production shape. The LF pin moves to the root .gitattributes with that reason, and yamlfmt no longer formats the corpus (closes 🔧 Protect the shared extraction corpus like the conformance corpora: LF line endings and no yamlfmt #2133).

Production fixes the fixtures surfaced (physical columns, the need-ref span, docstring tags, a block
comment's */ as an id) are in #2129, stacked on this one, so this stays reviewable as a move.

Review of this PR, addressed on the branch (taken over 2026-10-08 with Marco's go)

Four commits on top of Marco's, no source, corpus or snapshot change:

The README and the harness header state that the corpus and snapshots are shared with ubCode (the first bullet above).
The LF pin for the corpus and snapshots lives in the root .gitattributes, one commented block like the three conformance corpora, with the real reason: ubCode copies the bytes. The tests do not depend on it (YAML normalises line breaks, the harness writes each case's source as exact bytes, syrupy reads snapshots in text mode). The package-level .gitattributes is gone, and yamlfmt's exclude covers the corpus (closes 🔧 Protect the shared extraction corpus like the conformance corpora: LF line endings and no yamlfmt #2133).
_assert_portable_path is removed: its input is relative_to(...).as_posix(), which cannot fail its checks.
The harness's contract comment and the README are cut to the facts; the README lists the warnings sort among the deviations.

Found in that review, pre-existing and not changed here: production orders two marked-content entries on one row by tree-sitter's capture order, which differs between runs (#2150; no case here puts two entries on one row), and the scope walk binds a marker above a typedef struct, a union or a namespace to a struct further down the file now that struct_specifier is a scope kind (#2151).

Tests

The fixture harness and every test_extraction_fixture snapshot are converted to the production
shape. Additions:

scope.yaml (25 cases) and line_endings.yaml (2 cases).
Re-captures after 🐛 sphinx-codelinks: malformed one-line markers are reported in the build, at the source line #2092 (warning wording, the moved too_few_fields case) and 🐛 sphinx-codelinks: report a marked-rst block on the row of its start marker #2021 (marked-rst
rows); every one of those snapshot diffs is a master change.

uv run poe test-codelinks: 689 passed, 0 skipped, stable across reruns (also under xdist and
from the repository root); the same at the takeover tip, with lint, typecheck, check-workspace, the CLI suite under GITHUB_ACTIONS=1 and docs-codelinks (0 warnings) green, and every corpus and snapshot byte unchanged.

Checklist

Every line of this change has been read: Marco's review of 🧪 Capture marker scope association in the declarative extraction fixtures sphinx-codelinks#100, the review of 2026-10-08, and the takeover's line-by-line pass; it was not generated automatically from an issue.
I ran the package's tests (uv run poe test-<package>) and they pass.
Documentation is updated where behaviour or options change.
The package's docs/changelog.rst has an entry under Unreleased.
uv run poe lint and uv run poe typecheck pass.

Closes #2133

find_associated_scope's result (tagged_scope) flowed into production output but was invisible to the declarative extraction-fixture harness, which normalized only needs/need_refs/marked_rst/warnings. Add a "scope" key to both need and need_ref entries: the associated node's type plus the first (stripped) line of its text, or null when there is none. Production still serialises the full node text; the harness only needs enough to prove the same declaration was selected, without dumping whole function bodies into expected JSON.

Recapture all existing declarative extraction-fixture snapshots for the new "scope" key added to need/need_ref entries. Every diff is additive: the "scope" key only. Reviewed each case's captured scope_type/scope_first_line by hand; all match the source's expected association (libclang-engine cases show null, since the analysis explicitly skips scope association for libclang comments).

Add tests/data/extraction/scope.yaml, pinning find_associated_scope's outcome across the languages with scope machinery: cpp, python, csharp, rust, go, bash, yaml. Each language gets a marker directly above a function/method (binds to it), a marker at the end of a function body (falls back to the enclosing function/method via find_enclosing_scope), and a marker with nothing scope-like around it (null). Python additionally covers a docstring marker (find_enclosing_scope path). YAML additionally covers its bespoke find_yaml_associated_structure path: an inline same-row comment, a leading comment bound to a following key-value pair, and one bound to a following sequence item. ts/tsx and jsonc are intentionally not covered here.

The declarative extraction snapshots only captured a reduced projection of the real per-marker payload (id/title/type/links/metadata/line/scope for needs, need_id/line/scope for refs), leaving filepath, remote_url, the full source_map (columns and end positions), the MarkedContentType discriminator, the need-ref marker, and the real tagged_scope text completely untested. Extend _normalize() to surface the fields every Metadata subclass (OneLineNeed/NeedIdRefs/MarkedRst) actually carries: filepath (relative to tmp_path, since production's absolute path differs per run/machine), remote_url (deterministically null here — the harness forces git_remote_url/git_commit_rev to None before run()), the full source_map, a content_type discriminator (named to avoid colliding with a need's own "type" field), and scope as {scope_type, scope_text} with the node's full decoded text exactly as production serializes it (replacing the truncated {scope_type, scope_first_line} shape). marker is now included on need_ref entries. The existing needs decomposition and the line/start_line/end_line keys are kept as-is. Capturing full source_map fidelity surfaced two pre-existing production issues, left unfixed here (out of scope): need_id_refs' extract_marker computes start_column/end_column from the pre-strip() position but the post-strip() length, so every need-ref's column span is shifted by the stripped whitespace; and MarkedRst's source_map for multi-line RST blocks collapses start/end to the same row and uses raw offsets into the flattened multi-line comment text as "columns", which isn't a real position for anything past the first line. No fixture YAML changed; only the harness and the recaptured snapshots.

Update the extraction-fixture README's normalized-contract section to match the new snapshot shape: filepath, remote_url, source_map, content_type, and scope (scope_type + full scope_text) on every entry, plus marker on need_refs. Replace the old "volatile data is omitted" line with a per-field description and a trimmed exclusion list (only the tmp_path prefix and the raw comment/node objects are actually excluded now).

The extraction-fixture snapshot was a projection invented for the test: four buckets (needs/need_refs/marked_rst/warnings), a flattened need payload, links wrapped in a dict, need_id_refs exploded per id, scope wrapped as {scope_type, scope_text}, and a "content_type" rename. Rebuild it from what the tool actually writes. _build_marked_content consumes analyse.all_marked_content — the same flat, (filepath, source_map.start.row)-sorted list SourceAnalyse.dump_marked_content dumps — and calls each entry's own Metadata.to_dict(), so the shape and the ordering come from production, not from re-deriving them out of oneline_needs/need_id_refs/marked_rst separately. tagged_scope is now the plain full-text string (or null) exactly as to_dict() serializes it, need_id_refs carries one need_ids list per marker instead of being exploded, and the "type" discriminator keeps its real name and value. Warnings are a second, independent artefact (production never folds them into the data stream: dump_marked_content and dump_warnings are separate files), so they get their own snapshot via snapshot_extraction(name="warnings") instead of a "warnings" key on the merged object. _build_warnings mirrors AnalyseProjects.update_warnings()/dump_warnings(): a flat list of AnalyseWarning.__dict__ records. Two portability/additive adjustments only: filepath/file_path are rewritten relative to tmp_path (production's absolute path differs per run/machine), and each marked-content entry gets one extra key, tagged_scope_type (the associated node's tree-sitter kind) — not part of production's output, but free to add and useful for cross-language comparison. Recapturing surfaced two pre-existing production quirks, left unfixed (out of scope): a need-id-reference's source_map columns are shifted by the width of the whitespace stripped between the marker and its ids (extract_marker mixes a pre-strip offset with a post-strip length), and a multi-line rst block's source_map collapses start/end to the same row with raw string offsets as columns. No fixture YAML changed; only the harness, conftest's snapshot extension (now accepting list payloads), and the recaptured snapshots.

Update the extraction-fixture README to match the rewritten harness: two independent snapshots per case (marked content, warnings) instead of one merged object; the flat, production-ordered marked-content list with its real per-type payload (need/need_ids+marker/rst) instead of the old four-bucket, flattened/exploded projection; the single tmp_path-relative portability deviation; and the additive, test-only tagged_scope_type field. Also documents the two known production column/row quirks the recaptured snapshots now expose as-is.

…ot paths Path.write_text translates every \n to os.linesep on write, so on Windows an LF-only fixture source (and the compile_commands.json/raw variants _build_preprocessor writes) would land on disk as CRLF even though the committed fixture is LF-only. Write via write_bytes instead so the bytes on disk always match the fixture verbatim, independent of platform. Also turn the existing as_posix() relativisation into an enforced invariant: _assert_portable_path checks (via PurePosixPath and PureWindowsPath, since neither alone recognises every absolute form) that every filepath/file_path landing in a snapshot is relative and slash-normalized, so a non-portable path can never slip in silently.

Drive the same multi-line source through the write -> analyse -> normalize path twice — once as authored (LF) and once with every newline replaced by CRLF, written byte-exact both times — and assert the normalized marked content and warnings are identical. This is the check that would have caught the write_text regression this fixes: a real leak would show up as an embedded \r in the multi-line tagged_scope or a shifted source_map. It also confirms, independent of the write-side fix, that production's own CRLF handling (get_src_strings on the tree-sitter path; libclang_parser.extract_active_comments on the libclang path) normalizes line endings before computing positions/text.

The repository has no .gitattributes, so on a Windows checkout with core.autocrlf=true (the Git-for-Windows default) the fixture YAMLs under tests/data/** and the committed snapshots under tests/__snapshots__/** would arrive as CRLF, independent of anything the test harness does at run time. Both trees hold position-sensitive content (fixture sources feed tree-sitter/libclang directly; snapshots are byte-compared against a second implementation's mirror), so pin both to LF rather than leaving it to a platform default. Scoped to the test-data/snapshot trees rather than a blanket * text=auto, which would risk reformatting unrelated files repo-wide for no benefit here. Verified with `git add --renormalize`: every file in scope is already LF-only, so this adds a guard without changing any content.

Add a "Portability guarantees" section to tests/data/extraction/README.md covering the three things that keep one committed snapshot valid on Linux, macOS and Windows alike: the LF-pinned .gitattributes, byte-exact fixture writing, and relative/slash-normalised snapshot paths (now an enforced invariant, not just a convention).

The ported snapshots were captured against the old repository's last main. Re-capturing them on master changes six snapshot files, and every change traces to one master commit, #2092 (malformed one-line markers are reported in the build, at the source line). - Four `[warnings].json` snapshots (too_many_fields, too_few_fields, missing_square_brackets, not_start_or_end_with_square_brackets): #2092 reworded the one-line parser's messages (`5 given fields, maximum is 4`, `1 given fields, minimum is 2`, and `Field 'links' with ...` now quoted), and the snapshots record the production `msg` verbatim. - `one_character_start_without_separator`: #2092 added the case with a snapshot in the old four-bucket shape. It is now the flat marked-content list (`[]`) plus its own `[warnings].json` (`[]`), because the case yields neither a marker nor a warning. The same commit corrects the ported text that master (and the move into this workspace) made false, in the harness comments, the `snapshot_extraction` docstring and the README. No code changed. - #2092 also removed `AnalyseProjects.update_warnings` and `dump_warnings`, which that text named as the production counterpart of the warnings snapshot. The snapshot is unchanged: it is `SourceAnalyse.oneline_warnings`, which the `src-trace` directive and `codelinks analyse` report from, and the text now says so. The README's example record carries the new message wording. - The README put `.gitattributes` at the repository root; in this workspace it is `packages/sphinx-codelinks/.gitattributes`. No snapshot became unused, and no fixture YAML changed.

The C/C++ scope table named class_definition, which tree-sitter-cpp never emits, so a marker in a class body with no function after it had no tagged_scope. Name class_specifier and struct_specifier instead, and pin both in scope.yaml.

- _build_warnings sorts the records: production's order follows the tree-sitter capture order, which differs between two runs of the same input once several extractors are on, so it cannot be pinned. - test_extraction_is_crlf_insensitive becomes a line_endings knob on the case schema plus line_endings.yaml (CRLF and a lone CR); such a case is also run with LF endings and must produce the same output. - The harness, README and .gitattributes no longer describe the fixtures as mirrored into another implementation.

#2021 reports a marked-rst block on the row of its start marker; the two marked_rst cases pick up its new rows in the production shape.

read-the-docs-community · 2026-10-07T23:18:39Z

Documentation build overview

📚 sphinx-codelinks | 🛠️ Build #35016811 | 📁 Comparing 76a935a against latest (abab51f)

🔍 Preview build

2 files changed

± changelog.html
± src/sphinx_codelinks/analyse/utils.html

codecov · 2026-10-07T23:20:31Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.05%. Comparing base (abab51f) to head (76a935a).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2128      +/-   ##
==========================================
+ Coverage   92.60%   93.05%   +0.45%     
==========================================
  Files         136      138       +2     
  Lines       19447    20313     +866     
==========================================
+ Hits        18008    18903     +895     
+ Misses       1439     1410      -29

Flag	Coverage Δ
ai-index	`100.00% <ø> (ø)`
codelinks	`95.35% <ø> (ø)`
mounts	`94.27% <ø> (ø)`
pytests	`92.46% <ø> (+0.43%)`	⬆️
reports	`93.65% <ø> (+4.82%)`	⬆️
ub-test-reports	`90.20% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

chrisjsewell

Thanks — the production-shape harness is the right call (it pins columns, scope and the warning records, which the normalized shape omitted), the warnings sort is necessary (measured: tree-sitter's capture order gives a different warning order on nearly every run), and the class/struct scope fix is correct. Two things to change before merging, and a few suggestions.

Must 1 — the corpus and snapshots are shared with ubCode; say so, and file the ubCode side. The PR body says the harness, README and .gitattributes "no longer describe the fixtures as mirrored elsewhere", but the mirroring is real: ubCode copies 11 of the tests/data/extraction/*.yaml files byte for byte and takes these snapshots as its expected output (tools/sync_codelinks_expected.py in ubCode, rust/ubc_codelinks/tests/extraction_parity.rs). The README at a6ef2417 never mentions ubCode. Please state, in the README and the harness header, that the YAML corpus is shared with ubCode and that ubCode consumes the snapshots, so nobody edits either casually. The new shape breaks ubCode's next sync in three ways (every expected file becomes a list of production records where ubCode compares a normalized record; the [warnings].json sidecar names parse as a case named x][warnings; the shape pins columns, scope text and ref grouping that ubCode computes differently). The cheap fix lives in ubCode — project each production list and its sidecar back to the normalized shape, which reproduces every current expected file exactly — filed as useblocks/ubcode#3928; the README should point at it.

Must 2 — the .gitattributes rationale. The file says a CRLF byte at a line end "shifts every downstream column". For the YAML corpus that is not so (YAML normalises line breaks inside scalars; #2133 measured the checkout harmless for the tests). The real reason is byte-for-byte sharing and snapshot bytes — say that, or drop the file in favour of #2133, which asks for the same pin at the root plus the yamlfmt exclude.

Suggestions:

Put the pin in the root .gitattributes, one commented line like the three conformance corpora already there, and settle it together with #2133's yamlfmt exclude.
_assert_portable_path cannot fail (its input is relative_to(...).as_posix()); delete it.
The 16-line "Normalization contract" comment in the harness and the README restate each other and the comment points at a snapshot_extraction docstring that does not exist; the house style is one concise paragraph per rule. The README should list the warnings sort as a deviation from production output, as the harness does.
The README's two "known production quirks": one is fixed by #2129, the other (marked-rst rows) goes away when the multi-line needs PR removes the @rst surface; whichever lands second updates the paragraph.

(For the merge order: the multi-line needs branch rebases onto this PR mechanically — measured, 812 passed on the merged tree — so this one goes first.)

…hot shape On top of #2128, each multiline_needs.yaml case's snapshot is the marked content as dumped (its multiline-need records, beside any one-line need or reference) plus the [warnings] sidecar. The 37 cases' 74 files are regenerated in that shape. Projected back to the earlier normalized form (need, markup, open/close/first body lines, warning kinds and lines), they state the same facts as before for all 87 cases. The corpus header's paragraph on the snapshot says so, and that ubCode projects each record to {need, markup, start_line, end_line, content_start_line} for its parity test; the contract above it is unchanged.

…prose cut to the facts The README gains a "Shared with ubCode" section: ubCode copies the YAML files byte for byte and takes the snapshots as the expected output of its parity test, so a change to either is a contract change (useblocks/ubcode#3014, useblocks/ubcode#3928). The harness header says the same. The harness's normalization-contract comment becomes seven lines, without the pointer to a docstring that does not exist; the README states each fact once and lists the warnings sort as a deviation from production output.

…root, and keep yamlfmt off the corpus The package-level `.gitattributes` moves into the root one, as one commented block like the three conformance corpora, with the true rationale: ubCode copies the YAML cases and the snapshots byte for byte, so their bytes must be the same on every platform. A CRLF checkout does not break the tests (YAML normalises line breaks, the harness writes each source as exact bytes, the snapshots are read in text mode), so the old "shifts every downstream column" reason goes. yamlfmt now excludes the corpus, as it excludes the needflow conformance corpus: it once rewrote a block scalar in a case file, which is a content change to a shared contract.

Its only input is `Path.relative_to(root).as_posix()`: relative, or `relative_to` raises first, and slash-separated by construction.

…es, and the third columns quirk ubCode has 11 of the 14 YAML files (`scope.yaml`, `line_endings.yaml` and `marked_rst.yaml` are this repository's only), so the README and the harness header say "the files ubCode also has". The README's production quirks gain the comment-relative columns of a one-line need, which #2129 makes physical. `_write_exact`'s docstring drops the claim that a CRLF source shifts columns (every case extracts the same under CRLF and CR, as `line_endings.yaml` pins), and `snapshot_extraction`'s docstring points at the snapshot contract rather than a normalization one.

…hot shape On top of #2128, each multiline_needs.yaml case's snapshot is the marked content as dumped (its multiline-need records, beside any one-line need or reference) plus the [warnings] sidecar. The 37 cases' 74 files are regenerated in that shape. Projected back to the earlier normalized form (need, markup, open/close/first body lines, warning kinds and lines), they state the same facts as before for all 87 cases. The corpus header's paragraph on the snapshot says so, and that ubCode projects each record to {need, markup, start_line, end_line, content_start_line} for its parity test; the contract above it is unchanged.

@t

…oc field tags in docstrings are not needs (#2129) ## What `packages/sphinx-codelinks`: the production defects that snapshotting the real extraction output (#2128) exposed, fixed. - **Marker positions are physical.** A one-line need's or `@need-ids` reference's `source_map` columns count the characters before it on its line. They used to count from the start of the comment, so an indented marker, a comment after code, or a comment after a non-ASCII character reported a column that was not its column in the file. Each comment records the column, in characters, at which its text starts, on both the tree-sitter and the libclang path; it is added on the text's first row, and later rows start at column 0. A comment's rows are split on line breaks only, so a form feed no longer starts a row. `marked_content.json` changes accordingly. - **A need reference's span starts at its first id**, not at the whitespace after `@need-ids:`, and ends after its last id (`NeedIdRef.start_column` / `end_column`). A marker with no id after it is no reference. - **A block comment's closing `*/` is not marker text**: `/* @need-ids: REQ_1 */` named `*/` as a second id, and `/* @t, IMPL_1, impl */` had the type `impl */`. A one-line need on a comment's last line — the row of a block comment's `*/` or of a docstring's closing quotes — is no longer dropped. - **A Python docstring is read as its content**, without its prefix letters and quotes, so a marker on a one-row docstring, `"""@t, IMPL_1, impl"""`, no longer has the type `impl"""`. - **A field-shaped tag line in a Python docstring is a `docstring_tag` warning, not a need.** A line warns, and yields no need, only when all hold: it is inside a docstring (any bare string statement directly in a module, class or function body; never a `#` comment, never another language); nothing alphanumeric precedes the one-line start sequence; a tag of `PYTHON_DOCSTRING_TAGS` follows it; and the tag has the Epydoc field shape `:|[ \t]+[^ \t:]+:` — `:` directly (`@return: text`), or spaces or tabs, one word without space, tab or `:`, and `:` (`@param a: text`, `@raise Exc: text`, `@param नाम: text`). Every other line goes to the one-line parser as before: `# @todo fix the parser, IMPL_TODO, impl` is a need, so is the docstring line `@todo fix the parser, IMPL_DS, impl`, and so is `@return : text, IMPL_RET, impl`. Doxygen's comma form (`@param a the first, thing`) stays the phantom need it is in C++. The warning says to choose a start sequence the docstrings do not use. "Alphanumeric" here is Python's own notion, which differs from ubCode's on a few characters (combining marks, for one): a documented non-goal. Closes #2154 ## Why Any tool that reads the columns of `marked_content.json` — an editor jumping to a marker, say — was sent to the wrong place for every indented marker and every need reference. In a build the links are line-based, so there only the order in which unknown ids on one line are reported changes. A project with Epydoc-style Python docstrings got phantom needs from its `@param a: x, y` lines with the default `@` start sequence. ## Review of this PR, addressed on the branch (taken over 2026-10-08 with Marco's go) Rebased onto master (after #2128 and #2152): Marco's two commits replayed, the one conflict (`oneline_parser.py`: the warning kinds now live in `models.py`) resolved in place, and the one multi-line corpus case that holds `@need-ids` references re-captured at the new spans. Then: - `docstring_tag` is narrowed to the rule above. As first built it fired on every comment of a Python file and on any listed tag word, so `# @todo fix the parser, IMPL_TODO, impl` — a valid one-line need — and any docstring line opening with a listed tag word, such as `@todo fix the parser, IMPL_DS, impl`, became a warning instead of a need. A field-shaped docstring line such as `@return: x` still warns, by design. The field word is spelled so that this package and ubCode read it the same way in every script: a parameter named in Devanagari is a field word on both sides. The pins came first and were red against each earlier version. - A docstring is scanned as its content, the last line of a comment counts as terminated, a block comment's `*/` is not marker text, a comment's rows are split on `\n` only, and an empty `@need-ids:` is no reference — each the rule ubCode applies, each with a shared case. - The corpus README's two "known production quirks" are fixed by this PR and are gone; what remains is a rule of the corpus: no case puts two entries of one kind on one row (#2150). Its JSON example, and the examples in `docs/components/analyse.rst` (measured with `codelinks analyse`) and `write.rst`, show physical columns. - Nothing pinned the libclang column (setting it to 0 left the suite green); a test now checks that both engines report the same physical columns, with LF, CRLF and CR line endings, including a one-row block-comment need, a need on a `*/` row, a form feed and an empty marker. Found on the way, pre-existing and not changed here: production orders two entries of one kind on one row by tree-sitter's capture order (#2150); the scope walk can bind a marker to a declaration further down the file, past a decorated `def` or a `template <…>` too (#2151); a non-ASCII character before an inline YAML comment loses the comment (#2155). ## ubCode useblocks/ubcode#3014 implements every rule here — physical columns, reference spans from the first id, docstring content, terminated last lines, the `*/` rule, `\n`-only rows, no empty references, and the same `docstring_tag` (same tag list, the same field-shape pattern, docstrings only) — and re-vendors the corpus from master after this merges. Each tool keeps its own notion of "alphanumeric" before the start sequence and of the whitespace between `@need-ids:` ids (they differ on a few characters); the corpus says so and pins neither. ## Tests - `positions.yaml` (11 cases, new): indentation; code, or a non-ASCII character, before the comment; a non-ASCII title and id; trailing whitespace in a block comment; a one-line need and a reference on one row; a one-row block-comment need; a need on a block comment's `*/` row; a form feed inside a line. - `docstrings.yaml` (13 cases, new; its header states the rule): trailing whitespace; a docstring statement holding a second string; field-shaped tags warn (`@param a: the first, thing`, `@return: nothing`, a Devanagari parameter name, an `r"""…"""` one-row docstring); a tag word without the field shape, `@return : text`, `@Todo`, and a `#` comment's tag line (field-shaped or not) are needs; one-row docstrings' need and reference lose the quotes; a need on the closing-quote row. - `need_refs.yaml` gains `every_extractor`, `block_comment`, `space_separated_ids`, `marker_after_text`, `empty_marker_is_no_reference`, `line_comment_ending_in_closer`. - Unit: `docstring_tag()`'s field shape (26 rows, the same table as ubCode's); a field-shaped tag line in a `#` comment goes to the one-line parser; libclang columns equal tree-sitter columns (LF, CRLF, CR). `test_need_id_ref_record` expects the span from the first id. - `uv run poe test-codelinks`: **857 passed, 0 skipped** (the sphinx-7 cell too); `uv run poe docs-codelinks` builds with no warning. ## Checklist - [x] Every line of this change has been read: Marco's commits, the reviews and validations of 2026-10-08, and the takeover's line-by-line pass; it was not generated automatically from an issue. - [x] I ran the package's tests (`uv run poe test-<package>`) and they pass. - [x] Documentation is updated where behaviour or options change. - [x] The package's `docs/changelog.rst` has an entry under *Unreleased*. - [x] `uv run poe lint` and `uv run poe typecheck` pass. --------- Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>

@arnoox

…m sphinx-codelinks#82) (#2166) ## What `packages/sphinx-codelinks`: moves useblocks/sphinx-codelinks#82 (TypeScript and JavaScript comment type, by @arnoox) into the monorepo with the recipe from #1871, and addresses that PR's open review on top. `comment_type = "ts"` analyses `.ts`, `.tsx`, `.mts`, `.cts`, `.js`, `.jsx`, `.mjs` and `.cjs`. The grammar is chosen per file: TypeScript for `.ts`/`.mts`/`.cts`, where a `<T>x` cast is legal, and TSX for every other suffix, which also reads JSX. Legacy one-line `` markers. 4. 👌 A JSDoc tag line is a `jsdoc_tag` warning, not a need. The merge of master: master moved by one commit since this branch was cut, `132607bb` (#2168: `src-trace` renders multi-line needs). It is merged in (`023dd70b`), not rebased, so the eight commits above are unchanged. Two files conflicted. In `docs/changelog.rst` the multi-line needs bullet takes master's text with this branch's language list, and the TypeScript bullet follows master's new `src-trace` bullet. In `tests/data/extraction/multiline_needs.yaml` master's four language cases and this branch's two sit side by side after `bash_hash_run`. Nothing else moved: against master the diff is this branch's own (+2974/−149) plus the takeover's commits below. A `//` run and a JSDoc block — the two shapes the corpus pins — now render through `src-trace`: `test_typescript_blocks_render`. The review round (two reviews of the takeover; every commit after the merge): `export declare …` binds the declaration it holds (`find_next_scope` looks one container deeper, pinned red first by `typescript_export_declare`), and a `declare`-prefixed `const`, `let` or `var` is a scope, as it holds no value (pinned red first by `typescript_declare_const_binds`, which now holds one of each); JSDoc's `@modifies` and TypeScript's JSX pragmas `jsx`, `jsxFrag`, `jsxImportSource` and `jsxRuntime` join the shared tag list (112 names); the grammar choice, the scope predicates, the C++ side of the `jsdoc_tag` gate, `` now drops its `-->` as a block comment drops `*/`. - `@need-ids:` in a JSDoc block also produced a phantom need. Fixed on master by #2049: a reference-marker line is no one-line need. - Generated output was scanned as source. Fixed on that branch by the per-comment-type `exclude` default. - JSDoc tags with a comma became needs: `@param a First, second` gave a need with the id `second`. Now a `jsdoc_tag` warning applies, following the `docstring_tag` rule of #2129. A line of a JSDoc comment whose start sequence is directly followed by a JSDoc tag warns and yields no need (narrowed by the takeover to the lines the one-line parser would have read, under `@`: item 15). A JSDoc comment is one opened by exactly `/**`, under `comment_type = "ts"`. The tag must end at a space, a tab, `{`, `}` or the line's end. The tag list covers JSDoc's block tags, TSDoc's, TypeScript's and the inline tags; matching is case-sensitive. The warning says to choose a start sequence JSDoc does not use. `/* */`, `//` and `/*** */` comments are untouched, and so is C++ Doxygen. ## Tests - `scope.yaml`: 23 TypeScript and JavaScript cases (30 since the takeover's `typescript_export_declare`, `typescript_declare_const_binds`, and one case per scope predicate and for the abstract method signature), one association each, among them every declaration kind above. - `oneline.yaml`: 7 TypeScript-family cases: the three moved ones plus the one-row block, JSX inline, `` and cast cases (8 with the takeover's `jsx_text_oneliner_tsx`: JSX text is no comment, so `.tsx` → TSX is pinned). - `multiline_needs.yaml`: a `//` run and a JSDoc block (4 with the takeover's two `<!--` pins: a run of `<!--` lines holds no block, and a `<!--` row ends a `//` run). - `jsdoc.yaml` (new, 6 cases; 7 with the `@need-ids:` case, the takeover counted; 10 with its `jsdoc_more_tags`, `doxygen_in_cpp_is_no_jsdoc` and `jsdoc_tag_asks_the_one_line_parser`): tag lines warn and the non-tag marker is a need. It also covers where a tag ends, case sensitivity, comments that are no JSDoc, JavaScript, and a start sequence JSDoc does not use. - `test_jsdoc_tag`: the tag shape, the same table as ubCode's. - `test_every_comment_type_is_wired`. - The PR's own unit, discovery and CLI tests. - `test_multiline_needs_render.py::test_typescript_blocks_render` (the takeover's): a `ts` project beside the fixture's `src`, a `//` run with an RST body in a `.ts` file and a JSDoc block with a MyST body in a `.tsx` file, each traced on an RST page. Each need is recorded at its directive with its body's doctype and its links, its local URL is at the block's open line, its body renders the role link and the list, and the build warns nothing. With `get_multiline_needs` off for the project both needs are absent and the test fails. - The takeover's units: `test_ts_grammar_key` (`.ts`, `.mts`, `.cts`, `.d.ts` in any case → TypeScript; the rest → TSX); `test_a_ts_and_a_tsx_file_in_one_run_each_get_their_grammar` (one analysis, both orders); `test_discover_cli_excludes_replace_the_ts_default`; and the default-exclude test asserts what discovery lists, `build/`, `out/`, `coverage/` and a `.d.ts` included. - At the tip: `poe test-codelinks` 1051 passed, 0 skipped, on Sphinx 9.1 and on Sphinx 7.4 / Python 3.11; `poe docs-codelinks` builds with no warning; `poe import-check-codelinks` imports the wheel with `tree-sitter` 0.25.2 and `tree-sitter-typescript` 0.23.2 resolved from PyPI. The corpus is shared with ubCode, whose parity test replays every case above (useblocks/ubcode#2764, re-vendored from this branch the same day; see *ubCode* below). ## ubCode The twin is useblocks/ubcode#2764 (Marco's, taken over the same day and merged onto `main`). It re-vendors the shared corpus (`SOURCE.md`) from this branch while it is open and at this pull request's squash commit once it merges, and its parity test replays the TypeScript cases of `oneline.yaml`, `scope.yaml`, `jsdoc.yaml` and `multiline_needs.yaml`. The `jsdoc_tag` rule (the tag list, the end class, the exactly-`/**` rule, and since the review round the `@`-only and would-have-been-read clauses) is written once, here, in `jsdoc.yaml`'s header, and ubCode implements the same text. From the re-vendor the twin also mirrors the two-level container walk (`export declare …`), the ambient-variable arm of the declaration predicate (a `declare`-prefixed `const`, `let` or `var` is a scope) and the 112-name tag list. `js_html_row_breaks_a_line_run` in `multiline_needs.yaml` was written for the reference by ubCode's review of the twin. ## Checklist - [ ] I wrote this change myself and have read every line of it; it was not generated automatically from an issue. - [x] I ran the package's tests (`uv run poe test-codelinks`) and they pass. - [x] Documentation is updated where behaviour or options change. - [x] The package's `docs/changelog.rst` has an entry under *Unreleased*. - [x] `uv run poe lint` and `uv run poe typecheck` pass. --------- Co-authored-by: Arnaud Riess <arnaud.riesslgmail.com> Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>

ubmarco added 15 commits October 8, 2026 01:13

🧪 sphinx-codelinks: re-capture the marked-rst snapshots after #2021

a6ef241

#2021 reports a marked-rst block on the row of its start marker; the two marked_rst cases pick up its new rows in the production shape.

github-actions Bot added the pkg: sphinx-codelinks Concerns the sphinx-codelinks package (packages/sphinx-codelinks) label Oct 7, 2026

ubmarco mentioned this pull request Oct 7, 2026

🐛 sphinx-codelinks: physical marker columns, need-ref spans, and Epydoc field tags in docstrings are not needs #2129

Merged

5 tasks done

ubmarco mentioned this pull request Oct 7, 2026

🧪 Capture marker scope association in the declarative extraction fixtures useblocks/sphinx-codelinks#100

Closed

ubmarco marked this pull request as ready for review October 8, 2026 10:13

chrisjsewell requested changes Oct 8, 2026

View reviewed changes

chrisjsewell mentioned this pull request Oct 8, 2026

🧪 sphinx-codelinks: a stale extraction snapshot is invisible to --snapshot-warn-unused; add a file-to-case guard #2146

Open

chrisjsewell added 3 commits October 8, 2026 14:40

👌 sphinx-codelinks: drop _assert_portable_path, which cannot fail

9986efa

Its only input is `Path.relative_to(root).as_posix()`: relative, or `relative_to` raises first, and slash-separated by construction.

github-actions Bot added the pkg: workspace The repository as a whole: workflows, CI, release, docker, tooling, the workspace root label Oct 8, 2026

chrisjsewell approved these changes Oct 8, 2026

View reviewed changes

chrisjsewell merged commit 2f40ce2 into master Oct 8, 2026
41 of 43 checks passed

chrisjsewell deleted the test/codelinks-scope-association branch October 8, 2026 13:14

chrisjsewell mentioned this pull request Oct 8, 2026

✨ sphinx-codelinks: multi-line needs in source comments (@need … @endneed); the never-rendered @rst blocks are removed #2152

Merged

ubmarco mentioned this pull request Oct 9, 2026

✨ sphinx-codelinks: TypeScript and JavaScript comment type (moved from sphinx-codelinks#82) #2166

Merged

4 of 5 tasks

ubmarco mentioned this pull request Oct 19, 2026

✨ sphinx-codelinks: VHDL comment type #2077

Draft

Navigation Menu

Repository navigation

🧪 sphinx-codelinks: snapshot the real extraction output, with marker scope association (moved from sphinx-codelinks#100) - #2128

🧪 sphinx-codelinks: snapshot the real extraction output, with marker scope association (moved from sphinx-codelinks#100)#2128
chrisjsewell merged 19 commits into
masterfrom
test/codelinks-scope-association

ubmarco commented Oct 7, 2026 •

edited by chrisjsewell

Loading

Uh oh!

read-the-docs-community Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

chrisjsewell left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ubmarco commented Oct 7, 2026 • edited by chrisjsewell Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Review of this PR, addressed on the branch (taken over 2026-10-08 with Marco's go)

Tests

Checklist

Uh oh!

read-the-docs-community Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

codecov Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chrisjsewell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ubmarco commented Oct 7, 2026 •

edited by chrisjsewell

Loading

read-the-docs-community Bot commented Oct 7, 2026 •

edited

Loading

codecov Bot commented Oct 7, 2026 •

edited

Loading