Skip to content

fix(analyse): anchor newline-terminated one-line markers to comment start (#88) - #89

Merged
chrisjsewell merged 1 commit into
mainfrom
fix/oneline-marker-anchor-88
Jul 28, 2026
Merged

chrisjsewell merged 1 commit into
mainfrom
fix/oneline-marker-anchor-88

Conversation

@ubmarco

@ubmarco ubmarco commented Jul 8, 2026 •

Copy link
Copy Markdown
Member

Problem

Fixes #88.

The one-line marker scanner locates the start sequence with str.find, so it
matches the sequence (default @) anywhere in a comment. When @ appears
inside free-form prose followed by a comma, the text is mis-parsed as
title, id, … and the bogus id (with spaces/specials) later fails the
sphinx-needs id regex with InvalidNeedException:

// See @author, check the example      → id = "check the example"  → InvalidNeedException
// We parse things matching @pattern, then act
// Some prose that mentions @@another-tag(item_1, item_2): more text

Fix

Anchor markers whose end_sequence is the newline (i.e. they run to
end-of-line) to the start of the comment content: only comment decoration
(//, #, *, ///, //!, …) and whitespace may precede the start sequence.
A word character before it means the sequence is prose, and the line is ignored.

Explicitly-bounded markers (e.g. [[ … ]]) are self-delimiting, so they stay
position-independent and may still be embedded after prose — this is what keeps
the recommended bracket-style workaround (and the existing supercharge.cpp
fixture) working.

Tests

Workaround (no upgrade required)

Switch the affected project to a self-delimiting bracket style, which does not
collide with @ in prose:

[codelinks.projects.<name>.analyse.oneline_comment_style]
start_sequence = "[["
end_sequence   = "]]"

…tart (#88)

The one-line marker scanner located the start sequence with ``str.find``,
matching it anywhere in the comment. When the sequence (default ``@``) appeared
inside free-form prose followed by a comma, the text was mis-parsed as
``title, id, ...`` and the bogus id (containing spaces/specials) later failed
the sphinx-needs id regex with ``InvalidNeedException``.

Anchor markers whose ``end_sequence`` is the newline (i.e. they extend to the
end of the line) to the start of the comment content: only comment decoration
(``//``, ``#``, ``*`` ...) and whitespace may precede the start sequence. An
alphanumeric char before it means the sequence is prose and the line is
ignored. Explicitly-bounded markers (e.g. ``[[ ... ]]``) are self-delimiting
and stay position-independent.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.44%. Comparing base (43927d0) to head (929401d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #89      +/-   ##
==========================================
+ Coverage   91.40%   91.44%   +0.04%     
==========================================
  Files          34       34              
  Lines        3012     3027      +15     
  Branches      322      323       +1     
==========================================
+ Hits         2753     2768      +15     
  Misses        160      160              
  Partials       99       99              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ubmarco

ubmarco commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Correct, well-tested, and merge-ready. One backwards-incompatible behavior change worth a changelog note; the rest are minor.

The guard is well-placed — right after start_idx is located, before field parsing — and localized to oneline_parser.py.

Backwards-compat (intended, but please note in the release notes)

For newline-terminated styles (the default @), a marker is now recognized only if nothing alphanumeric precedes it on the comment line. Previously @ matched anywhere and the pre-@ text was discarded, so these previously parsed and are now ignored:

// TODO @My Title, ID
// NOTE: @My Title, ID
// some description @Title, ID

The drop is silent — oneline_parser returns None, which extract_oneline_need treats as "no marker", so no warning is emitted. That is the right call for the prose case in #88 (which otherwise crashed the build with InvalidNeedException), and the documented style always puts the marker first, so real-world impact is low. Still worth stating explicitly: newline-terminated markers must now start the comment content.

Correctness — verified

  • The guard uses the original start_idx (before it is advanced past the start sequence). ✔
  • All standard comment leaders (//, #, *, ///, //!, /*) are non-alphanumeric, so decoration-prefixed markers still parse — covered by the new tests. ✔
  • Bracket style is unaffected: end_sequence != "\n", so the guard is skipped and [[ … ]] stays position-independent (covered). ✔
  • Empty / whitespace-only prefix pass; no false negatives for the standard style.
  • Pure function, single O(n) scan — no perf or durability concern.

Minor

  • The comment "Explicitly-bounded markers … are self-delimiting … exempt" is keyed in code on end_sequence == newline, not on bracket-ness. A hypothetical start="[[", end="\n" config would still be anchored. Consider rewording to "markers not terminated by newline are exempt."
  • Latent: if a language with an alphanumeric comment leader is ever supported (e.g. Batch REM), REM @T, ID would be wrongly rejected by the alnum predicate. Not reachable today.

No outstanding review comments to address.

@chrisjsewell
chrisjsewell merged commit a089070 into main Jul 28, 2026
13 checks passed
@chrisjsewell
chrisjsewell deleted the fix/oneline-marker-anchor-88 branch July 28, 2026 07:58
ubmarco added a commit that referenced this pull request Jul 30, 2026
Follow-up to #97. Adds the traceability artifacts for the substantial
features that landed since 1.3.0 without any.

## Audit

Every merged PR in the `1.3.0` → `1.4.0` window, checked against
`features.rst` and the `@` markers in `src/`:

| PR | Substantial feature? | Traceability |
|---|---|---|
| #92 Bash language support | yes | ✅ `FE_BASH` + `FAULT_BASH_1/2` +
`IMPL_BASH_1/2/3` + added to `IMPL_LANG_1` |
| #85 Preprocessor-aware C/C++ engine (libclang) | yes | ❌ **none** — no
feature need, no markers |
| #89 Marker anchoring fix | no (fix to existing traced code,
`IMPL_OLP_1`) | n/a |
| #83 Declarative fixture tests | no (test infra, no `src/` change) |
n/a |
| #84 / #86 Dependency caps | no | n/a |

So the libclang engine was the one gap: 4 new modules and 353 lines in
`analyse/preproc/`, plus wiring in `analyse.py` / `config.py` / `cmd.py`
/ `src_trace.py`, with zero traceability.

## Changes

**`docs/source/components/features.rst`** — new `FE_PREPROC` feature
need, placed after the language-support block, with three `fault`
children:

- `FAULT_PREPROC_1` — objects in inactive branches are extracted (false
positives, the defect the engine exists to prevent)
- `FAULT_PREPROC_2` — objects in active branches are dropped (false
negatives)
- `FAULT_PREPROC_3` — flags resolved from the wrong compilation database
entry, so the wrong branches evaluate as active

**Six `@` markers**, one per capability listed in the need, so the
mapping is 1:1 and reviewable:

| Marker | Site | Capability |
|---|---|---|
| `IMPL_PREPROC_1` | `analyse.py::create_src_objects_libclang` | engine
entry / opt-in |
| `IMPL_PREPROC_2` |
`preproc/libclang_parser.py::extract_active_comments` | drop
inactive-branch comments |
| `IMPL_PREPROC_3` | `preproc/compile_db.py::find_compile_db` | walk-up
database discovery |
| `IMPL_PREPROC_4` | `preproc/compile_db.py::load_flags_map` | per-file
compiler flags |
| `IMPL_PREPROC_5` | `preproc/compile_db.py::defines_to_args` |
standalone header parsing |
| `IMPL_PREPROC_6` | `preproc/loader.py::load_clang_cindex` |
optional-dependency guard |

Comments only — no logic touched.

## Verification

A green `-nW` build alone would not prove the markers work: a marker
that fails to parse produces no need and no error. So the built output
was checked directly.

```
$ tox -e docs-clean          # sphinx-build -nW --keep-going
build succeeded.

$ grep -o 'id="IMPL_PREPROC_[0-9]"' docs/_build/html/development/traceability.html | sort -u
id="IMPL_PREPROC_1" … id="IMPL_PREPROC_6"        # all six needs materialized

$ grep -o 'href="[^"]*#FE_PREPROC"[^>]*title="IMPL_PREPROC_[0-9]"' … | sort -u
IMPL_PREPROC_1 … IMPL_PREPROC_6                  # all six link to the feature
```

- `tox -e py312-sphinx8-needs5` — 345 passed, 1 skipped, 58 snapshots
passed
- `tox -e mypy` — no issues in 20 source files
- `pre-commit run --files …` — all hooks pass

## Deliberately out of scope

Pre-existing gaps, all predating the 1.3.0 → 1.4.0 window this PR was
scoped to. Flagging rather than fixing, since each is a separate
judgment call:

- **`FE_CSHARP` is an orphan.** It has no incoming `impl` link — it is
missing from the `IMPL_LANG_1` list in `analyse/utils.py:110` even
though `init_tree_sitter` handles `CommentType.cs` (utils.py:122). C# is
the only supported language whose implementation is untraced. One-word
fix if you want it here.
- **No feature need for the `src-trace` directive, nor for source-link
URL generation** (`local-url` / `remote-url`), while the CLI has three
(`FE_CLI_*`). This asymmetry may be intentional — capability-level needs
covering *what* is traced regardless of interface — so changing it is a
feature-model decision, not a gap fix.
- **The test-needs layer is empty.** `docs/ubproject.toml` defines a
`test` type and `development/traceability.rst` runs a `src-trace` over
the `tests` project, but `tests/` carries no `@` markers, so zero
`TEST_` needs exist.

## Note on sequencing

No changelog entry here, deliberately: this PR is based on `main`, where
the `Unreleased` heading still holds the Bash entry that #97 converts
into the dated `1.4.0` section. Editing it would conflict with the open
release PR. Since these artifacts document a feature already listed in
the 1.4.0 notes, either fold a line into #97 before merging it, or let
the next release's notes pick this up — your call on the order.
ubmarco added a commit that referenced this pull request Jul 30, 2026
Bump the version to 1.4.0 and finalize the changelog: convert the
`Unreleased` heading into a dated 1.4.0 section and document the merged
PRs that did not add an entry (#83, #84, #85, #86, #89), plus the
traceability artifacts from #98, which is part of this release.
ubmarco added a commit that referenced this pull request Jul 30, 2026
Bump the version to 1.4.0 and finalize the changelog: convert the
`Unreleased` heading into a dated 1.4.0 section and document the merged
PRs that did not add an entry (#83, #84, #85, #86, #89), plus the
traceability artifacts from #98, which is part of this release.
patdhlk pushed a commit that referenced this pull request Jul 30, 2026
Prepares the **1.4.0** release.

## Release changes

- ⬆️ Bump version `1.3.0` → `1.4.0` in `pyproject.toml`.
- 📝 Finalize the changelog: convert the `Unreleased` heading into a
dated `1.4.0` section (released 30.07.2026). Only 1 of the 6 PRs merged
since 1.3.0 (#92) had added a changelog entry, so this also documents
the 5 that were missing one, plus the traceability artifacts from #98.
- 🗺️ No roadmap changes — none of the merged PRs completed a listed
roadmap item.

Minor bump: two new user-facing features, no removals or config-breaking
changes.

## Changelog for 1.4.0

**New and Improved**

- ✨ Added Bash language support for the `analyse` module (#92)
- ✨ Added an opt-in preprocessor-aware C/C++ extraction engine, powered
by libclang (#85)
- 📚 Traced the preprocessor-aware C/C++ engine in the feature
documentation (#98)
- 🧪 Added a declarative fixture and snapshot test layer for marker
extraction (#83)

**Fixes**

- 🐛 Anchor newline-terminated one-line markers to the start of the
comment (#89, fixes #88)
- 🐛 Pinned `typer` and `sphinxcontrib-typer` to keep the documentation
build working (#84, #86)

## Behavior change worth calling out

#89 is released as a fix, but it does narrow marker matching: a
newline-terminated one-line marker (default `@…`) is now only recognized
when nothing but comment decoration and whitespace precedes the start
sequence. A project that deliberately embedded such markers *after*
prose on the same line will stop seeing them — the fix is what stops `@`
in prose from producing bogus need IDs and `InvalidNeedException`.
Explicitly bounded markers (e.g. `[[ … ]]`) are unaffected.

## Verification

Rebased onto `main` at #98, so this branch carries the full 1.4.0
content.

- `tox -e py312-sphinx8-needs5` — 345 passed, 1 skipped (58 snapshots
passed).
- `tox -e docs-clean` (`sphinx-build -nW --keep-going`) — build
succeeded, 0 warnings.
- `pre-commit run --files docs/source/development/change_log.rst
pyproject.toml` — all hooks pass.

## Not included

Pre-existing traceability gaps are left for a follow-up, since all
predate 1.3.0: `FE_CSHARP` has no incoming `impl` link; the five
enum-site markers carrying a trailing `;` (`IMPL_BASH_1`, `IMPL_GO_1`,
`IMPL_RUST_1`, `IMPL_RUST_2`, `IMPL_JSONC_1`) are inert and produce no
needs, so file-extension discovery is untraced for every language; and
`IMPL_C_1` / `IMPL_PY_1` / `IMPL_GO_2` are anchored on
`COMMENT_MARKERS`, which has no readers. See the discussion in #98.

## After merge

- Tag `1.4.0` on the merge commit (matching the existing `1.0.0`–`1.3.0`
tags). The `Release` workflow triggers on that tag pattern and publishes
to PyPI.
pull Bot pushed a commit to twodrops/sphinx-needs that referenced this pull request Sep 6, 2026
…tart (useblocks/sphinx-codelinks#88) (useblocks/sphinx-codelinks#89)

## Problem

Fixes useblocks/sphinx-codelinks#88.

The one-line marker scanner locates the start sequence with `str.find`,
so it
matches the sequence (default `@`) **anywhere** in a comment. When `@`
appears
inside free-form prose followed by a comma, the text is mis-parsed as
`title, id, …` and the bogus id (with spaces/specials) later fails the
sphinx-needs id regex with `InvalidNeedException`:

```
// See @author, check the example      → id = "check the example"  → InvalidNeedException
// We parse things matching @pattern, then act
// Some prose that mentions @@another-tag(item_1, item_2): more text
```

## Fix

Anchor markers whose `end_sequence` is the newline (i.e. they run to
end-of-line) to the **start of the comment content**: only comment
decoration
(`//`, `#`, `*`, `///`, `//!`, …) and whitespace may precede the start
sequence.
A word character before it means the sequence is prose, and the line is
ignored.

Explicitly-bounded markers (e.g. `[[ … ]]`) are self-delimiting, so they
stay
position-independent and may still be embedded after prose — this is
what keeps
the recommended bracket-style workaround (and the existing
`supercharge.cpp`
fixture) working.

## Tests

- New: the three issue-useblocks/sphinx-codelinks#88 prose lines are ignored; decoration-prefixed
markers
(line-comment, indented, block `*`, `///`, `//!`) still parse; a bounded
  `[[ … ]]` marker embedded after prose still parses.
- Full suite green (271 passed), mypy clean.

## Workaround (no upgrade required)

Switch the affected project to a self-delimiting bracket style, which
does not
collide with `@` in prose:

```toml
[codelinks.projects.<name>.analyse.oneline_comment_style]
start_sequence = "[["
end_sequence   = "]]"
```
pull Bot pushed a commit to twodrops/sphinx-needs that referenced this pull request Sep 6, 2026
…cks/sphinx-codelinks#98)

Follow-up to useblocks/sphinx-codelinks#97. Adds the traceability artifacts for the substantial
features that landed since 1.3.0 without any.

## Audit

Every merged PR in the `1.3.0` → `1.4.0` window, checked against
`features.rst` and the `@` markers in `src/`:

| PR | Substantial feature? | Traceability |
|---|---|---|
| useblocks/sphinx-codelinks#92 Bash language support | yes | ✅ `FE_BASH` + `FAULT_BASH_1/2` +
`IMPL_BASH_1/2/3` + added to `IMPL_LANG_1` |
| useblocks/sphinx-codelinks#85 Preprocessor-aware C/C++ engine (libclang) | yes | ❌ **none** — no
feature need, no markers |
| useblocks/sphinx-codelinks#89 Marker anchoring fix | no (fix to existing traced code,
`IMPL_OLP_1`) | n/a |
| useblocks/sphinx-codelinks#83 Declarative fixture tests | no (test infra, no `src/` change) |
n/a |
| useblocks/sphinx-codelinks#84 / useblocks/sphinx-codelinks#86 Dependency caps | no | n/a |

So the libclang engine was the one gap: 4 new modules and 353 lines in
`analyse/preproc/`, plus wiring in `analyse.py` / `config.py` / `cmd.py`
/ `src_trace.py`, with zero traceability.

## Changes

**`docs/source/components/features.rst`** — new `FE_PREPROC` feature
need, placed after the language-support block, with three `fault`
children:

- `FAULT_PREPROC_1` — objects in inactive branches are extracted (false
positives, the defect the engine exists to prevent)
- `FAULT_PREPROC_2` — objects in active branches are dropped (false
negatives)
- `FAULT_PREPROC_3` — flags resolved from the wrong compilation database
entry, so the wrong branches evaluate as active

**Six `@` markers**, one per capability listed in the need, so the
mapping is 1:1 and reviewable:

| Marker | Site | Capability |
|---|---|---|
| `IMPL_PREPROC_1` | `analyse.py::create_src_objects_libclang` | engine
entry / opt-in |
| `IMPL_PREPROC_2` |
`preproc/libclang_parser.py::extract_active_comments` | drop
inactive-branch comments |
| `IMPL_PREPROC_3` | `preproc/compile_db.py::find_compile_db` | walk-up
database discovery |
| `IMPL_PREPROC_4` | `preproc/compile_db.py::load_flags_map` | per-file
compiler flags |
| `IMPL_PREPROC_5` | `preproc/compile_db.py::defines_to_args` |
standalone header parsing |
| `IMPL_PREPROC_6` | `preproc/loader.py::load_clang_cindex` |
optional-dependency guard |

Comments only — no logic touched.

## Verification

A green `-nW` build alone would not prove the markers work: a marker
that fails to parse produces no need and no error. So the built output
was checked directly.

```
$ tox -e docs-clean          # sphinx-build -nW --keep-going
build succeeded.

$ grep -o 'id="IMPL_PREPROC_[0-9]"' docs/_build/html/development/traceability.html | sort -u
id="IMPL_PREPROC_1" … id="IMPL_PREPROC_6"        # all six needs materialized

$ grep -o 'href="[^"]*#FE_PREPROC"[^>]*title="IMPL_PREPROC_[0-9]"' … | sort -u
IMPL_PREPROC_1 … IMPL_PREPROC_6                  # all six link to the feature
```

- `tox -e py312-sphinx8-needs5` — 345 passed, 1 skipped, 58 snapshots
passed
- `tox -e mypy` — no issues in 20 source files
- `pre-commit run --files …` — all hooks pass

## Deliberately out of scope

Pre-existing gaps, all predating the 1.3.0 → 1.4.0 window this PR was
scoped to. Flagging rather than fixing, since each is a separate
judgment call:

- **`FE_CSHARP` is an orphan.** It has no incoming `impl` link — it is
missing from the `IMPL_LANG_1` list in `analyse/utils.py:110` even
though `init_tree_sitter` handles `CommentType.cs` (utils.py:122). C# is
the only supported language whose implementation is untraced. One-word
fix if you want it here.
- **No feature need for the `src-trace` directive, nor for source-link
URL generation** (`local-url` / `remote-url`), while the CLI has three
(`FE_CLI_*`). This asymmetry may be intentional — capability-level needs
covering *what* is traced regardless of interface — so changing it is a
feature-model decision, not a gap fix.
- **The test-needs layer is empty.** `docs/ubproject.toml` defines a
`test` type and `development/traceability.rst` runs a `src-trace` over
the `tests` project, but `tests/` carries no `@` markers, so zero
`TEST_` needs exist.

## Note on sequencing

No changelog entry here, deliberately: this PR is based on `main`, where
the `Unreleased` heading still holds the Bash entry that useblocks/sphinx-codelinks#97 converts
into the dated `1.4.0` section. Editing it would conflict with the open
release PR. Since these artifacts document a feature already listed in
the 1.4.0 notes, either fold a line into useblocks/sphinx-codelinks#97 before merging it, or let
the next release's notes pick this up — your call on the order.
pull Bot pushed a commit to twodrops/sphinx-needs that referenced this pull request Sep 6, 2026
Prepares the **1.4.0** release.

## Release changes

- ⬆️ Bump version `1.3.0` → `1.4.0` in `pyproject.toml`.
- 📝 Finalize the changelog: convert the `Unreleased` heading into a
dated `1.4.0` section (released 30.07.2026). Only 1 of the 6 PRs merged
since 1.3.0 (useblocks/sphinx-codelinks#92) had added a changelog entry, so this also documents
the 5 that were missing one, plus the traceability artifacts from useblocks/sphinx-codelinks#98.
- 🗺️ No roadmap changes — none of the merged PRs completed a listed
roadmap item.

Minor bump: two new user-facing features, no removals or config-breaking
changes.

## Changelog for 1.4.0

**New and Improved**

- ✨ Added Bash language support for the `analyse` module (useblocks/sphinx-codelinks#92)
- ✨ Added an opt-in preprocessor-aware C/C++ extraction engine, powered
by libclang (useblocks/sphinx-codelinks#85)
- 📚 Traced the preprocessor-aware C/C++ engine in the feature
documentation (useblocks/sphinx-codelinks#98)
- 🧪 Added a declarative fixture and snapshot test layer for marker
extraction (useblocks/sphinx-codelinks#83)

**Fixes**

- 🐛 Anchor newline-terminated one-line markers to the start of the
comment (useblocks/sphinx-codelinks#89, fixes useblocks/sphinx-codelinks#88)
- 🐛 Pinned `typer` and `sphinxcontrib-typer` to keep the documentation
build working (useblocks/sphinx-codelinks#84, useblocks/sphinx-codelinks#86)

## Behavior change worth calling out

useblocks/sphinx-codelinks#89 is released as a fix, but it does narrow marker matching: a
newline-terminated one-line marker (default `@…`) is now only recognized
when nothing but comment decoration and whitespace precedes the start
sequence. A project that deliberately embedded such markers *after*
prose on the same line will stop seeing them — the fix is what stops `@`
in prose from producing bogus need IDs and `InvalidNeedException`.
Explicitly bounded markers (e.g. `[[ … ]]`) are unaffected.

## Verification

Rebased onto `main` at useblocks/sphinx-codelinks#98, so this branch carries the full 1.4.0
content.

- `tox -e py312-sphinx8-needs5` — 345 passed, 1 skipped (58 snapshots
passed).
- `tox -e docs-clean` (`sphinx-build -nW --keep-going`) — build
succeeded, 0 warnings.
- `pre-commit run --files docs/source/development/change_log.rst
pyproject.toml` — all hooks pass.

## Not included

Pre-existing traceability gaps are left for a follow-up, since all
predate 1.3.0: `FE_CSHARP` has no incoming `impl` link; the five
enum-site markers carrying a trailing `;` (`IMPL_BASH_1`, `IMPL_GO_1`,
`IMPL_RUST_1`, `IMPL_RUST_2`, `IMPL_JSONC_1`) are inert and produce no
needs, so file-extension discovery is untraced for every language; and
`IMPL_C_1` / `IMPL_PY_1` / `IMPL_GO_2` are anchored on
`COMMENT_MARKERS`, which has no readers. See the discussion in useblocks/sphinx-codelinks#98.

## After merge

- Tag `1.4.0` on the merge commit (matching the existing `1.0.0`–`1.3.0`
tags). The `Release` workflow triggers on that tag pattern and publishes
to PyPI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

codelinks: One-line marker scanner triggers on @-mentions anywhere in a comment line, not just at line start

3 participants