Skip to content

✨ sphinx-needs: the fingerprint derive kind — a digest of the whole need - #2204

Merged
chrisjsewell merged 11 commits into
masterfrom
claude/sleepy-cerf-ktybzb
Oct 13, 2026
Merged

chrisjsewell merged 11 commits into
masterfrom
claude/sleepy-cerf-ktybzb

Conversation

@chrisjsewell

@chrisjsewell chrisjsewell commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Summary

A new derive kind, fingerprint, gives a field the SHA-256 of the whole need, for change detection:

[needs.fields.fp]
schema = { type = "string" }
derive = { kind = "fingerprint" }

.. spec:: Reviewed against REQ_1
   :id: SPEC_1
   :links: REQ_1[fp == "<REQ_1's fingerprint when it was reviewed>"]

The condition holds while REQ_1 is unchanged, and reports needs.link_condition_failed once it changes;
writing the new value accepts the change. A prefix is enough: REQ_1[fp.startswith('3f9a1c2b7d4e')].

Design

  • The set: id, type, title, status, tags, content (a need nested in another's body is part of the
    parent's content), every field the project declares (needs_fields, needs_extra_options) and every link type,
    links included, none derived. Never: derived fields and link types (hashes, other fingerprints), back links,
    parent_needs, parts, constraints, location, presentation, bookkeeping, or fields an extension or service
    registers (unless the project declares that name itself).
  • The entry rule: a name enters only where its value differs from the field's default, compared as JSON text
    (a literal default, else null when nullable, else [] for a link type or tags, else — a non-nullable field
    without a literal default — it always enters; id/type/title always; a number field's value and default as
    floats, so default = 1 is 1.0). A value written equal to its default is the same as unset, and -0.0 differs
    from 0.0. Declaring a field or link type moves no fingerprint while the needs leave it unset, if its default is
    a literal or there is none; a [[…]] or variant default, and predicates, give the needs a value, which enters as
    a written one would (adding owner with default = "[[copy('title')]]" moves every need's fingerprint).
  • Frozen: no later release changes which names a fingerprint includes, when a name enters, or how the text is
    written: a fingerprint written down stays current until the need, or the configuration of the fields it
    reads, changes.
  • The text: one compact JSON object, keys sorted, written as hash writes values; SHA-256, lowercase hex.
  • Link conditions dropped (targets kept, deduplicated, in the export's order), so pinning an upstream need's
    fingerprint never moves the pinning need's own.
  • exclude: the one optional role, a list of names the fingerprint includes, each once ([] allowed);
    anything else is needs.derive_invalid, as are fields/after on it and exclude on hash.
  • Late, like hash: computed from final values (needextend, variants, [[…]]); a non-late reader is
    needs.derive_scope; no rule runs on imported or external needs.

Tests

  • Seven cases appended to the derived-field conformance corpus (EXPECTED_CASE_COUNT 36 → 43; only additions, the
    36 existing cases byte-for-byte): fingerprint-whole-need, fingerprint-new-field-keeps-digest,
    fingerprint-exclude-and-empty-values, fingerprint-in-a-link-condition, fingerprint-invalid-shapes,
    fingerprint-late-and-scope, fingerprint-defaults-compared-as-text. Their hexes were computed by an independent
    prototype of the byte format (amended for the float rule of a number field's integer default) before the
    implementation existed, and the implementation reproduces them.
  • tests/test_derive_fingerprint.py: every load check with its text and order, the included names, the entry
    rule's edges, registered fields, final values, hidden/nested/imported/external needs, change detection.
  • One order test in tests/test_functions_order.py.

ubCode follows in a twin PR, vendoring the corpus.

Changelog

The 9.0.0 "Unreleased" entry for derived fields gains fingerprint in its list of kinds, a sentence on what it
digests (every declared, non-derived field and link type, links among them) and how a link condition uses it, and
the frozen promise.

uv run poe lint and uv run poe typecheck pass.

When squashing, strip the Co-authored-by trailer GitHub proposes.

claude added 11 commits October 13, 2026 05:30
Six cases for the new value kind fingerprint, appended to the derived-field
conformance corpus: the whole need as one sorted JSON object, declaring more
fields moving nothing, exclude and empty values, a fingerprint in a link
condition, the invalid shapes and the late stage. The hexes are computed by an
independent prototype of the byte format; the kind does not exist yet, so the
six cases fail. EXPECTED_CASE_COUNT 36 -> 42.
A value kind that digests the whole need: the SHA-256 of one compact JSON
object with sorted keys, holding id, type, title, status, tags, content, every
field the project declares and every link type, neither derived (parent_needs
aside). A name enters only where the need's value differs from the field's
default, compared as JSON text; a link list is written as hash writes it, the
conditions dropped. One optional role, exclude, leaves names out; each must
be a name the fingerprint includes, once. The rule runs in the late step.

The six corpus cases pass, with the hexes the prototype computed. The
unknown-kind message lists the new kind.
A read of a late field is reported as computed in the late step, by a rule of
the kind hash or with after = "derived"; fingerprint is the second late kind.
How the rule is read and refused (each load check, in order), the names it
includes, the entry rule's edges (a value written equal to its default, empty
values against null, -0.0 against 0.0, a number field's integer default, a
default that is a dynamic function, a link type compared normalised), fields
added or registered by an extension moving nothing, final values after a
variant and a needextend, a hidden and a nested need, imported and external
needs, change detection through a link condition, and the order of the reads.
The kinds table, the exclude role, the load checks, what a fingerprint
includes and never includes, the entry rule, the frozen set, and detecting a
changed need with a link condition (in full, or by a prefix); the order of the
reads; the 9.0.0 changelog entry of derived fields.
A default that is a dynamic function is no literal default; on a link type and
on tags the fingerprint is to compare the value against [], their empty value,
rather than enter it always. Red: REQ_1's computed [] enters both names.
A link type or tags whose default is a dynamic function or a variant has no
literal default; its value is compared against [], its empty value, so a need
leaving it empty adds no key. A field that is neither nullable nor has a
literal default still always enters. The docs say so, and say that a number
field's default is compared as a float.
fingerprint-new-field-keeps-digest declares a number field with default = 1,
left unset on three needs and written 1 on REQ_10: a number field's default and
value are compared as floats, so neither enters, and the four hexes are
unchanged. Red without the float comparison (REQ_10 moves).
A seventh fingerprint case, fingerprint-defaults-compared-as-text: -0.0 against
a number field's default 0.0 enters, written -0.0, while 0 written is the
default; a legacy extra option enters where it is set and adds no key where it
is not. Hexes computed first by the prototype, amended for a number field's
float default, then equal from the kind. Red under a value comparison of the
defaults, and with legacy extra options left out of the set.
EXPECTED_CASE_COUNT 42 -> 43.
Declaring a field moves no fingerprint while the needs leave it unset and its
default is a literal or absent; a computed default and predicates give a
value, which enters. A nested need is part of its parent's content. A changed
default moves the needs that wrote the old or the new one. A field an
extension registers enters when the project declares it itself. The frozen
promise covers the names, when a name enters and how the text is written. The
changelog says not derived, links among them. A comment at the link types says
why every one is the project's today.
@github-actions github-actions Bot added the pkg: sphinx-needs The sphinx-needs distribution (packages/sphinx-needs): its code, tests and docs label Oct 13, 2026
@codecov

codecov Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.69%. Comparing base (8ac3974) to head (9997e39).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2204      +/-   ##
==========================================
+ Coverage   93.63%   93.69%   +0.06%     
==========================================
  Files         141      141              
  Lines       21965    22060      +95     
==========================================
+ Hits        20566    20670     +104     
+ Misses       1399     1390       -9     
Flag Coverage Δ
ai-index 100.00% <ø> (ø)
codelinks 96.22% <ø> (ø)
mounts 94.27% <ø> (ø)
pytests 92.95% <100.00%> (+0.11%) ⬆️
reports 96.23% <ø> (ø)
ub-test-reports 90.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@chrisjsewell
chrisjsewell merged commit 1b2d3f8 into master Oct 13, 2026
43 of 44 checks passed
@chrisjsewell
chrisjsewell deleted the claude/sleepy-cerf-ktybzb branch October 13, 2026 07:25
chrisjsewell added a commit that referenced this pull request Oct 14, 2026
Six members of the workspace released together, in one pull request and
three tag waves: sphinx-needs 9.0.0 (the Sphinx half of the
derived-values lockstep with ubCode's next release) and the five
packages around it. Everything mechanical is `uv run poe bump`'s, one
commit per member; the prose, the gap bullets and the version-marker
sweep are by hand, each claim measured against the tree.

### What this releases

| member | from → to | level | why |
|---|---|---|---|
| sphinx-ai-index | 0.1.0 → **1.0.0** | major | the `ai_docs_index.json`
it writes has been the format's `"1.0"` since the first release, and
ubCode reads it in production — that is the stability 1.0 declares
(#2034); this is also its first release from this repository, on the
workspace's Python 3.11 / Sphinx 7.4–9 / docutils 0.21 floors |
| sphinx-mounts | 0.2.0 → **0.3.0** | minor (0.x) | variant data from
`[variants]` through ub-project, with new refusals and two new
`-W`-failing warnings; Python 3.11 again; docutils 0.21; first release
from this repository |
| sphinx-needs | 8.5.0 → **9.0.0** | major | computed values (derived
fields, dependency order), `needextend` filters and the needs.json
readers can change need data in certain corner cases; `[[…]]` in content
and `need_func` removed; Python 3.11; `[variants]`; `needstable.js` |
| ub-test-reports | 1.0.0 → **1.1.0** | minor | parser and converter
fixes that add `ReportReadError` and change which exceptions escape and
what a nested-suite report converts to |
| sphinx-codelinks | 1.4.0 → **2.0.0** | major | requires Sphinx-Needs
9; `get_rst` / `[analyse.marked_rst]` refused, the `@rst` blocks
replaced by multi-line needs; first release from this repository |
| sphinx-test-reports | 2.0.0 → **3.0.0** | major | the Sphinx-free half
is ub-test-reports; the import name moves to `sphinx_test_reports`;
requires Sphinx-Needs 9 and ub-test-reports 1.1 |

ub-project stays at 1.1.0: nothing to release.

The six versions, the `__version__` literals, the floors
(`sphinx-ai-index>=1.0.0,<2` in four `docs` extras;
`sphinx-needs>=9.0.0,<10` in codelinks and test-reports;
`ub-test-reports>=1.1.0,<2` in test-reports), the docker fallback
(`sphinx-needs-v9.0.0`), the lock and the six changelog headings are `uv
run poe bump`'s, one commit per member in the planner's order.

### What was added by hand

- **sphinx-needs 9.0.0**: bullets for the changes the entry had not
recorded — ‼️ `[[…]]` in need content is plain text and the `need_func`
role is removed (#1919); ✨ the needtable script (`needstable.js`,
`needs_table_page_size` / `needs_table_page_sizes`, Excel / PDF export
and column reordering removed; #1920); ✨ `[variants]` read through
ub-project, a new dependency, with two new warning types (#1996, #1998);
⬆️ docutils 0.21 (#1977); ⬆️ jsonschema-rs up to 0.58 (#1851, #1980); 👌
deterministic CSS order (#1914); 🔧 the uv workspace, the
`#subdirectory=packages/sphinx-needs` git install and the
`sphinx-needs-v<version>` tags (#1839, #1846), in a new "Internal
changes" section; 🔧 the sdist without `tests/` / `performance/` (#1905);
🐛 needgantt's integer completion (#1870); and the derived-fields bullet
cites #2204 and #2205.
- **sphinx-mounts 0.3.0**: the move bullet (#1850).
- **sphinx-codelinks 2.0.0**: 🐛 `get_current_rev` reads `packed-refs`
(#1871); the ‼️ floor bullet says Sphinx-Needs 9.0 / `>=9.0.0,<10`;
`codelinks write rst` "will be removed in 3.0.0"; the documentation is
served by Read the Docs; the `ingest_need_record` note trimmed.
- **sphinx-test-reports 3.0.0**: the floors it ships
(`sphinx-needs>=9.0.0,<10`, ub-test-reports 1.1.0).
- **ub-test-reports 1.1.0**: the UTF-8 bullet links #2024.
- **The six summary paragraphs**, one per entry (sphinx-ai-index's
replaces its old lead).
- **Pages outside the changelogs** (each states what this release
ships):
- sphinx-codelinks' `codelinks write rst` deprecation
(`WRITE_RST_DEPRECATED` in `cmd.py`, and `docs/components/write.rst`)
now promises removal in 3.0.0, not 2.0.0, and is dated 2.0.0, not an
uncut 1.5.0: 2.0.0 ships with the command.
- `docs/basics/introduction.rst` says `src-trace` renders multi-line
needs.
- sphinx-needs' `docs/directives/choose.rst` says `versionadded::
9.0.0`, not an uncut 8.6.0, and `docs/configuration.rst` says
sphinx-mounts reads `[variants]` "from 0.3.0", not "from its next
release".
- sphinx-test-reports' `docs/install.rst` names the Sphinx-Needs 9 floor
(`>=9.0.0,<10`).
- Comments and agent notes naming the old `>=8.5.0,<9` floor
(`compat-requirements.txt`, two `AGENTS.md`, `src_trace.py`) name 9.0.0.
- A sweep of all 216 version markers (`versionadded` / `versionchanged`
/ `deprecated`) in the six members' docs and sources, against each
member's released versions, found 13 more that named a release never
cut. Each now names the release that shipped it: mounts 0.2.0 ×2,
sphinx-needs 3.0.0 ×7, 4.0.0 ×1 and 7.0.0 ×1, test-reports 1.1.0 ×2.

### Merge and tag order

Merge RIGHT BEFORE tagging. **Re-stamp `:Released:` on the merge day
first** (it is stamped 2026-10-14, the earliest tag day); one line per
file, each in its own format:

| file | line | format |
|---|---|---|
| `packages/sphinx-ai-index/docs/changelog.rst` | l.11 `:Released:
2026-10-14` | ISO `YYYY-MM-DD` |
| `packages/sphinx-mounts/docs/changelog.rst` | l.11 `:Released:
2026-10-14` | ISO |
| `packages/sphinx-needs/docs/changelog.rst` | l.12 `:Released:
14.10.2026` | `DD.MM.YYYY` |
| `packages/ub-test-reports/docs/changelog.rst` | l.11 `:Released:
2026-10-14` | ISO |
| `packages/sphinx-codelinks/docs/changelog.rst` | l.11 `:Released:
14.10.2026` | `DD.MM.YYYY` |
| `packages/sphinx-test-reports/docs/changelog.rst` | l.11 `:Released:
14.10.2026` | `DD.MM.YYYY` |

Then tag the squash commit `<sha>` explicitly (so nothing merged later
can slip into a release), in three waves, each after the previous wave's
`publish` jobs are green — the plan job's check 4 refuses a member whose
workspace dependencies (extras included) are not on PyPI at the version
this tree builds:

```bash
# wave 1 — dependencies already on PyPI (ub-project 1.1.0) or none
git tag sphinx-ai-index-v1.0.0 <sha> && git push origin sphinx-ai-index-v1.0.0
git tag ub-test-reports-v1.1.0 <sha> && git push origin ub-test-reports-v1.1.0
# wave 2 — once sphinx-ai-index 1.0.0 is on PyPI
git tag sphinx-mounts-v0.3.0 <sha> && git push origin sphinx-mounts-v0.3.0
git tag sphinx-needs-v9.0.0 <sha> && git push origin sphinx-needs-v9.0.0
# wave 3 — once sphinx-needs 9.0.0 and ub-test-reports 1.1.0 are on PyPI
git tag sphinx-codelinks-v2.0.0 <sha> && git push origin sphinx-codelinks-v2.0.0
git tag sphinx-test-reports-v3.0.0 <sha> && git push origin sphinx-test-reports-v3.0.0
```

The bare `9.0.0` tag is pushed by `release.yaml`'s last step for
sphinx-needs ("Push the bare companion tag"); it is what moves Read the
Docs' `stable` and keeps `git+…@<version>` pins working. Only if that
step fails: `git tag 9.0.0 sphinx-needs-v9.0.0 && git push origin
refs/tags/9.0.0`. A tag pushed too early fails only its plan job, with
nothing published: re-run it once the dependency is up. After a green
`publish`, a red later job is re-run with **Re-run failed jobs**, never
*Re-run all jobs*.

### Before wave 1

Five of these PyPI projects release through `release.yaml` for the first
time — sphinx-ai-index, sphinx-mounts, sphinx-needs (8.5.0 went out
through the old token-based workflow), sphinx-codelinks and
sphinx-test-reports. Each must list the trusted publisher **owner
`useblocks`, repository `sphinx-needs`, workflow `release.yaml`,
environment `pypi-<dist>`**. The GitHub side is ready (the five
`pypi-<dist>` environments exist, each with the tag policy `<dist>-v*`);
the PyPI side can only be checked by a project owner. A wrong publisher
fails `publish` with nothing uploaded — fix it and re-run from the same
tag — but sphinx-ai-index's is the very first tag, so it would stall the
whole train.

### Rehearsal

Two rehearsals can pass before the merge, from this branch:

```bash
gh workflow run release.yaml --ref release/9.0.0-train -f tag=sphinx-ai-index-v1.0.0
gh workflow run release.yaml --ref release/9.0.0-train -f tag=ub-test-reports-v1.1.0
```

Dispatched from this branch at `9a2dd5ec`: run
[38479550037](https://github.com/useblocks/sphinx-needs/actions/runs/38479550037)
(`sphinx-ai-index-v1.0.0`) and run
[38479553168](https://github.com/useblocks/sphinx-needs/actions/runs/38479553168)
(`ub-test-reports-v1.1.0`). A dispatch runs `plan` and `build` (the
resolver gate and the compat cell) and never `publish`. It must name
this branch: at `master` the tag's version is not the tree's (check 2).
The other four cannot pass yet, because check 4 stays fatal in a
rehearsal: sphinx-mounts and sphinx-needs need sphinx-ai-index 1.0.0 on
PyPI (their `docs` extras), sphinx-codelinks and sphinx-test-reports
need sphinx-needs 9.0.0 (and test-reports ub-test-reports 1.1.0). They
can be rehearsed from `master` once the wave before them has published.

### What goes red and why

- **Docs-Linkcheck** (not a required check): the new
`8.5.0...sphinx-needs-v9.0.0` compare link 404s until the
`sphinx-needs-v9.0.0` tag exists.
- **`uv run poe import-check-codelinks` / `import-check-reports`**: red
until wave 2 publishes — "only sphinx-needs<=8.5.0 is available and
sphinx-codelinks==2.0.0 depends on sphinx-needs>=9.0.0,<10". Run them on
`master` between waves 2 and 3: red then would be a real missing symbol.
The other five import checks are green on this branch.
- **Docker**: the fallback ref becomes `sphinx-needs-v9.0.0`, so the
`docker/**` pull-request and manual-dispatch path, red since the move,
goes green once that tag exists. This PR does not run it (its trigger is
`docker/**` only), and the tag push builds the image from the tag
itself.

### For a human (outside the repository)

- **PyPI trusted publishers**: the five checks under "Before wave 1".
- **codelinks.useblocks.com deep links**: the old GitHub Pages site
served pages at the root (`…/components/analyse.html`). Under Read the
Docs they 404 and live under `/en/latest/`. A Read the Docs page
redirect (old path → `/en/latest/<path>`) on the `sphinx-codelinks`
project would keep bookmarks and inbound links working.
- **Read the Docs `stable` for sphinx-mounts and sphinx-codelinks**
serves the Sphinx-Needs 8.5.0 documentation today (both projects build
from this repository, whose highest PEP 440 tag is sphinx-needs' bare
`8.5.0`). After this train it follows the bare `9.0.0` tag, then every
later sphinx-needs release, and never a mounts- or codelinks-only one.
Consider deactivating `stable`, or making `latest` the default version,
on those two projects.

### ubCode

ubCode's next release names sphinx-needs 9.0.0 as the version its
derived-values reading pairs with. This changelog names no ubCode
version, by design: it keeps the entry's "phase-2 release" wording. The
ubCode release pull request is prepared separately.

Closes #2034
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pkg: sphinx-needs The sphinx-needs distribution (packages/sphinx-needs): its code, tests and docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants