Skip to content

fix(#9751): leave the node-locally built vector graph, its side files and last-tx-id.bin out of the HA checksum answers - #9757

Merged
robfrank merged 2 commits into
mainfrom
fix/9751-vector-side-files-ha-checksum
Oct 12, 2026
Merged

robfrank merged 2 commits into
mainfrom
fix/9751-vector-side-files-ha-checksum

Conversation

@robfrank

@robfrank robfrank commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #9751

Summary

An LSM vector index's proximity graph is built by every node for itself, on the inner database instance and without the WAL (#8292, #8905). Its .vecgraph pages, and the .vecpq product quantization trained in the same pass, therefore carry different bytes on two healthy nodes holding the same vectors. The .vecgraphfp manifest and the .vecordmap ordinal map are plain files that no snapshot ships, so a follower that installed a snapshot has none of them until it builds its own graph. last-tx-id.bin is rewritten whenever a node rotates its WAL or closes, so its value depends on that node's own timing. Every one of these files reached GET /api/v1/ha/snapshot/{db}/checksums. The .vecgraph component (registered, so enumerated) also reached POST /api/v1/cluster/verify/{db} on both of its paths. All of them are now declared in NodeLocalFiles, and both collectors read that one declaration. The vectors themselves live in the index's .lsmvecidx pages. Those pages are replicated and still compared.

The issue's open question: is .vecgraph deterministic across nodes?

No. I measured it by building the same 400 vectors into two independent databases, once with NONE and once with PRODUCT quantization, then comparing the files byte for byte. The probe was a scratch test and is not committed.

file identical on both?
.lsmvecidx (vector pages) yes
.bucket, .dict, configuration.json, statistics.json yes
.vecgraph (both indexes) no
.vecpq no
.vecgraphfp, .vecordmap yes when both nodes built from the same data, but present only on a node that built its graph

So, as the issue suspected, the three side files were only the visible part. Without this change the checksum endpoint and the cluster verify report every vector index as divergent once two nodes have built their graphs.

Finding ledger

# Finding (from the issue) Status
1 *.vecgraphfp not shipped but checksummed fixed: declared node-local
2 *.vecordmap not shipped but checksummed fixed: declared node-local
3 *.vecpq not shipped but checksummed fixed: declared node-local
4 Open question: is .vecgraph deterministic? answered: no (measured, table above). Declared node-local, and left out of the cluster verify too
5 last-tx-id.bin rewritten per node on WAL rotation and close fixed: declared node-local. Still shipped (synthesized) by a snapshot, just no longer compared
6 Suggested fix: extend the #9638 guard with a vector index and a TimeSeries type done: the guard now drives two vector indexes (NONE and PRODUCT, since only NONE writes an ordinal map and only PRODUCT writes a PQ) and a compacted TimeSeries type, and asserts each family is on disk before it checks anything

Completeness

Invariant: a file whose bytes legitimately differ between two healthy nodes holding the same data never appears in a cross-node checksum answer.

Sweep commands and the results:

  • grep -rn "isNodeLocalFileName\|NodeLocalFiles\." --include='*.java' . | grep -v /test/: one consumer, SnapshotManager.computeFileChecksums, which serves /checksums.
  • grep -n "isTemporaryFileName" PostVerifyDatabaseHandler.java: the two enumeration paths of the cluster verify (window at line 664, flush-suspension fallback at line 703). Neither used NodeLocalFiles.
  • grep -rn "FILE_EXT\s*=" engine/src/main/java: every engine file extension. bfidx is a replicated component (its javadoc covers this). vcidx and sparseseg are replicated components. ts.sealed is shipped and compared on purpose (PostVerifyDatabaseHandler never checksums .ts.sealed, so a TimeSeries sealed-store divergence is invisible to /verify #7338).
  • The probe above settles which of these differ between independently built nodes.
Entry point Status
/checksums -> SnapshotManager.computeFileChecksums, both branches (window and live) fixed here (NodeLocalFiles.isNodeLocalFileName)
/cluster/verify -> PostVerifyDatabaseHandler.computeLocalChecksums, window path fixed here (isNotCompared)
/cluster/verify, flush-suspension fallback path fixed here (isNotCompared)
.lsmvecidx vector pages argued: replicated content, still compared (asserted by tests)
.bfidx, .vcidx, .sparseseg argued: replicated through the WAL/compaction replication, so identical across nodes

Test plan

  • Issue9751VectorSideFilesChecksumTest (5 tests): the probe with the real writer-built names (including the legacy PQ name), shape-not-substring matching, a node that built its graph vs one that installed a snapshot agree, a real database with both index kinds writes every side file and only the vector pages are checksummed, and the cluster verify drops .vecgraph on both paths
  • Issue9638NodeLocalFilesChecksumTest guard extended with vector indexes and a TimeSeries type
  • Proven to fail without the fix: with the main-code changes reverted, 4 of the 5 new tests fail and so does the extended guard. The one that passes is the "still checksummed" negative test, which is expected to pass either way
  • Related suites green (80 tests): SnapshotManagerTest, Issue6116ChecksumsPointInTimeTest, Issue7338VerifyChecksumsSealedStoresTest, Issue7634VerifyWindowHoldsNoReadLockTest, Issue8018ChecksumTransientFilesTest, Issue8738SealedStoreReplacedAfterT0Test, Issue8942UnwrittenValidationVerdictTest, Issue8019SnapshotShipSkipsCompactionTemporaryTest, Issue7456SnapshotShipConfigurationFromWindowTest, Issue8905VectorBuildHATest, plus both classes above

Coverage

Covered: /api/v1/ha/snapshot/{db}/checksums (both branches) and /api/v1/cluster/verify/{db} (both paths).

Known gaps: #9758. This PR leaves snapshot shipping unchanged: .vecgraph is still shipped, and its manifest still is not. A follower that installs a snapshot therefore has graph pages with no manifest beside them. Its load path judges such a graph only by node count and can reuse it unverified, which is the weak #6106 comparison. That concerns what is shipped, not what is compared, so it is filed separately.

Residual risk

A .vecgraph that is corrupt on one node is no longer flagged by the divergence detector. That is the intended trade: it is a rebuildable cache, the same as the GAV CSR in #9638, and the graph is a cache that can be rebuilt from the compared .lsmvecidx pages. A real divergence in the vectors still shows, because the .lsmvecidx pages are compared.

Adversarial pass

Not run as a separate subagent: this session has no subagent-spawning tool. I reviewed the diff myself against the issue body instead. Findings:

  • Checked whether another engine file family behaves the same way, using grep -rn "FILE_EXT\s*=" engine/src/main/java. .bfidx (its javadoc: a component precisely so that it replicates), .vcidx and .sparseseg are replicated components. Not real.
  • Checked for a /checksums consumer that decides freshness from last-tx-id.bin. The only references are SnapshotHttpHandler itself and the OpenAPI spec (PluginApiSpec). Not real.

Review cycles

cycle head changes bot outcome
1 6d3ee18591 initial fix plus tests LGTM with suggestions: a note that node-local does not mean not shipped, javadoc wrap, import order, constant alignment, check the production predicate on a real listing, confirm a graph without a manifest is rebuilt
2 1c97cc2c54 applied all of those. Answering the last one showed that the original claim ("a follower without a manifest rebuilds anyway") was wrong: the load path reuses such a graph after a count check. Corrected the javadoc and filed #9758 approved in spirit, 4 non-blocking notes, none requiring a change (below)

Deferred / skipped items

Final state

clean-approval

🤖 Generated with Claude Code

… and last-tx-id.bin out of the HA checksum answers

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@mergify

mergify Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@claude

claude Bot commented Oct 11, 2026

Copy link
Copy Markdown

Review

Overall this is a solid, well-scoped fix. Both collectors now read one declaration (NodeLocalFiles), the matching is by extension shape rather than substring, and the tests assert that each file family really is on disk before they check exclusion. The PR description is thorough, including the measured answer to the open question.

Positives

Suggestions

  1. Residual risk worth a follow-up: a corrupt .vecgraph is no longer flagged by the divergence detector. The PR acknowledges this and I agree it is acceptable, since the manifest guards the load path. Please confirm that a graph whose manifest is missing or stale is always rebuilt rather than served. Issue8905VectorBuildHATest seems to cover part of that.
  2. last-tx-id.bin semantics: NodeLocalFiles now means "not comparable" and no longer means "never shipped". Any future consumer that treats isNodeLocalFileName as "skip when shipping" would silently drop it. The Javadoc says this, but a short note on the method itself would make it harder to misuse.
  3. Style nits:
    • The reworded Javadoc line in NodeLocalFiles.isNodeLocalFileName ("The shapes describe the files the engine itself creates; ...") is now well over the 120-column wrap used around it. Please re-wrap it.
    • In Issue9638NodeLocalFilesChecksumTest the new import LocalTimeSeriesType is placed before LocalSchema. Alphabetically it should come after.
    • Issue9751VectorSideFilesChecksumTest has an odd alignment in the constants (private static final int DIMENSIONS, VECTORS =). The extra spaces are leftover.
  4. Test cost: aRealVectorIndexWritesItsSideFilesAndOnlyItsPagesAreChecksummed and the cluster-verify test each build two vector indexes with a graph. That is small (400 x 16 dims), but if the CI timings creep, one shared fixture would halve it. Not blocking.
  5. Test naming: isNodeLocalVectorFile in the test re-implements the production rule using the same extensions. That is fine as an independent oracle, but the assertions would be a little stronger if they also checked that NodeLocalFiles.isVectorGraphFileName agrees on a real directory listing (it is only exercised indirectly now).

No security or performance concerns. The checks are a few endsWith calls per file on a non-hot path. No new dependencies.

LGTM once the nits are addressed.

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e2b443db-2bfe-4bc1-be29-b67ff9822cec


📥 Commits

Reviewing files that changed from the base of the PR and between 123ad9e and 1c97cc2.



📒 Files selected for processing (5)
  • engine/src/main/java/com/arcadedb/engine/NodeLocalFiles.java
  • ha-raft/src/main/java/com/arcadedb/server/ha/raft/PostVerifyDatabaseHandler.java
  • ha-raft/src/main/java/com/arcadedb/server/ha/raft/SnapshotManager.java
  • ha-raft/src/test/java/com/arcadedb/server/ha/raft/Issue9638NodeLocalFilesChecksumTest.java
  • ha-raft/src/test/java/com/arcadedb/server/ha/raft/Issue9751VectorSideFilesChecksumTest.java


Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.




📝 Walkthrough
📝 Walkthrough

Walkthrough

NodeLocalFiles now classifies vector-index artifacts and last-tx-id.bin as node-local. HA checksum collection skips temporary and node-local files. Regression tests cover file creation, checksum results, and cluster verification.

Changes

HA checksum handling

Layer / File(s) Summary
Node-local file classification
engine/src/main/java/com/arcadedb/engine/NodeLocalFiles.java, ha-raft/src/main/java/com/arcadedb/server/ha/raft/SnapshotManager.java
NodeLocalFiles recognizes last-tx-id.bin and vector graph, manifest, ordinal-map, and product-quantization files. Its vector-file predicate requires a nonempty base name before the extension. Documentation describes snapshot handling and clarifies that node-local classification does not itself exclude files from snapshots or backups.
Checksum exclusion paths
ha-raft/src/main/java/com/arcadedb/server/ha/raft/PostVerifyDatabaseHandler.java
Both page-checksum collection paths now skip files that are temporary or classified as node-local.
Checksum regression tests
ha-raft/src/test/java/com/arcadedb/server/ha/raft/Issue9638NodeLocalFilesChecksumTest.java, ha-raft/src/test/java/com/arcadedb/server/ha/raft/Issue9751VectorSideFilesChecksumTest.java
Tests create vector-index and time-series files and verify their file presence and checksum handling. They also check that vector-index pages remain included and exercise cluster verification with page snapshots enabled and disabled.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested reviewers: lvca



Merge Risk: ⚪ Minimal · up to 1c97c

No actionable merge-blocking issue is identified; the change is mergeable after normal checks.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 18 functions across 5 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed The PR addresses the coding requirements in #9751. NodeLocalFiles classifies .vecgraph, .vecgraphfp, .vecordmap, .vecpq, and last-tx-id.bin as node-local. SnapshotManager uses that class…
Out of Scope Changes check Passed The changes stay within #9751. Production changes update node-local file classification and the two checksum collectors. Test changes provide the requested real-database guard and regression coverage.…
Title check Passed The title clearly identifies the main change: excluding node-local vector graph files and last-tx-id.bin from HA checksum results. It is specific and related to the pull request.
Description check Passed The description thoroughly explains the change, motivation, related issue, implementation scope, tests, coverage, risks, and known gaps. It does not use the exact template headings and omits the expli…

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR





🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR



  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codacy-production

codacy-production Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 9 complexity

Metric Results
Complexity 9

View in Codacy

🟢 Coverage 27.27% diff coverage · -5.29% coverage variation

Metric Results
Coverage variation ✅ -5.29% coverage variation
Diff coverage ✅ 27.27% diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (123ad9e) 229612 197489 86.01%
Head commit (1c97cc2) 262238 (+32626) 211667 (+14178) 80.72% (-5.29%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#9757) 11 3 27.27%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

…apshot-install claim (#9758), check the production predicate on a real listing, nits

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Oct 11, 2026

Copy link
Copy Markdown

Review

Solid, well-scoped change. The single declaration in NodeLocalFiles read by both collectors (/checksums and cluster verify) removes the risk of the two drifting apart, and the measured answer to the issue's open question (.vecgraph is not deterministic) is the right basis for the fix.

What I checked

Minor, non-blocking

  1. engine NodeLocalFiles now imports four com.arcadedb.index.vector classes just for FILE_EXT constants. This is fine within the module, but it ties a small utility to the vector package. Keep it in mind if those classes ever move to an optional module.
  2. Excluding .vecgraph from verify means a corrupted or truncated graph file on one node is no longer detected by checksum. This is the correct trade-off, since the graph is a cache rebuildable from the .lsmvecidx pages that are still compared. It may be worth a line in the docs.
  3. Issue9751VectorSideFilesChecksumTest builds real indexes with 400 vectors and a graph build several times. If CI time becomes a concern, consider @Tag("slow") on the real-index tests. I would not add it otherwise.
  4. last-tx-id.bin is now excluded from checksums. Please confirm nothing else consumes /checksums to decide snapshot freshness from that file. I found no such consumer in the diff, but I have not audited callers outside it.

No bugs, security, or performance issues found. Approving in spirit.

@codecov

codecov Bot commented Oct 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 18.18182% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.77%. Comparing base (123ad9e) to head (1c97cc2).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
.../main/java/com/arcadedb/engine/NodeLocalFiles.java 0.00% 8 Missing ⚠️
...dedb/server/ha/raft/PostVerifyDatabaseHandler.java 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9757      +/-   ##
============================================
+ Coverage     77.76%   77.77%   +0.01%     
- Complexity     4439     4443       +4     
============================================
  Files          2197     2197              
  Lines        229612   229621       +9     
  Branches      49699    49705       +6     
============================================
+ Hits         178556   178592      +36     
+ Misses        32162    32147      -15     
+ Partials      18894    18882      -12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HA checksum endpoint: vector index side files (.vecgraphfp/.vecordmap/.vecpq) are not shipped but are checksummed (follow-up to #9638)

1 participant