Repository navigation
fix(#9751): leave the node-locally built vector graph, its side files and last-tx-id.bin out of the HA checksum answers - #9757
Conversation
… and last-tx-id.bin out of the HA checksum answers Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
ReviewOverall this is a solid, well-scoped fix. Both collectors now read one declaration ( Positives
Suggestions
No security or performance concerns. The checks are a few LGTM once the nits are addressed. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to No actionable merge-blocking issue is identified; the change is mergeable after normal checks. Pre-merge checks |
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 9 |
🟢 Coverage 27.27% diff coverage · -5.29% coverage variation
Metric Results Coverage variation ✅ -5.29% coverage variation Diff coverage ✅ 27.27% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (123ad9e) 229612 197489 86.01% Head commit (1c97cc2) 262238 (+32626) 211667 (+14178) 80.72% (-5.29%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#9757) 11 3 27.27% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
…apshot-install claim (#9758), check the production predicate on a real listing, nits Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ReviewSolid, well-scoped change. The single declaration in What I checked
Minor, non-blocking
No bugs, security, or performance issues found. Approving in spirit. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9757 +/- ##
============================================
+ Coverage 77.76% 77.77% +0.01%
- Complexity 4439 4443 +4
============================================
Files 2197 2197
Lines 229612 229621 +9
Branches 49699 49705 +6
============================================
+ Hits 178556 178592 +36
+ Misses 32162 32147 -15
+ Partials 18894 18882 -12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Closes #9751
Summary
An LSM vector index's proximity graph is built by every node for itself, on the inner database instance and without the WAL (#8292, #8905). Its
.vecgraphpages, and the.vecpqproduct quantization trained in the same pass, therefore carry different bytes on two healthy nodes holding the same vectors. The.vecgraphfpmanifest and the.vecordmapordinal map are plain files that no snapshot ships, so a follower that installed a snapshot has none of them until it builds its own graph.last-tx-id.binis rewritten whenever a node rotates its WAL or closes, so its value depends on that node's own timing. Every one of these files reachedGET /api/v1/ha/snapshot/{db}/checksums. The.vecgraphcomponent (registered, so enumerated) also reachedPOST /api/v1/cluster/verify/{db}on both of its paths. All of them are now declared inNodeLocalFiles, and both collectors read that one declaration. The vectors themselves live in the index's.lsmvecidxpages. Those pages are replicated and still compared.The issue's open question: is
.vecgraphdeterministic across nodes?No. I measured it by building the same 400 vectors into two independent databases, once with
NONEand once withPRODUCTquantization, then comparing the files byte for byte. The probe was a scratch test and is not committed..lsmvecidx(vector pages).bucket,.dict,configuration.json,statistics.json.vecgraph(both indexes).vecpq.vecgraphfp,.vecordmapSo, as the issue suspected, the three side files were only the visible part. Without this change the checksum endpoint and the cluster verify report every vector index as divergent once two nodes have built their graphs.
Finding ledger
*.vecgraphfpnot shipped but checksummed*.vecordmapnot shipped but checksummed*.vecpqnot shipped but checksummed.vecgraphdeterministic?last-tx-id.binrewritten per node on WAL rotation and closeNONEandPRODUCT, since onlyNONEwrites an ordinal map and onlyPRODUCTwrites a PQ) and a compacted TimeSeries type, and asserts each family is on disk before it checks anythingCompleteness
Invariant: a file whose bytes legitimately differ between two healthy nodes holding the same data never appears in a cross-node checksum answer.
Sweep commands and the results:
grep -rn "isNodeLocalFileName\|NodeLocalFiles\." --include='*.java' . | grep -v /test/: one consumer,SnapshotManager.computeFileChecksums, which serves/checksums.grep -n "isTemporaryFileName" PostVerifyDatabaseHandler.java: the two enumeration paths of the cluster verify (window at line 664, flush-suspension fallback at line 703). Neither usedNodeLocalFiles.grep -rn "FILE_EXT\s*=" engine/src/main/java: every engine file extension.bfidxis a replicated component (its javadoc covers this).vcidxandsparsesegare replicated components.ts.sealedis shipped and compared on purpose (PostVerifyDatabaseHandler never checksums .ts.sealed, so a TimeSeries sealed-store divergence is invisible to /verify #7338)./checksums->SnapshotManager.computeFileChecksums, both branches (window and live)NodeLocalFiles.isNodeLocalFileName)/cluster/verify->PostVerifyDatabaseHandler.computeLocalChecksums, window pathisNotCompared)/cluster/verify, flush-suspension fallback pathisNotCompared).lsmvecidxvector pages.bfidx,.vcidx,.sparsesegTest plan
Issue9751VectorSideFilesChecksumTest(5 tests): the probe with the real writer-built names (including the legacy PQ name), shape-not-substring matching, a node that built its graph vs one that installed a snapshot agree, a real database with both index kinds writes every side file and only the vector pages are checksummed, and the cluster verify drops.vecgraphon both pathsIssue9638NodeLocalFilesChecksumTestguard extended with vector indexes and a TimeSeries typeSnapshotManagerTest,Issue6116ChecksumsPointInTimeTest,Issue7338VerifyChecksumsSealedStoresTest,Issue7634VerifyWindowHoldsNoReadLockTest,Issue8018ChecksumTransientFilesTest,Issue8738SealedStoreReplacedAfterT0Test,Issue8942UnwrittenValidationVerdictTest,Issue8019SnapshotShipSkipsCompactionTemporaryTest,Issue7456SnapshotShipConfigurationFromWindowTest,Issue8905VectorBuildHATest, plus both classes aboveCoverage
Covered:
/api/v1/ha/snapshot/{db}/checksums(both branches) and/api/v1/cluster/verify/{db}(both paths).Known gaps: #9758. This PR leaves snapshot shipping unchanged:
.vecgraphis still shipped, and its manifest still is not. A follower that installs a snapshot therefore has graph pages with no manifest beside them. Its load path judges such a graph only by node count and can reuse it unverified, which is the weak #6106 comparison. That concerns what is shipped, not what is compared, so it is filed separately.Residual risk
A
.vecgraphthat is corrupt on one node is no longer flagged by the divergence detector. That is the intended trade: it is a rebuildable cache, the same as the GAV CSR in #9638, and the graph is a cache that can be rebuilt from the compared.lsmvecidxpages. A real divergence in the vectors still shows, because the.lsmvecidxpages are compared.Adversarial pass
Not run as a separate subagent: this session has no subagent-spawning tool. I reviewed the diff myself against the issue body instead. Findings:
grep -rn "FILE_EXT\s*=" engine/src/main/java..bfidx(its javadoc: a component precisely so that it replicates),.vcidxand.sparsesegare replicated components. Not real./checksumsconsumer that decides freshness fromlast-tx-id.bin. The only references areSnapshotHttpHandleritself and the OpenAPI spec (PluginApiSpec). Not real.Review cycles
6d3ee185911c97cc2c54Deferred / skipped items
NodeLocalFilesimports four vector classes just forFILE_EXT": skipped. These are same-module constants, and importing them means a renamed extension moves the classification with it, which was the point of statistics.json` and a Graph Analytical View's persisted CSR are written per node, and only on the leader, so the HA checksum endpoint reports them as data divergence on a healthy cluster #9638's design.@Tag("slow")if CI time becomes a concern": skipped. The reviewer itself would not add it, and the class runs in about 1 s./checksumsfor freshness": checked by grepping forCHECKSUMS_SUFFIXand/checksums, which finds onlySnapshotHttpHandlerandPluginApiSpec. No consumer.Final state
clean-approval
🤖 Generated with Claude Code