Repository navigation
feat(#9498): operator override to lift the stale-snapshot read floor on a sole voter - #9593
Conversation
…on a sole voter POST /api/v1/cluster/accept-stale-snapshot (root only) and its gRPC twin AcceptStaleSnapshot lift the node-wide stale-snapshot read floor (#6111) on a node that is the sole voter of its cluster, where no peer can ever serve the resync that is otherwise the only way out. The marker index is persisted as the applied position so the next reinitialize() does not raise the floor again, and the override is logged at WARNING with who made it. Refused with a peer, while a download is running, and when no floor stands. RemoteServer and RemoteGrpcServer clients, OpenAPI, alerts and ha-raft/CLAUDE.md updated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (21)
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review. 📝 WalkthroughWalkthroughAdds a root-authorized operation to accept a node-wide stale-snapshot floor on a sole-voter Raft node. The operation persists acceptance state and is available through HTTP and gRPC, with remote client methods and integration tests. ChangesStale-snapshot floor acceptance
Priority: ⬆️ High Estimated code review effort: 4 (Complex) | ~45 minutes Change: Feature · Severity of issue fixed: Medium Sequence Diagram(s)sequenceDiagram
participant Operator
participant PostAcceptStaleSnapshotHandler
participant RaftHAPlugin
participant ArcadeStateMachine
Operator->>PostAcceptStaleSnapshotHandler: POST accept-stale-snapshot
PostAcceptStaleSnapshotHandler->>RaftHAPlugin: acceptStaleSnapshot with caller identity
RaftHAPlugin->>ArcadeStateMachine: acceptStaleSnapshotFloor
ArcadeStateMachine-->>RaftHAPlugin: acceptance details
RaftHAPlugin-->>PostAcceptStaleSnapshotHandler: acceptance result
PostAcceptStaleSnapshotHandler-->>Operator: HTTP response
Suggested reviewers: Merge Risk: ⚪ Minimal · up to No actionable merge-blocking issue was established for the stale-snapshot recovery operation. It is mergeable after normal checks. 🚥 Pre-merge checks | ✅ 4 | ❓ 1❌ Failed checks (1 inconclusive)✅ Passed checks (4 passed)Full details: Docstring CoverageExplanation Docstring coverage is 33.68% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 95 functions across 18 files. (3 skipped: 2 unsupported, 1 too large.) ✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again. |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 19 |
🟢 Coverage 78.57% diff coverage · -5.63% coverage variation
Metric Results Coverage variation ✅ -5.63% coverage variation Diff coverage ✅ 78.57% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (4141f12) 222297 190288 85.60% Head commit (4176531) 255026 (+32729) 203936 (+13648) 79.97% (-5.63%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#9593) 112 88 78.57% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
ReviewOverall this is a solid, carefully scoped change. It mirrors the #9449 override, keeps the same safety rails (root only, sole voter only, refused during a download, 404 when no floor stands) and persists before lifting. The PR describes the lock order as What looks good
Suggestions (non-blocking)
SecurityRoot-only on both transports. The route takes no user-controlled input beyond credentials, and there are no new dependencies. The override is hard to reach by accident and is clearly audited. CLAUDE.md conformance
No blocking issues found. Nice work. |
…he stale floor, clearer sole-voter report wording (#9498) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Review of #9593 (accept-stale-snapshot override) I read the state machine change, the alert text and the start of the new handler. I did not read the HAServerPlugin, gRPC or test diffs in full. Good
Points to check
Security
Overall this is a solid, carefully reasoned change. Points 1 and 2 are the ones I would fix before merge. |
…database positions (#9498) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Review: sole-voter override to lift the stale-snapshot read floor (#9498) I read the diff but did not build or run the tests. The design is sound and mirrors the #9449 accept-diverged override. What looks good
Points to consider
Verdict: Well tested and conservative: it refuses by default and fails closed. Only point 2 might hide a real functional gap, so I would check that before merging. The rest are minor. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9593 +/- ##
============================================
- Coverage 77.38% 77.36% -0.02%
- Complexity 4275 4291 +16
============================================
Files 2160 2161 +1
Lines 222297 222409 +112
Branches 47628 47648 +20
============================================
+ Hits 172014 172060 +46
- Misses 32048 32096 +48
- Partials 18235 18253 +18 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Closes #9498
Summary
A sole voter whose Ratis snapshot marker runs ahead of the entries it applied holds the node-wide stale-snapshot read floor of #6111:
/api/v1/readystays 503 and LINEARIZABLE reads are clamped. Only a full resync from a peer lifts it, and a leader refuses to resync from itself, so on a one-voter cluster the wedge was terminal with no operator action short of rebuilding the node. This PR adds the node-wide counterpart of the #9449 override:POST /api/v1/cluster/accept-stale-snapshot(root only) and the gRPCAcceptStaleSnapshotRPC, both throughHAServerPlugin.acceptStaleSnapshot. It is allowed only on a sole voter. It accepts the gap, persistsmax(marker index, global applied)as the applied position (global, plus every present database that has no quarantine or per-database floor of its own), so the nextreinitialize()does not raise the floor again. It also drops the queued download, clears the floor, wakes the apply waiters, and logs a WARNING audit line with who did it, the floor and the marker index. The write is not best-effort: if it fails, the in-memory positions are put back and the floor stays. The override is refused with a peer (409 / FAILED_PRECONDITION), while a snapshot download is running (409), and when no floor stands (404 / NOT_FOUND).Coverage
POST /api/v1/cluster/accept-stale-snapshot(PostAcceptStaleSnapshotHandler, worker thread)Issue9498AcceptStaleSnapshotITAcceptStaleSnapshot(ArcadeDbGrpcAdminService)Issue9498GrpcAcceptStaleSnapshotIT,...WithoutHaITRemoteServer.acceptStaleSnapshot()Issue9498AcceptStaleSnapshotITRemoteGrpcServer.acceptStaleSnapshot()Issue9498RemoteGrpcAcceptStaleSnapshotITPluginApiSpec,HA_RAFT_PATHS)PluginApiSpecTest,OpenApiSpecGenerationIT,RaftHAPluginRegisteredRoutesMatchApiSpecTestArcadeStateMachine.acceptStaleSnapshotFloor,RaftHAPlugin.acceptStaleSnapshot)Issue9498AcceptStaleSnapshotFloorTest(10 tests)local-resync-in-progressalert, sole-voter no-peer SEVERE line,ha-raft/CLAUDE.md)reinitialize()is the only writer that raises itreinitialize())Known gaps: None filed. Same audit residue as #9449: the gRPC audit line names the user, not the caller address.
Test plan
Issue9498AcceptStaleSnapshotFloorTest(10 tests; real state machine, real Ratis storage + snapshot marker, real applied-index file): lift + survivesreinitialize()+ persisted for a fresh state machine; refused with a peer; 404 with nothing standing; refused while a download runs; failed write lifts nothing and puts back exactly the per-database positions it moved; quarantined DB keeps its quarantine and position; never regresses an apply position past the marker; default HA plugin refuses; sole-voter report names the routeneedsSnapshotDownload.set(false)/ the global position write makes the core test failIssue9498AcceptStaleSnapshotIT(one-voter cluster: ready 503 -> 200, 404 after; RemoteServer client)Issue9498GrpcAcceptStaleSnapshotIT,Issue9498GrpcAcceptStaleSnapshotWithoutHaIT,Issue9498RemoteGrpcAcceptStaleSnapshotITIssue9449*(unit + 3 ITs + client IT),Issue6111StaleSnapshotReadFloorTest,Issue8529LeaderServiceGapHandOffTest,Issue9308SoleVoterApplyErrorTest,Issue7621BlockingHandlersWorkerThreadTest,PluginApiSpecTest,ApiSpecPathConstantsTest,OpenApiSpecGenerationITAdversarial pass
Not run as an independent subagent: this run had no subagent tool. I did a self-review in its place. It found nothing that needed a change: lock order is
snapshotDownloadLock(tryLock only) thenappliedIndexFileLock, so it cannot deadlock, and no other path raises the floor besidesreinitialize().Release note
New root-only operator route
POST /api/v1/cluster/accept-stale-snapshot(gRPCAcceptStaleSnapshot) for a sole voter stuck behind the stale-snapshot read floor. Add it to the 26.11.1 docs checklist (arcadedb-docs#477) next to accept-diverged.Review cycles
65d366a7f6. claude-review found no blocking issues. Applied: a package-private test hook (ArcadeStateMachine.publishStaleSnapshotFloor) in place of reflection inBaseRaftHATest(BaseRaftHATestis in the same package as the state machine, so no public hook is needed), and clearer wording in the sole-voter report. Pushed asaf436317bd.af436317bd. Applied: the rollback now runs in afinallyon any failure, including a RuntimeException, and restores only the per-database entries the override moved, so lock-free readers never see a cleared map. The download-running message now also covers an install holding the lock. Added a test for the rollback of a database with no prior position. Pushed as417653143a.417653143a. claude-review: "Well tested and conservative... it refuses by default and fails closed". Nothing actionable (see below). Clean.Deferred / skipped review items
resulttext in the gRPC response" (cycle 1): skipped, to keep parity withAcceptDivergedDatabaseResponse, which carries none either.reinitialize()compares only the GLOBAL persisted position against the marker. The per-database floor is published only by an install that gave up (HA: a snapshot install that gives up on a database still ACKs the snapshot index, clearing the read floor and enabling stale linearizable reads #6760), never by a restart. A lower per-database position only stops the bootstrap replay-skip from skipping that database's entries, which is the conservative direction. The resync path (writePersistedAppliedIndexForAllDatabases) uses the sameserver.getDatabaseNames()set.IllegalStateExceptionthatacceptStaleSnapshotFloorthrows is the download/install refusal.BaseGraphServerTestteardown runsGlobalConfiguration.resetAll().Final state
clean-approval. Merge is the developer's.
🤖 Generated with Claude Code
Summary by CodeRabbit