Repository navigation
GH-5430 SQLite in Balanced mode assigns node numbers and reads health checks as UTC - #5433
Merged
Merged
Conversation
… checks as UTC
Two bugs left released inbox rows unrecovered under Balanced mode on SQLite: chaos round three
lost about 4,700 messages per run.
- Every node was node 1. SqliteNodePersistence.PersistAsync wrote back the node's default
AssignedNodeNumber instead of assigning one. It now takes the max over the node table and the
node event records, plus one, in a single INSERT ... SELECT (atomic under SQLite's single
writer). Including the records means a stale node that was ejected and later re-registers
never shares its number with a newcomer.
- Health checks were read as local time. datetime('now') is UTC text with no offset, and
DateTimeOffset.Parse gave it the machine's offset. West of Greenwich, every health check looked
hours in the future, so the leader never ejected a stale node, never released its rows, and kept
assigning the database agent to a dead node. Node and record timestamps now parse as UTC. CI
runs in UTC, which hid this.
Tests:
- NodePersistenceCompliance.persisting_nodes_assigns_each_its_own_number is new. It was red on
SQLite, and it passes on PostgreSQL, SQL Server, MySQL, Oracle, RavenDB and Cosmos DB.
- Bug_5430_node_numbers_are_assigned_by_the_store covers the no-reuse rule and the UTC read-back.
Each was red without its fix (the UTC test only on a machine outside UTC).
- SqliteTests: 498 passed, 2 skipped.
- Chaos rig leg 2 (seed 55, SQLite queue, Balanced, 14 kills and 4 SIGTERMs) went from 4,727
lost to clean.
Balanced mode stays supported on SQLite. Refusing it would be a breaking change.
Closes #5430
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nKJrqsJw3j23Xd46q37ob
Merged
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5430
Two bugs left released inbox rows unrecovered under Balanced mode on SQLite. Durability chaos round three (#5414) lost about 4,700 messages per run.
1. Every node was node 1
SqliteNodePersistence.PersistAsyncwrote back the node's defaultAssignedNodeNumber(1) instead of assigning one, as every other store does with a sequence or identity column.It now assigns
max + 1over the node table and the node event records, in a singleINSERT ... SELECT, which is atomic under SQLite's single writer. Including the records makes the number only move forward: a node ejected as stale keeps its number when it resumes and re-registers, so a newcomer must not get it.2. Health checks were read as local time
datetime('now')is UTC text with no offset, andDateTimeOffset.Parsegave it the machine's offset. West of Greenwich, every health check looked hours in the future, with three effects:Node and record timestamps now parse as UTC. CI runs in UTC, which hid this.
Tests
NodePersistenceCompliance.persisting_nodes_assigns_each_its_own_number. It was red on SQLite, and it passes on PostgreSQL, SQL Server, MySQL, Oracle, RavenDB and Cosmos DB.Bug_5430_node_numbers_are_assigned_by_the_store: covers the no-reuse rule and the UTC read-back. Each test was red without its fix; the UTC test can only fail on a machine outside UTC.Balanced mode stays supported on SQLite, since refusing it would be a breaking change.
🤖 Generated with Claude Code
https://claude.ai/code/session_013nKJrqsJw3j23Xd46q37ob