Skip to content

GH-5430 SQLite in Balanced mode assigns node numbers and reads health checks as UTC - #5433

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-5430-sqlite-node-numbers
Nov 7, 2026
Merged

jeremydmiller merged 1 commit into
mainfrom
gh-5430-sqlite-node-numbers

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #5430

Two bugs left released inbox rows unrecovered under Balanced mode on SQLite. Durability chaos round three (#5414) lost about 4,700 messages per run.

1. Every node was node 1

SqliteNodePersistence.PersistAsync wrote back the node's default AssignedNodeNumber (1) instead of assigning one, as every other store does with a sequence or identity column.

It now assigns max + 1 over the node table and the node event records, in a single INSERT ... SELECT, which is atomic under SQLite's single writer. Including the records makes the number only move forward: a node ejected as stale keeps its number when it resumes and re-registers, so a newcomer must not get it.

2. Health checks were read as local time

datetime('now') is UTC text with no offset, and DateTimeOffset.Parse gave it the machine's offset. West of Greenwich, every health check looked hours in the future, with three effects:

  • the leader never ejected a stale node
  • the stale nodes' rows were never released
  • the database agent kept being assigned to a dead node

Node and record timestamps now parse as UTC. CI runs in UTC, which hid this.

Tests

  • New compliance test: NodePersistenceCompliance.persisting_nodes_assigns_each_its_own_number. It was red on SQLite, and it passes on PostgreSQL, SQL Server, MySQL, Oracle, RavenDB and Cosmos DB.
  • Bug_5430_node_numbers_are_assigned_by_the_store: covers the no-reuse rule and the UTC read-back. Each test was red without its fix; the UTC test can only fail on a machine outside UTC.
  • SqliteTests: 498 passed, 2 skipped.
  • Chaos rig leg 2 (seed 55, SQLite queue, Balanced, 14 kills and 4 SIGTERMs) went from 4,727 lost to clean:
    • with fix 1 alone it still lost about 4,900
    • with both fixes it settled in 73 s

Balanced mode stays supported on SQLite, since refusing it would be a breaking change.

🤖 Generated with Claude Code

https://claude.ai/code/session_013nKJrqsJw3j23Xd46q37ob

… checks as UTC

Two bugs left released inbox rows unrecovered under Balanced mode on SQLite: chaos round three
lost about 4,700 messages per run.

- Every node was node 1. SqliteNodePersistence.PersistAsync wrote back the node's default
  AssignedNodeNumber instead of assigning one. It now takes the max over the node table and the
  node event records, plus one, in a single INSERT ... SELECT (atomic under SQLite's single
  writer). Including the records means a stale node that was ejected and later re-registers
  never shares its number with a newcomer.
- Health checks were read as local time. datetime('now') is UTC text with no offset, and
  DateTimeOffset.Parse gave it the machine's offset. West of Greenwich, every health check looked
  hours in the future, so the leader never ejected a stale node, never released its rows, and kept
  assigning the database agent to a dead node. Node and record timestamps now parse as UTC. CI
  runs in UTC, which hid this.

Tests:
- NodePersistenceCompliance.persisting_nodes_assigns_each_its_own_number is new. It was red on
  SQLite, and it passes on PostgreSQL, SQL Server, MySQL, Oracle, RavenDB and Cosmos DB.
- Bug_5430_node_numbers_are_assigned_by_the_store covers the no-reuse rule and the UTC read-back.
  Each was red without its fix (the UTC test only on a machine outside UTC).
- SqliteTests: 498 passed, 2 skipped.
- Chaos rig leg 2 (seed 55, SQLite queue, Balanced, 14 kills and 4 SIGTERMs) went from 4,727
  lost to clean.

Balanced mode stays supported on SQLite. Refusing it would be a breaking change.

Closes #5430

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nKJrqsJw3j23Xd46q37ob
@jeremydmiller jeremydmiller added this to the 6.63 milestone Nov 7, 2026
@jeremydmiller
jeremydmiller merged commit fb22174 into main Nov 7, 2026
This was referenced Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite in Balanced mode: every node registers as node 1, and released inbox rows are never recovered after restarts

1 participant