[andrewcai8]

Stacked on #265 (the upstream sync). Review only the last three commits.

Problem

The home host shares one NVMe between the T3 server and heavy agent work. Whenever agents load the disk, the server's event loop freezes: 36 event loop stalled warnings in 30 minutes, worst 15.4 s, /api/health at 2–10 s, and every client stuck on "Syncing messages…". The kernel showed the main thread in do_get_write_access (the ext4 journal) and folio_wait_bit_common (page-cache misses on the 10 GB database). Every SQLite statement on the writer connection runs synchronously on the main thread, so a write that waits on the disk freezes every RPC and health check behind it.

Upstream's pingdotgg#23902, synced in #265, moves more reads to the reader worker, but writes, and reads inside write transactions, still run on the main thread.

Fix

The writer connection runs on a worker thread, as the reader already does (writerWorker: true in persistence/Sqlite.ts). A busy disk now delays writes, but the event loop stays free.

• One statement at a time, in order. Each statement is a message to the worker. The client's writer lock already admits one borrower or transaction at a time, and messages run in order, so BEGIN … COMMIT stays on one connection with nothing interleaved. The event sink's single transaction for events, projections, receipt and outbox is unchanged.
• Errors keep their types. The worker sends SQLite's errcode, and the main thread classifies it into the same SqlError reasons (UniqueViolation, LockTimeoutError, …) as before.
• Interruption. A statement the worker has started finishes before its fiber sees an interrupt, as a synchronous statement did, so a later statement can never overtake it. Waiting for checkpoint admission stays interruptible.
• Checkpoint coordination stays on the main thread, mostly unchanged. It now reads isTransaction, WAL frame counts and journal_size_limit from state the worker reports with each reply, instead of querying SQLite synchronously. The statement authorizer that classifies writes moves into the worker. When the gate is closed, the worker answers "needs admission" without running the statement, and the main thread waits as before and then resends it.
• Shutdown. The close message queues behind statements already sent, so they finish and commit first. Then the worker closes the database and exits. Statements after that fail with Database writer is closed. Restart intent is written by ordinary transactions before the layers close, so its ordering is unchanged.
• Fallbacks. In-memory databases, and any environment where a worker cannot start, keep the main-thread writer.
• A lost writer restarts the server. If the worker stops on its own, writes cannot move to a new connection: a transaction's remaining statements would run outside it. The client logs the loss and calls onWriterLost, and the server exits with code 1 so systemd or the launcher restarts it; the database, restart intent and job records are on disk. A normal shutdown closes the worker without this.
• Unreadable replies fail loudly. If a reply cannot be deserialized on the main thread (messageerror, for example an allocation failure on a huge row), the statement waiting on it fails with an error instead of holding the writer lock forever. The reader worker gets the same handling. The worker keeps running.

Measurements

Measured in Docker Desktop's Linux VM: kernel 6.12, ext4, 7.75 GB RAM, with Node 24.22.0 and SQLite 3.53.4 as on the host. The database is a 9.9 GB copy of the host's statev2.sqlite. The copy, taken from the live file, was torn, so it was rebuilt table by table; two unreadable rowid ranges were lost.

The harness uses the real Sqlite.layerFromPath, EventStore, ProjectionStore and EventSink. Eight agents each stream an assistant-message update every 50 ms, about 150 event writes/s; each write is one 7-statement transaction. A worker thread sends an HTTP request to the main thread every 100 ms ("health"), and monitorEventLoopDelay runs at 10 ms resolution. For saturation, 32 synced 4 KB writers, 8 bulk writers and 24 small-file churners share the ext4 volume, holding I/O pressure (PSI "some") at 70–80%. The page cache is dropped before each run, and each run measures 60 s after a 5 s warmup.

	Before sync (77eb8d5)	Synced (#265)	Writer worker
Idle disk: event loop p99 / max	7.1 / 248 ms	4.5 / 53 ms	2.0 / 38 ms
health p99 / max	100 / 244 ms	8.0 / 22 ms	1.8 / 5.6 ms
event writes/s (about 147 offered)	146.5	146.9	147.5
write latency p50 / p99	1.0 / 3.3 ms	0.3 / 2.3 ms	1.0 / 10.4 ms
shell snapshot p50, 2,713 threads	223 ms	218 ms	183 ms
Saturated disk: event loop p99 / max	97 ms / 5.45 s	261 ms / 3.92 s	5.6 / 13.4 ms
health p99 / max	2.40 / 7.58 s	2.90 / 3.84 s	3.7 / 5.8 ms
health requests over 1 s	5	9	0
event writes/s	44.5	38.1	49.5
Saturated, plus a shell snapshot every 5 s: health	no reply in the whole 65 s window; 155 s in, the main thread was still in folio_wait_bit_common	p99 182 ms, max 9.23 s; event loop max 4.23 s	p99 3.5 ms, max 11.2 ms; event loop max 13.7 ms
Peak ingestion, unthrottled, Docker VM	3,374/s	3,560/s	1,957/s
Peak ingestion, unthrottled, bare-metal Mac (fresh database, two runs)	not run	1,002 and 995/s	1,718 and 1,695/s

The before-sync build reproduces the host: the main thread blocks in folio_wait_bit_common reading pages of the large database. The sync helps on an idle disk but not under saturation. With the worker, the main thread never stalls longer than 14 ms, even while writes wait seconds on the disk.

Peak throughput depends on thread wake-ups. Each event write makes seven round trips. In the Docker VM a worker message round trip costs 33 µs, and even a bare futex ping-pong costs 26 µs, so peak ingestion there drops 45%. On bare-metal macOS those figures are 8.6 µs and 4.5 µs, and the worker build is about 70% faster at peak, because SQLite runs alongside the main thread's own work. The home host is bare-metal Linux. At the realistic rate of about 150 writes/s, throughput is unchanged in every build.

Risks

• Every writer statement now costs a message round trip. That is about 7 µs in a Mac micro-benchmark, where a 10-statement transaction took 0.33 ms instead of 0.26 ms. Event ingestion throughput is measured above.

Tests

nodeSqliteClient.writerWorker.test.ts covers, against a real worker and a file database:

• statements run without blocking the event loop;
• a transaction commits all or nothing, and readers see it only after commit;
• each caller's statements keep their order, and transactions stay contiguous;
• SQLite failures arrive as typed SqlErrors (unique violation, prepare failure, lock timeout);
• a statement sent before close finishes and persists, and later ones fail;
• the WAL keeps being checkpointed through the worker, with and without it;
• a write held for checkpoint admission runs once;
• a writer or reader reply that cannot be read fails only its own statement, and the next statement works;
• a writer worker that stops on its own is reported once through onWriterLost, a normal shutdown reports nothing, and later writes fail.

Mutating the client to run on the main thread, to skip the post-commit checkpoint check, to ignore the admission gate, to execute a held statement twice, or to report a lost writer during shutdown each fails a test.

Verification

• nodeSqliteClient.test.ts (17) and nodeSqliteClient.writerWorker.test.ts (11) pass on the head commit.
• With the worker on, 216 server tests in 40 files pass: every file that opens a database through layerFromPath, plus serverRuntimeStartup. The exception is one known flake, below.
• Known flake, also on chore: sync upstream through 66ee628e08 #265: ProvisionOperationStore.test.ts "two server processes cannot both claim the same allocation effect" fails about once in three parallel batches. Its contender processes run with --input-type=module, so evaluated workers cannot require and the writer falls back to the main thread. That fallback runs SELECT sqlite_version() before busy_timeout is set, while the other process recovers the WAL. The worker now sets busy_timeout before its own startup reads; the main-thread path is unchanged.
• Independent review (Opus) of transaction ordering, interruption, checkpoint coordination, errors and shutdown found no correctness, data-loss or liveness defect. Its findings are fixed in the second and third commits: an interrupted caller skipped the WAL check for its write, which had still run; an unreadable reply could hold the writer lock forever; and a dead writer left the server half alive.
• Typecheck runs in CI; this machine blocks local tsc.

Not verified: the real server against the host's database, and the web, desktop and mobile clients.

Follow-ups (not in this PR)

• Large reads that go through the writer are structured-cloned to the main thread. They should be readOnly so they run on the reader, which upstream is already moving toward.
• When the checkpoint worker asks to truncate while a statement is in flight, the acknowledgement waits for the next write's check rather than going out when that statement finishes. Truncation is still safe; it is just later than it could be.
• The worker source carries its own copy of the statement-authorizer and mutating-pragma rules next to the main-thread copy, which the fallback still uses. A rule change has to be made in both.
• A long read on the reader pins the WAL, and the checkpoint gate then holds every write until it ends. This predates the PR; upstream's open fix(server): keep writes moving while readers pin the WAL pingdotgg/t3code#23496 changes it.
• Evaluated workers cannot require when the parent process runs with --input-type=module, so in such processes the writer falls back to the main thread. The reader and checkpoint workers have the same limit, and the server, which runs from a file, is unaffected.
• The main-thread fallback runs SELECT sqlite_version() before busy_timeout is set, which makes the multi-process ProvisionOperationStore test flaky on chore: sync upstream through 66ee628e08 #265 as well.

🤖 Generated with Claude Code

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	5.0 KiB	—	6.8 KiB	✅
Codex	Thread snapshot wire	—	3.8 KiB	—	4.9 KiB	✅
Codex	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Codex	Live turn WebSocket decoded	—	20.9 KiB	—	29.3 KiB	✅
Codex	Live turn messages	—	2	—	8	✅
Claude	Total thread wire	—	5.0 KiB	—	6.8 KiB	✅
Claude	Thread snapshot wire	—	3.8 KiB	—	4.9 KiB	✅
Claude	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Claude	Live turn WebSocket decoded	—	21.2 KiB	—	29.3 KiB	✅
Claude	Live turn messages	—	2	—	8	✅

Baseline: unavailable · PR result: e2fc9bf · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 108.5 KiB
• Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.