[Bug]: Upgrading is brittle on large database #17937

Activity

1. morgaesis

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   needs-triageIssue needs maintainer review and initial categorization.Issue needs maintainer review and initial categorization.

   on Oct 10, 2026
2. JanMolcik commented on Oct 13, 2026

   

   JanMolcik

   on Oct 13, 2026

   More actions

   Same failure here, with a worse outcome: the database came back empty, not malformed. All threads, projects and scheduled tasks were gone. Adding our timeline and the code paths involved, since it pins down the exact race.

   Setup: Linux VM, t3code.service (service launcher from 0.0.46-nightly.20261010.2908), desktop on macOS connected over SSH. Update 0.0.46-nightly.20261011.2955 → 0.0.46-nightly.20261013.3042, statev2.sqlite several hundred MB, slow disk (IO pressure "full" 36–64 %).

   Timeline (local time):

   • 09:39:46: the active server hands off to the launcher. The launcher stops it, and backupDatabaseOnce starts copying the DB file. This took ~5 minutes on the slow disk, and port 3773 was down the whole time.
   • 09:40–09:45: the desktop's SSH route found nothing on 3773 and started its own t3 serve --port 3773 --base-dir ~/.t3 against the same DB. We run a watchdog that kills foreign t3 serve processes on that base dir, so this repeated 5 times (one new server per minute). The first ones served the old, intact DB (their logs show the old threads).
   • 09:44:54: backup done, the trial child starts and fails: listen EADDRINUSE 127.0.0.1:3773 (a desktop-launched server holds the port).
   • 09:44:5x: #handleExit → #returnToPrevious → restoreDatabaseBackup runs copyFile(backup, statev2.sqlite) over the live file (same inode, truncate + write).
   • 09:44:59: the next desktop-launched server opened the DB mid-restore, saw an empty file, and ran all 61 migrations from scratch. Every row in effect_sql_migrations is stamped with that second.
   • Right after: discardDatabaseBackup deleted the backup, so nothing was left to recover. 2955 then started on the fresh DB (Database migration history diverges … 61:PullRequestReviewGuides (unknown to this build)).
   • 09:46:40: the user retried and the update succeeded, only because the DB was now 4 MB and the backup took seconds.

   What we think needs fixing (from reading the launcher in 2908/3042):

   1. restoreDatabaseBackup overwrites the live DB with copyFile without checking for other open handles. Restoring under an open SQLite connection gives an empty or corrupt DB.
   2. #returnToPrevious discards the backup immediately after restoring, even if the restore or the rollback start fails. Keep it until the previous version is confirmed healthy (ideally keep the last N).
   3. backupDatabaseOnce is a raw file copy of a live DB (plus -wal/-shm). The SQLite online backup API or VACUUM INTO would be consistent and is usually faster than copy + fsync of a large file.
   4. t3 serve takes no exclusive lock on --base-dir, so two servers can own one DB (same root cause as [Bug]: Desktop starts a second backend against the background service database #6097 and [Bug]: Overlapping server lifecycles during a restart leave state.sqlite malformed and make the app unusable #11084; the unmerged fix(server): prevent duplicate servers for one state directory #9652 tried to address it).
   5. The desktop SSH route starts a replacement server while a service-managed server (serviceManaged) is mid-update. It should wait for the service (it can see runtime/service-state.json with update.status = pending).
   6. Backup duration, rollback and backup deletion are not logged, which made this hard to reconstruct.

   Mitigations we are putting in place: connect the desktop over HTTPS (pairing link) instead of SSH launch, so it never spawns a server on the host; take our own SQLite online backups outside ~/.t3/runtime; and run updates through a script that pauses the watchdog, backs up first and verifies thread counts afterwards.

   A 1-minute explainer of the failure sequence:

   t3-db-wipe-explainer-en.mp4
3. t3dotgg commented on Oct 14, 2026

   

   t3dotgg

   on Oct 14, 2026

   Member

   More actions

   Note

   🤖 Claude Opus 5.5 responding on behalf of Theo

   Closing as fixed by #16102 and #19311, shipped in 0.0.46-nightly.20261014.3105.

   • Second server during the update: a server must take an exclusive lock on the T3 home before it opens the database (fix(server): enforce one database owner and guard update rollback #16102). The SSH-launched server and the updated trial can no longer both own it.
   • Rollback over a live database: if another server owns the home, the trial exits with code 78, and the launcher waits for the owner to leave (#idleWhileStateDirOwned, apps/server/src/serviceLauncher.ts:823). It no longer copies the backup over a database that another process has open.
   • Large databases: fix(server): service updates no longer fail on full disks or large databases #19311 checks that the backup fits before the server stops, and raises the prepare window from 120 s to 30 minutes, so a slow first start on a big database no longer forces a rollback.

   One mixed-version case is not covered: a server from before the lock whose runtime record another old server already removed. If you hit this again with both sides on a build that has the lock, please reopen with the launcher log.
4. t3dotgg

   closed this as completedon Oct 14, 2026