[Bug]: Service unit stays pinned to the runtime that installed it, so deleting old runtimes breaks the next restart

Before submitting

I searched existing issues and did not find a duplicate.
I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

On Linux, run t3 service install with version A. The systemd unit gets ExecStart=~/.t3/runtime/versions/A/t3 __service-launcher.
Update the server several times with Update server from a client (A → B → C → D). Each update hands off to the running launcher. service-state.json ends up with activeVersion: D and update.fromVersion: C.
Delete old runtime directories, keeping only those service-state.json refers to (active plus fromVersion), the retention rule proposed in [Bug]: Remote server updates accumulate every immutable runtime and consume disk space #8345 and the workaround people describe there. Version A is deleted.
Wait for the service process to exit. For us, an OOM kill. systemctl --user restart t3code.service does the same.

Expected behavior

The service restarts on the active runtime. The unit should not depend on a runtime directory that nothing in service-state.json records.

Actual behavior

systemd cannot run the unit's ExecStart binary, so every restart fails with 203/EXEC until the start limit is hit. The server stays down until someone edits the unit or runs t3 service install with the active version.

The unit still pointed at 0.0.43-nightly.20260917.1851 three weeks and several remote updates later. The launcher had been running from that directory the whole time, so nothing showed the problem until the process restarted.

Cause on main @ ec80933:

bootService.ts#L607-L630: the unit's program is pinnedRuntimePaths(..., input.cliVersion), the runtime of the CLI that ran install.
selfUpdate.ts#L313-L332: a remote update calls launcher.requestUpdate and never re-renders the unit. Only t3 update from the CLI does (cli/update.ts, target.install(...)).
service-state.json records activeVersion and update.{fromVersion,targetVersion} but not the version the unit executes, so any cleanup based on that file, manual or automatic, can delete it.

Possible fixes, for maintainers to choose: re-render the unit (without restarting) after a remote update commits, or record the unit's runtime version in service-state.json so retention keeps it. Either should be settled before #8345's pruning ships, since its proposed rule would delete this runtime.

Impact

Major degradation or frequent failure

Version or commit

0.0.46-nightly.20261008.2819 (active); unit pinned to 0.0.43-nightly.20260917.1851. Code references are to main @ ec80933.

Environment

Ubuntu 24.04.4 LTS, x86_64, kernel 6.8.0, systemd 255, systemd user service, self-contained runtime. Connected from the macOS desktop app over Tailscale Serve.

Logs or stack traces

$ systemctl --user cat t3code.service | grep ExecStart
ExecStart=~/.t3/runtime/versions/0.0.43-nightly.20260917.1851/t3 __service-launcher

$ ls ~/.t3/runtime/versions
0.0.46-nightly.20261005.2676  0.0.46-nightly.20261005.2702  0.0.46-nightly.20261007.2787  0.0.46-nightly.20261008.2819

$ cat ~/.t3/runtime/service-state.json
{ "protocol": 3, "activeVersion": "0.0.46-nightly.20261008.2819",
  "update": { "fromVersion": "0.0.46-nightly.20261005.2702", "targetVersion": "0.0.46-nightly.20261008.2819", "status": "committed" } }

$ journalctl --user -u t3code.service
14:41:39 t3code.service: Failed with result 'oom-kill'.
14:41:44 Started t3code.service - T3 Code server.
14:41:44 t3code.service: Main process exited, code=exited, status=203/EXEC
...
14:42:10 t3code.service: Start request repeated too quickly.
14:42:10 Failed to start t3code.service - T3 Code server.

Workaround

Point ExecStart at the active runtime and restart:

sed -i 's#versions/0.0.43-nightly.20260917.1851/t3#versions/0.0.46-nightly.20261008.2819/t3#' ~/.config/systemd/user/t3code.service
systemctl --user daemon-reload && systemctl --user reset-failed t3code.service && systemctl --user start t3code.service

Running t3 service install with the active runtime's CLI should also re-render the unit. Before deleting old runtimes, check which one the unit's ExecStart uses.

Related: #8345 (runtime accumulation and the retention rule in step 3); #14609 (remote self-update doesn't update the CLI symlink; same pattern of the remote path skipping what t3 update does, but a different artifact).

Investigated and filed by Claude Code (claude-opus-5-5).

Same on macOS with the launchd service.

The LaunchAgent ProgramArguments still points at the runtime that ran t3 service install on Sep 25, after several remote updates since:

$ /usr/libexec/PlistBuddy -c "Print :ProgramArguments:0" ~/Library/LaunchAgents/com.t3tools.t3code.service.plist
~/.t3/runtime/versions/0.0.43-nightly.20260925.2251/t3

$ jq '{activeVersion, update: {fromVersion: .update.fromVersion, status: .update.status}}' ~/.t3/runtime/service-state.json
{ "activeVersion": "0.0.46-nightly.20261011.2967",
  "update": { "fromVersion": "0.0.46-nightly.20261008.2813", "status": "committed" } }

The launcher process runs from 0.0.43-nightly.20260925.2251, the server from 0.0.46-nightly.20261011.2967.
~/.t3/runtime/versions holds 20 runtimes, 6.8 GiB.
Pruning by the #8345 rule (active plus fromVersion) would delete the runtime the LaunchAgent executes, so the next launchctl kickstart or reboot would fail the same way.

Version: 0.0.46-nightly.20261011.2967, macOS 26.5.2, Apple Silicon.

Ran into this right after cleaning up old runtimes to work around #8345. Deleting the old versions removed the one the service unit pointed to, and T3 stopped connecting until I reinstalled the service.

Seems like these two should be fixed together, so that cleanup never deletes the version the service is using.

Reproduced on Ubuntu/systemd using 70e5e384da, built as three local version labels, in an isolated service account. Installed 0.1.1, then used Settings → Connections to update to 0.1.2 and 0.1.3. Both committed; the unit still executed 0.1.1 while state recorded only 0.1.3 and 0.1.2. Removing 0.1.1 and restarting produced ExecMainStatus=203 / 203/EXEC.

Current automatic cleanup preserves the running launcher, so it did not cause this failure; state-only manual cleanup still does. Runtime transcript. macOS was not verified.

Fix in #23247. On 0df1ba9eae, the three focused test files pass 63/63. The isolated Ubuntu/systemd runtime check retained launcher A after remote updates through C and state-based cleanup, then restarted successfully into server C.

Existing installations need a service reinstall with the fixed CLI and a restart into the fixed launcher before relying on launcherVersion; remote updating alone does not migrate older launchers. macOS launchd was not verified.

Two more data points, one on macOS (launchd) and one on Ubuntu (systemd). Both show a user-visible symptom that is not listed here yet: t3 service status reports a false "needs repair".

Symptom

T3 Code service
  Status: needs an update or repair
  Next: Run `t3 service install` to repair it.

The server itself was healthy on both machines (port 3773 returned 200, and clients connected).

Environment

	macOS 27.0 arm64 (launchd)	Ubuntu, kernel 7.0.0 x64 (systemd --user)
Unit pinned to	`0.0.46-nightly.20261013.3053` `__service-launcher`	`0.1.1-nightly.20261020.3386` `__service-launcher`
Server actually running	`0.1.2-nightly.20261021.3486` `t3 serve`	`0.1.2-nightly.20261021.3486` `t3 serve`
Update path	in-app/remote updates only	0.1.1 → 0.1.2-nightly.…3473 → …3486 (remote), `service-state.json` shows `activeVersion` 3486, `status: committed`

On macOS, the plist EnvironmentVariables.PATH also still had a stale runtime dir (versions/0.0.46-nightly.20261004.2657, a different build from the one in ProgramArguments). In addition, ~/.local/bin/t3 still pointed at 0.0.46 (#14609). Running that old CLI against the newer DB printed Database migration history diverges from this build (migrations 62–67 unknown).

What did and did not help

t3 service restart (run with the 3486 binary): no change. The status still says "needs an update or repair", because restart does not re-render the unit.
t3 service install (run with the 3486 binary): fixed on both machines. The status is now installed · t3@0.1.2-nightly.20261021.3486, and the launcher and server both run from 3486.

As far as I can tell, the status check (bootService.ts) compares the rendered unit with the unit on disk. So the stale ExecStart/ProgramArguments is the only failing condition. This matches the root cause above. Users will probably see this status message before they hit the 203/EXEC failure. It would be good if the fix for this issue also made the status say why (for example, "service unit points at runtime X, active is Y").

Diagnosed with t3 triage (Claude Code, Opus 5.5) on both machines. I wrote this comment with Claude Code.

Navigation Menu

Repository navigation

[Bug]: Service unit stays pinned to the runtime that installed it, so deleting old runtimes breaks the next restart #17490

Before submitting

Area

Steps to reproduce

Expected behavior

Actual behavior

Impact

Version or commit

Environment

Logs or stack traces

Workaround

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Service unit stays pinned to the runtime that installed it, so deleting old runtimes breaks the next restart #17490

Description

Before submitting

Area

Steps to reproduce

Expected behavior

Actual behavior

Impact

Version or commit

Environment

Logs or stack traces

Workaround

Activity

spiky02plateau commented on Oct 11, 2026

JasonHassold commented on Oct 14, 2026

juliusmarminge commented on Oct 22, 2026

juliusmarminge commented on Oct 22, 2026

architpai commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions