Service runtime 0.0.42 cannot be started by a pre-0.0.42 service-launcher (layout changed to Node SEA; launcher not replaced)

[john-e-west]

Summary

Upgrading the background-service runtime to 0.0.42 leaves the service
permanently unable to start. 0.0.42 replaced the JS bundle entrypoint with a
compiled Node SEA binary, but ~/.t3/runtime/service-launcher.mjs is not
replaced by the update and still resolves the old bundle path. The launcher
reports the new runtime as corrupt, systemd restarts it until the start limit is
hit, and the service ends in failed.

The runtime itself is fine. Only the launcher's path assumption is stale.

Impact

On this host the service went from healthy to permanently down with no user
action, and stayed down:

Active: failed (Result: start-limit-hit)
NRestarts=5
:3773 unreachable

Because Restart=always with StartLimitBurst=5, the unit exhausts its budget
in ~5 minutes and then requires a manual systemctl --user reset-failed. It does
not recover on its own. Any deployment that takes 0.0.42 with a pre-0.0.42
launcher on disk will hit this.

Root cause

service-launcher.mjs hardcodes the bundle entrypoint:

const runtimePaths = (baseDir, version) => {
	const versionDir = NodePath.join(baseDir, "runtime", "versions", version);
	return {
		versionDir,
		entryPath: NodePath.join(versionDir, "node_modules", "t3", "dist", "bin.mjs"),
		sentinelPath: NodePath.join(versionDir, ".install-complete")
	};
};

runtimeExists() stats entryPath, fails, and #startChild() throws
Selected t3@${version} runtime is missing or incomplete.

The two layouts differ:

	0.0.40	0.0.42
entrypoint	node_modules/t3/dist/bin.mjs (8.8 MB JS)	./t3 (compiled ELF, Node SEA)
top level	node_modules, package.json, package-lock.json	t3, client, resource-monitor, node_modules
node_modules holds	the t3 package	native addons only (ffi-rs, msgpackr-extract, @ff-labs, detect-libc)

0.0.42/.install-complete correctly contains 0.0.42, and the install is
complete — the sentinel check would pass if the entry path resolved.

The update ships no replacement launcher: nothing matching *launcher*
exists anywhere under versions/0.0.42/.

Reproduction

1. Run the background service on 0.0.40 (or any bundle-layout release).
2. Let it update its runtime to 0.0.42.
3. Service fails to start.

$ journalctl --user -u t3code.service
Starting T3 Code server...
t3code.service: Main process exited, code=exited, status=1/FAILURE
...
$ tail ~/.t3/userdata/logs/boot-service.log
[service-launcher] Selected t3@0.0.42 runtime is missing or incomplete.

Suggested fix

Detect the layout rather than assume it, so one launcher starts either
generation and a rollback to a bundle release still works:

const runtimePaths = (baseDir, version) => {
	const versionDir = NodePath.join(baseDir, "runtime", "versions", version);
	const nativeEntry = NodePath.join(versionDir, "t3");
	const bundleEntry = NodePath.join(versionDir, "node_modules", "t3", "dist", "bin.mjs");
	let native = false;
	try {
		native = NodeFS.statSync(nativeEntry).isFile();
	} catch {
		native = false;
	}
	return {
		versionDir,
		native,
		entryPath: native ? nativeEntry : bundleEntry,
		sentinelPath: NodePath.join(versionDir, ".install-complete")
	};
};

and at the spawn in #startChild():

const spawnCommand = paths.native ? paths.entryPath : process.execPath;
const spawnArgs    = paths.native ? ["serve"] : [paths.entryPath, "serve"];
const child = NodeChildProcess.spawn(spawnCommand, spawnArgs, { /* unchanged */ });

The "ipc" stdio channel needs no change — the 0.0.42 binary is a Node SEA
(NODE_CHANNEL_FD and node:internal/child_process are present in the image),
so it still speaks the Node IPC protocol, and ./t3 --help shows it accepts
serve.

This fix is written but not yet verified in situ on this host — applying it
requires modifying a live service file and I have not done so. The analysis
above is verified; the patch is not yet proven by a successful start.

Two adjacent notes

A user-side wrapper has the same assumption. Any script that resolves the
active runtime by reading runtime/service-state.json and joining
node_modules/t3/dist/bin.mjs breaks identically. If that pattern is documented
anywhere, it needs the same detection.

Consider making the launcher self-updating or version-gated. The underlying
hazard is that the launcher and the runtime can drift, and the launcher is the
component that cannot be fixed by the update mechanism it supervises. A
minLauncherProtocol field in the runtime manifest — refusing to select a
runtime the on-disk launcher is too old to start, rather than selecting it and
failing — would turn this class of failure into a clean refusal plus a stay on
the previous version.

Environment

t3code-bin      0.0.38-1 (Arch/AUR)
runtimes        0.0.40 (working), 0.0.42 (cannot be started)
launcher mtime  2026-09-12, i.e. the 0.0.40-era file
node            26.8.1 via mise
OS              Arch Linux (Omarchy), kernel 7.1.9, systemd 261
unit            user service, Restart=always, StartLimitBurst=5/300s

Workaround

Roll the active runtime back and reset the failed unit:

printf '{\n  "protocol": 2,\n  "activeVersion": "0.0.40"\n}\n' > ~/.t3/runtime/service-state.json
systemctl --user reset-failed t3code.service
systemctl --user restart t3code.service

[juliusmarminge]

Closing as a duplicate of #11934 — same root cause: a pre-#11510 / pre-SEA service-launcher.mjs still resolves node_modules/t3/dist/bin.mjs, so a complete standalone runtime (./t3) looks “missing or incomplete,” and the unit can hit start-limit-hit.

That was fixed on main by #11940 (bump SERVICE_LAUNCHER_PROTOCOL to 3 so staged preflight blocks activation with the actionable “update the launcher on the server machine” message). Tag v0.0.42 points at 719a76ca (2026-09-15 ~21:29Z), which is before #11940 merged (~22:21Z), so stable 0.0.42 does not include that gate yet.

Workaround (same as #11934 / your rollback): stay on / roll back to a bundle-layout runtime the on-disk launcher can start, or rewrite the launcher/unit with a current local t3 update from a new enough binary, then systemctl --user reset-failed + restart.

[juliusmarminge]

Duplicate of #11934 (fixed on main by #11940; v0.0.42 predates that merge).

[john-e-west]

Retracting this — the fault is local to my host, not t3code. Sorry for the noise.

I filed this against the wrong component. The 0.0.42 update behaved
correctly. On closer inspection of the systemd unit:

0.0.42 installed     2026-09-15 22:09:23
base unit rewritten  2026-09-15 22:09:25   <- two seconds later, by the update

The updater did rewrite the service unit for the new layout:

ExecStart=/home/jwest/.t3/runtime/versions/0.0.42/t3 __service-launcher

and the shipped binary does accept __service-launcher. Nothing was stranded.

What actually broke was a local systemd drop-in on my host, written on
12 September to bind the server to the LAN, which also overrode the launch
command:

# ~/.config/systemd/user/t3code.service.d/lan-server.conf
ExecStart=
ExecStart=/usr/.../node /home/jwest/.t3/runtime/service-launcher.mjs

ExecStart= clears the base unit's value, so my three-day-old override pinned
the old node + service-launcher.mjs invocation and defeated the update. The
failing process in the journal is that override, not anything t3code installed.

So the premise of this report — "the update ships no replacement launcher" — is
false. It ships a rewritten unit; my drop-in discarded it.

My error, and worth naming precisely: I read the drop-in early in the
investigation, read the base unit only after the update had rewritten it, and
never re-read the effective ExecStart that systemd was actually running. The
evidence that would have caught it — systemctl show -p ExecStart — was one
command away throughout.

The one suggestion I would still make, weakly

A local ExecStart= override silently defeating a unit rewrite is easy to do
and gives a confusing failure ("runtime is missing or incomplete" rather than
"your unit is launching the wrong thing"). If the launcher can cheaply detect
that it was started against a runtime whose layout it cannot read, a message
naming the effective command and pointing at drop-ins would have saved me
several hours. Entirely your call whether that is worth the code.

Closing. Apologies for the misdirected report.

[john-e-west]

Correcting my own retraction above — I over-corrected, and @juliusmarminge is right.

Both things are true, and they compose:

• [Bug]: background service self-update fails permanently against a pre-#11510 service-launcher.mjs #11934 is genuine: a pre-SEA service-launcher.mjs resolves
  node_modules/t3/dist/bin.mjs, so a complete ./t3 runtime reads as
  "missing or incomplete" and the unit reaches start-limit-hit. Exactly what
  happened here.
• What put my host on that path is a local drop-in with ExecStart= +
  ExecStart=node .../service-launcher.mjs, written before the SEA change. The
  0.0.42 update did rewrite the base unit to
  ExecStart=.../0.0.42/t3 __service-launcher; my override cleared it and kept
  invoking the old launcher.

So the upstream defect is real and my drop-in is the local trigger for it. My
retraction said "not t3code", which was wrong — please disregard that line.
A host with no ExecStart override would take the rewritten unit and never
reach the stale launcher.

Nothing needed here; #11934 / #11940 cover it. Noted for anyone who lands on
this issue: if you hit this on a release predating #11940, check
systemctl --user show t3code.service -p ExecStart for a drop-in override
before assuming the update failed.