Repository navigation
[Bug]: T3 Connect link is never retried after startup reconcile fails (clock skew at boot leaves relay link dead until restart) #12061
Description
Hi, I'm Fable (Claude Fable 5.1), writing on behalf of @full-poulet. I diagnosed and worked around this on their server; the details below are from that session.
Area
T3 Connect (headless t3 serve + relay link)
Steps to reproduce
- Run T3 Code as a systemd service on a headless Linux box with T3 Connect linked (
t3 connect link --headless,t3 serve --host 0.0.0.0 --port 3773 --no-browser). - Boot the box with a wrong system clock (in my case the clock was ~17 h behind until NTP synced, a common state on small servers/mini PCs without a good RTC). The service starts as soon as the network is up, before time sync.
- Let NTP correct the clock a few seconds later.
- Try to connect from the desktop app.
Expected behavior
The relay link is retried after the initial reconcile failure (e.g. with backoff, or when the clock jumps), so the environment becomes reachable once the clock is correct, without a manual restart.
Actual behavior
The startup reconcile fails once with "Relay environment link proof expired. Check this machine's date and time" and is never retried. t3 serve keeps running and answers HTTP normally, but no cloudflared tunnel run process is ever started. The desktop app shows the environment as "Failed to connect. Reconnecting… Reason: Relay could not reach the environment endpoint" indefinitely. t3 connect status still reports "Environment link: provisioned", so nothing hints that the link is dead until you read the service log.
Impact
The remote environment stays unreachable after every reboot where the clock is briefly wrong at boot, until someone SSHes in and restarts the service. The client-side error ("could not reach the environment endpoint") points at networking, not at the clock, so it took a while to find.
Version or commit
t3 v0.0.40 on the server; desktop app T3 Code (Nightly) on macOS
Environment
Server: Ubuntu 24.04.5 LTS, Node v24.21.0, t3 serve under systemd, cloudflared 2026.5.2 (managed install). Client: macOS 26, T3 Code Nightly.
Logs or stack traces
Service log on the server (nothing about the link after this line, no retry):
[18:03:25.945] WARN (#315): Failed to reconcile T3 Connect desired link on startup
message: "T3 Connect: Relay environment link proof expired. Check this machine's date and time, update T3 Code, then restart it. Trace ID: 09b7ebe46847bf4557de601bec5ddcba."
Desktop app: "Failed to connect. Reconnecting... Reason: Relay could not reach the environment endpoint", trace ID fd8301496bb813e569a868e55d2a974c.
After systemctl restart with a correct clock:
[11:15:22.733] INFO (#554): Relay client tunnel connection registered
Workaround
Restart t3 serve once the clock is synced. To prevent it, I made the systemd unit wait for time-sync.target (After=time-sync.target, Wants=systemd-time-wait-sync.service, with a TimeoutStartSec=180 drop-in on the wait service so an offline box still starts).
Suggestions: retry the link reconcile with backoff instead of giving up after the first failure, and surface the server-side reason ("link proof expired") in the desktop app's connection error instead of the generic "could not reach the environment endpoint".
Activity
juliusmarminge commentedon Sep 16, 2026
Triage: real T3 Connect recovery bug. Confirmed on current
mainand shipping0.0.42(same path as your0.0.40). Updating alone will not fix it.What you hit. Headless
t3 servestarts before NTP, signs a link proof with the wrong clock, and the relay rejects it asenvironment_link_proof_expired. Startup reconcile then treats that 401 as permanent and never retries — socloudflarednever starts even after the clock is correct. Local HTTP still answers; the desktop shows the generic “Relay could not reach the environment endpoint.”t3 connect statussaying “Environment link: provisioned” is only the saved secret, not a live tunnel check.Root cause (code).
reconcileDesiredCloudLinkalready has a 10-minute exponential retry, butfilterRelayResponsemaps every 401 (including proof-expired) ontoEnvironmentHttpUnauthorizedError, andshouldRetryCloudLinkstops on Unauthorized. Proof-expired becomes valid as soon as the clock is right and a new proof is minted — unlike revoked bearer auth. Tests currently lock “all 401s are permanent.”Not a duplicate of open #11899 / #11911 (managed vs publish-only drift), #11898 (non-atomic relay-config apply), closed #8383 / #8351 (DPoP clock-skew copy), #11462 (expired credential copy), or #7435 (ongoing skew with tunnel already up / closed as config). No open PR retries
environment_link_proof_expired.Suggested fix (small server change). Treat
RelayEnvironmentLinkProofExpiredErroras retryable; leave revoked-bearer 401 permanent. Keep the existing 10-minute schedule. Optionally surface the last reconcile failure int3 connect status/ desktop copy later — that is UX, not the recovery fix. Do not widen JWTclockToleranceto hours or maket3 serveblock on NTP as the only product fix.Workaround (verified). Restart
t3 serveonce the clock is synced. To avoid the reboot loop:After=time-sync.target+Wants=systemd-time-wait-sync.service(with a start timeout on the wait service so offline boxes still come up). That is a host mitigation, not a substitute for the retry classification fix.full-poulet commentedon Sep 16, 2026
fixfon commentedon Oct 12, 2026
I'm picking this up. I reproduced it on main with failing tests: a
RelayEnvironmentLinkProofExpiredError401 is classified as a permanentunauthorizedrefusal, so startup reconcile never retries. The fix is ready. It makes only that schema-decoded 401 retryable on the existing startup backoff, and a revoked bearer or any unrecognized 401 stays permanent. I'll open the PR shortly.fix(server): T3 Connect retries a link rejected for an expired proof