Repository navigation
[Bug]: Remote environment can't connect while unknown-remote probes saturate VcsProcess ("did not respond during connection setup") #16503
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Run
t3 serveon a Linux VM as a remote environment (reverse proxy in front, desktop connects overwss://). - Open several projects whose
originis a self-hosted GitLab with a hostname that has nogitlabDNS label (e.g.git.example.com), so it is classifiedunknown. - Have
glabinstalled on the server and authenticated forgitlab.comonly, not for that host. - Reload the desktop window (or restart the app) so it opens a new WebSocket to the remote environment.
Expected behavior
The connection is established in well under a second. Slow or failing source-control probes may delay PR/VCS status, but must not block connection setup.
Actual behavior
The desktop stays on "Reconnecting: did not respond during connection setup." for minutes, and restarting the desktop app does not help.
What the server trace shows for each attempt:
- nginx:
GET /ws?wsTicket=...returns101. The server sends ~110 bytes and the client closes after ~7s. EnvironmentAuth.authenticateWebSocketUpgradesucceeds.ws.rpc.subscribeServerConfigis thenInterruptedafter 7.2s. In the same trace:VcsDriverRegistry.detect7075 msdetectRepository6983 msVcsProcess.run6982 ms
- At the same time,
VcsProcess.runaverages 7-9 s across ~200 runs/min. 1829 runs over 25 minutes wereVcsProcessTimeoutError ... source-control.discovery.refine-unknown-remote: glab ... after 5000ms. - Sampling the children of
t3 serveshows onlyglab auth status, repeated in every project dir.
So the connection-setup path (subscribeServerConfig → VcsDriverRegistry.detect) waits behind the unknown-remote refinement storm described in #12060. Each refinement there costs a full 5s glab auth status timeout rather than a fast ENOENT. The client gives up at ~7s and retries, and each retry lands behind the same queue.
On 0.0.45 the cause matches #12060. resolveHostingProvider and the RepositoryIdentityResolver refine pass an explicit context, so resolveHandle skips providerContextCache (TTL 5s, failures not cached) and calls refineUnknownRemoteProvider on every read.
Impact
Major degradation or frequent failure. The remote environment is unusable until the queue happens to drain. A connection that succeeds is luck: one detect call has to finish under the client's ~7s setup timeout.
Version or commit
0.0.45 (server and desktop)
Environment
- Server: Linux x64 VM,
t3 serveas a systemd user service, 4 vCPU / 8 GB. - Desktop: macOS, connecting through an nginx reverse proxy over a WireGuard mesh.
glab1.119.0 on the server PATH.
Logs or stack traces
ws.rpc.subscribeServerConfig 7197 ms Interrupted InterruptError: All fibers interrupted without error
VcsDriverRegistry.detect 7075 ms
detectRepository 6983 ms
VcsProcess.run 6982 ms
VcsProcessTimeoutError: VCS process timed out in source-control.discovery.refine-unknown-remote: glab (<project dir>) after 5000ms
Per minute on the server, before and after the workaround:
before: VcsProcess.run n≈200/min avg 7-9 s detectRepository 7-15 s
after: VcsProcess.run n≈380/min avg 0.9-1.4 s detectRepository ~1 s
Workaround
Run glab auth login --hostname <that host> on the server. The refinement still runs on every read, but each call returns in about a second instead of timing out, and connection setup goes through again. This is fragile: once the token expires the storm and the connection failures come back.
Suggested fixes:
- Don't make connection setup (
subscribeServerConfig) wait on VCS detection, or give it a separate, unshared process budget. - Route explicit-
contextcalls through a cache keyed by remote host, with a TTL in minutes that includes negative results (the Unrecognised GitHub Enterprise host re-probes fj, tea and glab once per project on every sweep #12060 fix). - Allow declaring self-hosted GitLab/GitHub hosts in settings so they never enter the
unknownbucket ([Bug]: Bare pull request numbers can't be linked on a self-hosted GitLab whose host name doesn't say "gitlab" #15390 and [Feature]: GitHub Enterprise support in the source control connector #5087 would also benefit).
Activity
juliusmarminge commentedon Oct 6, 2026
Note
Grok responding on behalf of Julius.
Triage
Thanks @IvanKhramov for the thorough report and the captured span, that pointed straight at the right place. From what I can see this is present on 0.0.45 and still on
main, and no later release moves it off the connection path yet. It looks like a companion to #12060 rather than a duplicate.What I found
session.readywaits for the firstsubscribeServerConfigsnapshot. That snapshot isloadServerConfig, which awaits the scratch-root probe before returning:gitWorkflow.isRepository(config.baseDir)in the websocket handler on 0.0.45, and the same call viamanagedFolders.scratchRootonmain.isRepositoryisVcsDriverRegistry.detect.Duration.zeroinVcsDriverRegistry. If the socket drops mid-probe, the scratch cache is invalidated and the next attempt detects again. The detect needs one of the 8 sharedVcsProcesspermits; onlyghhas its own semaphore, soglabshares the same pool.gitlablabel is classifiedunknown.resolveHostingProviderandPullRequestService.refineUnknownProjectKindscallresolveHandlewith an explicitcontext, which skipsproviderContextCache.glab auth statuswith a 5s probe timeout. The timeout is swallowed and not remembered, so every status read pays it again.gitdetect queued behind thoseglabprocesses, not VCS work the snapshot needs. The coded establishment budget on 0.0.45 is 15s, so a close around 7s may be a proxy or client abort while the handler is still inside the detect. Either way, while the queue is saturated the snapshot never goes out, the probe is cancelled, and the retry lines up behind the sameglab auth statusprocesses.Likely fix area
loadServerConfig/managedFolders.scratchRoot,VcsDriverRegistry, and GitLab refinement. Some options:VcsProcesspool.glab auth statusper host instead of paying the 5s each read.Workaround:
glab auth login --hostname <your host>. Refinement still runs on every read, butauth statusreturns in about a second instead of timing out at 5s, so the detect can finish inside the setup budget. It may come back if that token stops answering quickly.A maintainer will decide on the fix direction.