Unrecognised GitHub Enterprise host re-probes fj, tea and glab once per project on every sweep #12060

Activity

1. juliusmarminge commented on Sep 16, 2026

   

   juliusmarminge

   on Sep 16, 2026

   Member

   More actions

   Triage

   Confirmed on current main (ccf220be2) and shipping 0.0.42. This is a real server perf bug, not a wash of #5087 / #5089 / #11220. Those would stop this host from entering the unknown bucket. This report is the cost of the bucket itself: an unrefinable host is re-probed on every read, and firstSuccessOf multiplies that by every project that shares it.

   What the code does

   <tenant>.ghe.com is classified unknown. isGitHubHost is host === "github.com" or a complete github DNS label (packages/shared/src/sourceControl.ts). That is #5932 / #6175. acme.ghe.com has neither, so detectSourceControlProviderFromRemoteUrl returns { kind: "unknown", baseUrl: "https://<tenant>.ghe.com" }. There is no ghe.com exception.

   Nothing can refine it. Of the five discovery specs, only GitLab and Forgejo (managed fj/tea) register refineUnknownRemote. GitHub’s spec is version + gh auth status only — no remote hook. parseGitHubAuthStatus already walks every host in that JSON, including enterprise ones, and then findAuthenticatedGitHubAccount keeps a single account for the one github discovery row. The authenticated GHE host is sitting in the parse and is never asked “does this remote belong to you?”

   Each refinement therefore always runs fj, tea, and glab against a host none of them will claim. GitHub’s gh is on PATH and already logged in to that host, and is not in this loop.

   The result is discarded, then asked again:

   • SourceControlProviderRegistry.resolveHandle only hits providerContextCache (5s, Duration.zero on failure) when context is omitted. Both hot callers pass context and skip it: PullRequestService.refineUnknownProjectKinds, and server.ts’s identity refine (which also forces kind: "unknown").
   • Forgejo’s managed refiner orElseSucceeds missing CLIs to []. CLI specs orElseSucceed to null. A host-level “nobody claimed this” is indistinguishable from a one-off miss, and is not stored.
   • RepositoryIdentityResolver does cache the identity for 1 minute. That is a different cache. refineUnknownProjectKinds never consults it.

   firstSuccessOf then multiplies it. Candidates are grouped by baseUrl. Every unknown project on the GHE host lands in one group. When no candidate can succeed, the whole list runs. That path is load-bearing for broken checkouts — it does not want eleven healthy checkouts of an unclaimable host.

   That is the 12-span canonicalRef. requireProject first refines listWorkspaceProjects({ projectId }) (one candidate). Unknown has no PR API, so it falls through to listWorkspaceProjects({ host }) and firstSuccessOfs the other projects on that host. 1 + N = N+1 refineUnknownRemoteProvider → 2(N+1) ForgejoCli.listLogins. Matches the trace.

   PullRequestSyncReactor ticks Schedule.spaced("1 minute") at concurrency: 8. Open + unsettled and null snapshots always are due. Eight of those canonicalRefs at ~20s is enough to blow the 60s gap. The 119s sweep span is that backlog, not a separate scheduler bug.

   WSL2 is the amplifier, not the defect. A missing fj/tea/glab walks every PATH entry; /mnt/c/... on 9p is ~100× a native ENOENT. Same storm on macOS/Linux, cheaper per spawn. Desktop “Reconnecting” under extra load is the event-loop losing a health probe, not a separate reconnect bug.

   Related, not duplicates

   Work	What it did	Why this still happens
   Closed #5087 / closed unmerged #6052	GHE as a first-class connector	Never landed. Detection still ends at github.com / a github label.
   Open #5089	.ghe.com + GitHub refineUnknownRemote via gh hosts; new github-enterprise kind	Would take this tenant out of unknown. XXL, unvouched, not the negative-cache / firstSuccessOf hole. Do not wait on it, and do not attribute Closes #12060.
   Open #11220	Idle PR-discovery / settlement cadence; 60s caches that expire before the next spaced tick	Adjacent load. Different loops, different caches. A longer identity TTL does not stop refineUnknownProjectKinds.
   Merged #6175 (closed #5932)	Complete DNS-label match so notgithub.example.com is not GitHub	Why acme.ghe.com is unknown. Keep the strict match; add an explicit GHE rule or a gh-host refine, do not revert substring matching.

   No open PR caches a negative refinement or stops firstSuccessOf after a host-level miss.

   Suggested fix

   Keep this a small server bugfix on unknown-remote refinement. Do not land #5089 to close this, and do not rebuild provider detection.

   1. Cache refinement by baseUrl (and host), including “still unknown”. One TTL for the whole group, minutes not 5s, on both success and “no spec claimed this.” resolveHandle({ context }) must read that map. Identity-resolver refine should too. Failures that are cwd-specific (broken checkout, spawn crash) stay uncached so the broken-then-healthy checkout test still walks to /healthy.
   2. Stop exhausting the group on a host-level miss. After the first candidate returns kind: "unknown" because every spec returned null, the rest of that baseUrl group will too. firstSuccessOf only needs the next cwd when this checkout failed to produce an answer.
   3. Do not do the host-wide refine after the project-scoped one already failed. requireProject’s listWorkspaceProjects({ host }) fallback is for “route this PR to another checkout on a supported host.” For unknown it is a second full probe storm that cannot create an API.
   4. Optional, separate, small: classify *.ghe.com as GitHub (or add a GitHub refineUnknownRemote that claims a remote whose host is in gh auth status). That removes the GHE trigger. It does not replace (1)–(3) for GHES vanity domains and everything else that stays unknown.
   5. Tests: N unknown projects on one *.ghe.com (or git.example.internal) with fj/tea/glab missing → one refine per host per window, not N; the “broken then healthy GitLab cwd” test still asks both; canonicalRef on an unknown project does not call listWorkspaceProjects({ host }) after the project-scoped refine already returned unknown.
   6. Trace close: same multi-project GHE machine, one canonicalRef is one refineUnknownRemoteProvider (or one per host), PullRequestSyncReactor.sweep stays inside the 60s gap, sidebar VCS settles in hundreds of ms.

   Do not treat PATH stubs, skipping the idle sweep, or “install glab” as the fix.

   Workaround

   The reporter’s ~/.local/bin/{tea,fj,glab} exit 1 stubs are the right local mitigation on WSL: discovery still reports missing, refinement still returns unknown, PATH no longer walks /mnt/c. Measured ~2× on that box; the storm is still there. No in-app way to mark a host as GitHub.

   Classification: bug · accepted · medium (server source-control; uncached unknown-host refine × projects on the host; WSL PATH makes it user-visible; desktop reconnect is the sideshow)
   Labels: add bug, accepted, via-triage
   Discord tags: performance, linux
2. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   acceptedfeature request acceptedfeature request accepted

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 16, 2026
3. renemifsud commented on Sep 30, 2026

   

   renemifsud

   on Sep 30, 2026

   More actions

   Adding a macOS data point. On macOS the same probe storm does more than slow down the server. It exhausts Gatekeeper (syspolicyd), and after that other apps cannot start new processes.

   Setup

   • T3 Code 0.0.43-nightly.20260927.2344, macOS 26.3 (25D125), Apple Silicon
   • 3 projects whose remotes are classified unknown: one <tenant>.ghe.com remote, and two that use an SSH host alias for github.com (git@<alias>:owner/repo.git, same shape as [Bug]: Pull request pane treats SSH aliases as GitHub API hosts #6198)
   • 54 thread PR links across 11 non-archived threads. snapshot_json is NULL on all 54, so every link is due on every sweep
   • tea and glab are not installed. The server's PATH has 21 entries

   Per sweep (every ~60 s, 4 consecutive minutes from server.trace.ndjson)

   • ~804 failed processRunner.runProcessCore spans (glab auth status and tea login list --output json, all ENOENT), 402 ForgejoCli.listLogins

   • 17,208 to 17,658 kernel log lines per minute, all for the T3 executable:

     kernel: (AppleSystemPolicy) ASP: Security policy would not allow process: <pid>, /Applications/T3 Code (Nightly).app/Contents/MacOS/T3 Code (Nightly)
     

     That is about 21 to 22 lines per failed spawn, close to the number of PATH entries.

   What breaks

   • syspolicyd runs out of file descriptors. About 5,000 of each of these lines per 10 s:

     syspolicyd: [com.apple.securityd:security_exception] UNIX error exception: 24
     syspolicyd: [com.apple.syspolicy:default] Failed to generate SecStaticCode for <private> error: 100024
     

     spctl --assess --type execute on an unrelated, notarized app returns Too many open files.

   • The kernel then denies new processes of other apps. Google Chrome renderer helpers stayed at _dyld_start (96 KB footprint) and ASP denied them, so no tab loaded. OrbStack was denied 90 times in 3 minutes.

   • sudo killall syspolicyd helps only briefly. The relaunched syspolicyd logged EMFILE errors again after the next sweep, about one minute later.

   So on macOS the size of each burst is the problem, not only the cost of each probe. I have not tested the stub workaround from the original report on macOS.
4. Torrey-Smith-165 commented on Oct 7, 2026

   

   Torrey-Smith-165

   on Oct 7, 2026

   More actions

   Another macOS data point. The behavior is unchanged at v0.0.46-nightly.20261006.2752 (d8d037eae); line numbers below are from that tag.

   Setup

   • T3 Code 0.0.46-nightly.20261005.2689, macOS, Apple Silicon. 194 projects, 1,158 PRs linked to 451 non-archived threads. glab and tea are not installed.
   • Remotes use SSH host aliases from ~/.ssh/config (git@github-work:owner/repo.git, git@github-personal:owner/repo.git), the same shape as [Bug]: Pull request pane treats SSH aliases as GitHub API hosts #6198. isGitHubHost (packages/shared/src/sourceControl.ts:188) needs a github DNS label, so these are unknown. GitHubCli.ts:502-504 and :512 already read a dotless alias containing github as github.com.
   • Most links sit on threads whose project root is a folder that holds several checkouts, not a checkout itself. None of the 194 project roots is a checkout with a github.com remote. requireProject (PullRequestService.ts:867) never finds the thread's own repository, so it always falls through to listWorkspaceProjects({ host }) (:894). That lists every project and refines every unknown one (:679-753). resolveHandle gets an explicit context (:727), so it skips providerContextCache (SourceControlProviderRegistry.ts:261-264).
   • No read ever succeeds, so all 1,909 links still have a null snapshot. isDue returns true for them on every sweep (PullRequestSyncReactor.ts:174), and lastSyncedAt is only set after a successful read (:328).

   Measured from ~/.t3/userdata/logs/server.trace.ndjson*

   	10-06 15:32	10-06 18:27
   process spawns/s (glab + tea)	~190	~68
   ProjectEnrichmentService.request/s	~3,600	~3,360
   PullRequestService.canonicalRef/s	not measured	~17
   spans/s	~13,000	~12,400
   trace writes	~7 MB/s	~5.5 MB/s
   server process CPU	107-124%	45-70%

   The trace file rotates a full 10 MB every 1.5 to 2 s. An endpoint security scanner on the same machine ran at 200-290% CPU scanning the spawns and trace writes, and the whole machine lagged.

   What would help, in addition to #12071

   1. Cache a negative refinement per remote URL or host, as fix(server): stop re-probing hosting CLIs for unrecognised remotes #12071 does.
   2. Classify a dotless SSH alias containing github as GitHub in detectSourceControlProviderFromRemoteUrl, matching gitHubApiHostForRemote. Or resolve the alias with ssh -G <alias> and use its hostname.
   3. Don't enumerate every project per PR. Build a host and repository to project index once per sweep.
   4. Back off links whose read keeps failing, for example on the 15-minute cadence closed PRs get, instead of treating a null snapshot as due every minute.
5. treyecollins commented on Oct 7, 2026

   

   treyecollins

   on Oct 7, 2026

   More actions

   Another macOS data point, on the newest nightly. Here the probe storm makes the desktop app's own local server show " is reconnecting", with no remote environment involved.

   Environment

   • T3 Code Desktop 0.0.46-nightly.20261007.2774 (611132c171f3), macOS 26 (Darwin 25.6.0), Apple Silicon, Node v26.8.2
   • Desktop app against its bundled local server (127.0.0.1:3773)
   • 49 projects. 13 had remotes of the form git@github-wiens:org/repo.git, a dotless SSH alias for github.com. A per-directory includeIf gitconfig had an insteadOf rule rewriting git@github.com:org/ to that alias.
   • tea is not installed. glab is installed but not authenticated for that host.

   What happened
   The banner " is reconnecting" (ChatView.tsx environment-unavailable item) keeps appearing even though the server is local and its process is alive.

   Evidence from server.trace.ndjson

   • The trace rotates about every minute, roughly 10 MB/min.
   • In one minute of trace: 207 tea spawns (ForgejoCli.listLogins, all spawn tea ENOENT), 212 glab spawns, about 530 VcsProcess.run, 215 refineUnknownRemoteProvider, and 870 ProjectEnrichmentService.request.
   • PullRequestSyncReactor.sweep took 236s.
   • During bursts, /api/auth/session took 6s, and each /.well-known/t3/environment probe ran for 8 to 10s before it was interrupted inside ServerSecretStore.get. The /ws socket then closed and the banner appeared.

   Diagnosis
   Same as above. isGitHubHost (packages/shared/src/sourceControl.ts:188) needs a github DNS label, so github-wiens is classified unknown. A remote like hub.lumenfield.work-foo passes, because splitting on the dot yields a github label. When the caller passes context, resolveHandle bypasses providerContextCache (SourceControlProviderRegistry.ts:262), so the failed refinement is reprobed on every read. The #16321 fix for #6198 changed only the PR pane path, not this detection, so the problem persists on .2774.

   Workaround that worked: keep remotes on plain github.com and pin the SSH key per directory, rather than using a host alias. In the includeIf gitconfig:

   [url "git@github.com:org/"]
   	insteadOf = https://github.com/org/
   [core]
   	sshCommand = ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes

   Related: #16503 (same saturation blocking connection setup for a remote environment), #6198.

   Produced via t3 triage by Claude Code (Opus 5.5).