Skip to content

fix(desktop): cap xterm WebGL glyph-atlas GPU memory growth - #6352

Merged
Kitenite merged 5 commits into
mainfrom
fix-gpu-atlas-clamp
Aug 12, 2026
Merged

Kitenite merged 5 commits into
mainfrom
fix-gpu-atlas-clamp

Conversation

@Kitenite

@Kitenite Kitenite commented Aug 11, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Patches @xterm/addon-webgl (bun patch, beta.297) so the intended FORCED_MAX_TEXTURE_SIZE=4096 clamp actually applies (it was dead code using raw gl.MAX_TEXTURE_SIZE=16384, letting atlas pages merge-double toward 1GiB) and zeroes merged-away atlas canvases instead of leaving them for lazy GC.
  • Adds an app-side safety net: onAddTextureAtlasCanvas → clearTextureAtlas() after 32 page-adds in terminal-addons.ts.

Why / Context

SUPER-1793: the GPU helper leaks IOAccelerator/IOSurface unboundedly under high foreground-color cardinality — the real-world trigger is Claude Code's animated gradient "thinking" shimmer (continuous distinct RGB). External reporter hit 7.5GB in 32 minutes on an 8GB Mac; recovery only via GPU-process kill. Every distinct (glyph, fg) rasterizes a new atlas entry and each new glyph forces a full-page texImage2D respecification. (The upstream page-count clamp — "hunk C" — is already present in beta.297, which this PR bumps to.)

Manual QA (end-to-end, dev desktop over CDP, footprint on the dev app's GPU Helper, screenshots captured)

  • BEFORE: truecolor shimmer flood → 1.28GB → 10–11GB in 90s, unbounded.
  • AFTER: same flood for 6 min → bounded 1.9–2.4GB (~5×). Honest caveat: the 1.5GB plateau target was not met; if 2.1GB is still too high for 8GB Macs, lower the hunk-D reset threshold (32) — trade-off is more frequent atlas rebuilds.
  • Regressions: plain 8-color heavy output (yes flood) stays flat (2805→2718MB) and never trips the reset; text renders crisp after atlas resets (screenshots); normal typing/scrolling unaffected.

Risks / Rollout

  • The bun patch must be re-created on the next @xterm/addon-webgl version bump (it will fail loudly at install if the version changes).
  • Worth filing upstream: the dead clamp and lazy-GC'd merged canvases are xterm.js core defects (upstream HEAD only fixed out-of-range indexing).

Summary by cubic

Caps Desktop xterm WebGL glyph‑atlas GPU memory by clamping atlas textures to 4096 and resetting the atlas after bursts of truecolor output. Before, atlas pages could reach 1 GiB and GPU use spiked to 10–11 GB; now stress runs plateau around 1.9–2.4 GB with occasional clears.

  • Patch @xterm/addon-webgl@0.20.0-beta.297: enforce the 4096 clamp for glyph pages and the oversized‑glyph overflow page; eagerly free canvases for merged and evicted atlas pages.
  • App guard: subscribe to onAddTextureAtlasCanvas before addon activation; after 32 page adds, queue a clearTextureAtlas() via microtask to avoid clearing mid‑draw.
  • Guard rails: add patchedDependencies for the exact version; CI test apps/desktop/src/webgl-atlas-patch.test.ts asserts both clamps and both eager‑release sites; docs in patches/README.md.
  • Migration: when bumping @xterm/addon-webgl, regenerate or drop the patch and keep the test updated.

Written for commit 7db22a9. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes
    • Improved terminal rendering stability during extended sessions.
    • Reduced visual artifacts when displaying large amounts of text.
    • Improved graphics resource handling for smoother text rendering.
    • Added safeguards to refresh terminal rendering resources without interrupting active text rendering.
    • Improved WebGL rendering reliability when terminals display complex or extensive text.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c620b512-b1e2-4361-ab8a-6be7298c74c2

📥 Commits

Reviewing files that changed from the base of the PR and between 4bf519f and 7db22a9.

📒 Files selected for processing (1)
  • apps/desktop/src/webgl-atlas-patch.test.ts

📝 Walkthrough

Walkthrough

The desktop terminal tracks WebGL texture-atlas growth and defers atlas clearing after 32 page additions. The root package configuration applies and verifies a local patch to @xterm/addon-webgl@0.20.0-beta.297.

Changes

WebGL texture atlas reset

Layer / File(s) Summary
Configure and verify the xterm WebGL patch
package.json, apps/desktop/src/webgl-atlas-patch.test.ts, patches/README.md
The package configuration applies the local patch. Tests verify the CommonJS and ESM bundles. Documentation describes the patch and its regeneration and removal criteria.
Defer repeated atlas resets
apps/desktop/src/renderer/lib/terminal/terminal-addons.ts
The terminal addon tracks atlas page additions. After 32 additions, it resets the counter and queues texture-atlas clearing with queueMicrotask.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TerminalAddon
  participant WebGLAtlas
  participant MicrotaskQueue
  TerminalAddon->>WebGLAtlas: Observe atlas page additions
  WebGLAtlas-->>TerminalAddon: Report each page addition
  TerminalAddon->>MicrotaskQueue: Queue clearing after 32 additions
  MicrotaskQueue->>WebGLAtlas: Clear atlas after glyph rasterization
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title uses conventional commit format and clearly states that the change limits xterm WebGL glyph-atlas GPU memory growth.
Description check ✅ Passed The description explains the problem, implementation, testing results, risks, and upgrade procedure, although it does not reproduce the template checklist.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-gpu-atlas-clamp

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
apps/desktop/src/renderer/lib/terminal/terminal-addons.ts (1)

87-95: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Coalesce atlas resets that are already queued.

When one synchronous render produces 64 or more page-add events, this code queues multiple microtasks before the first reset runs. Each callback then invokes clearTextureAtlas() after the atlas has already been cleared. Track one pending reset with a boolean.

The upstream WebGL renderer also clears its model and requests a viewport redraw for each call. (raw.githubusercontent.com)

Proposed coalescing change
 let atlasPageAdds = 0;
+let atlasResetQueued = false;
 webglAddon.onAddTextureAtlasCanvas(() => {
 	if (++atlasPageAdds >= ATLAS_PAGE_ADDS_BEFORE_RESET) {
 		atlasPageAdds = 0;
+		if (atlasResetQueued) return;
+		atlasResetQueued = true;
 		// Defer: the event fires mid-glyph-draw; clearing synchronously
 		// would wipe the atlas under the in-flight rasterization.
-		queueMicrotask(() => webglAddon?.clearTextureAtlas());
+		queueMicrotask(() => {
+			atlasResetQueued = false;
+			if (!disposed) webglAddon?.clearTextureAtlas();
+		});
 	}
 });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/desktop/src/renderer/lib/terminal/terminal-addons.ts` around lines 87 -
95, Update the atlas reset logic in the webglAddon.onAddTextureAtlasCanvas
callback to track whether a reset microtask is already pending, queue at most
one reset at a time, and clear the pending state when that microtask runs before
calling webglAddon?.clearTextureAtlas(). Preserve the existing page-add
threshold and counter reset behavior.

Source: MCP tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@apps/desktop/src/renderer/lib/terminal/terminal-addons.ts`:
- Around line 87-95: Update the atlas reset logic in the
webglAddon.onAddTextureAtlasCanvas callback to track whether a reset microtask
is already pending, queue at most one reset at a time, and clear the pending
state when that microtask runs before calling webglAddon?.clearTextureAtlas().
Preserve the existing page-add threshold and counter reset behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3797458e-21e6-4001-8cfc-fa3067af0d6b

📥 Commits

Reviewing files that changed from the base of the PR and between 9b12164 and fa3c60a.

⛔ Files ignored due to path filters (1)
  • bun.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • apps/desktop/src/renderer/lib/terminal/terminal-addons.ts
  • package.json
  • patches/@xterm%2Faddon-webgl@0.20.0-beta.297.patch

@github-actions

github-actions Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

🔗 Preview Links

Service Status Link
Neon Database (Neon) ✅ View Branch
Vercel API (Vercel) ✅ Open Preview
Vercel Web (Vercel) ✅ Open Preview
Vercel Marketing (Vercel) ✅ Open Preview
Vercel Admin (Vercel) ✅ Open Preview
Vercel Docs (Vercel) ✅ Open Preview

Preview updates automatically with new commits

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 4 files

You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="patches/@xterm%2Faddon-webgl@0.20.0-beta.297.patch">

<violation number="1">
P2: A wide or oversized glyph can still allocate a 16384² overflow atlas page despite this new 4096 clamp. The forced limit should also apply to the `deviceMaxTextureSize` passed into `acquireTextureAtlas`, or the overflow-page allocation should use the clamped value.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant Term as Terminal Instance
    participant Addons as terminal-addons.ts
    participant WebGL as WebGL Addon
    participant Patch as Patched Addon Code
    participant Atlas as Glyph Atlas
    participant GPU as GPU Memory

    Note over Term,GPU: WebGL Glyph Rendering Pipeline

    Term->>Addons: load terminal addons
    Addons->>WebGL: Create WebGL addon instance
    WebGL->>Term: Register addon

    Note over Term,GPU: Normal Glyph Rendering (per distinct glyph/fg combo)

    Term->>WebGL: render glyph data
    WebGL->>Atlas: Check for existing glyph entry
    alt Glyph exists
        Atlas-->>WebGL: Cache hit
        WebGL-->>GPU: Draw from existing atlas texture
    else New glyph/color combo
        WebGL->>Patch: requestNewPage()
        Patch->>GPU: Query actual max texture size
        alt Max texture size > 4096 (e.g., 16384)
            Patch->>Patch: CHANGED: Cap page size at FORCED_MAX_TEXTURE_SIZE=4096
            Patch->>Atlas: Add new page with capped dimensions
        else Max texture size <= 4096
            Patch->>Atlas: Add new page with native max size
        end
        Atlas->>GPU: Upload full page via texImage2D
        WebGL-->>GPU: Rasterize glyph into atlas page
    end

    Note over Addons: Memory Growth Protection

    loop Throughout rendering
        Patch->>Addons: CHANGED: Fire onAddTextureAtlasCanvas event
        Addons->>Addons: Increment atlasPageAdds counter
        alt atlasPageAdds >= 32
            Addons->>Addons: Reset counter to 0
            Addons->>Addons: queueMicrotask(clearTextureAtlas)
            Note over Addons: Deferred to avoid disrupting in-flight draw
            Addons->>WebGL: CHANGED: clearTextureAtlas()
            WebGL->>Atlas: Clear all atlas pages
            Atlas->>GPU: CHANGED: Release textures immediately
            WebGL-->>Addons: Atlas cleared
            Addons-->>Term: Refresh display
        end
    end

    Note over Patch,GPU: Memory Cleanup (after atlas merge/eviction)

    Patch->>Atlas: Merge/evict old atlas pages
    Atlas->>GPU: Detect merged-away canvases
    alt Merged canvas exists
        Patch->>Patch: CHANGED: Zero canvas buffer
        Patch-->>GPU: Release memory immediately (no lazy GC)
        GPU-->>Atlas: Memory freed
        Atlas-->>WebGL: Update texture references
    end

    Note over GPU: Bounded memory (~1.9-2.4GB max vs 10-11GB before)
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

@@ -0,0 +1,71 @@
diff --git a/lib/addon-webgl.js b/lib/addon-webgl.js

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: A wide or oversized glyph can still allocate a 16384² overflow atlas page despite this new 4096 clamp. The forced limit should also apply to the deviceMaxTextureSize passed into acquireTextureAtlas, or the overflow-page allocation should use the clamped value.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At patches/@xterm%2Faddon-webgl@0.20.0-beta.297.patch:

<comment>A wide or oversized glyph can still allocate a 16384² overflow atlas page despite this new 4096 clamp. The forced limit should also apply to the `deviceMaxTextureSize` passed into `acquireTextureAtlas`, or the overflow-page allocation should use the clamped value.</comment>

Truecolor-heavy TUI output (e.g. Claude Code's shimmer) mints a glyph
variant per RGB fg, growing the WebGL atlas without bound (SUPER-1793).

- patch @xterm/addon-webgl: enforce the intended 4096 FORCED_MAX_TEXTURE_SIZE
  clamp (raw gl.MAX_TEXTURE_SIZE=16384 allows 1GiB pages) and zero
  merged-away/evicted atlas canvas dims so backing stores free immediately
  instead of on lazy GC
- app safety net: clear the texture atlas after 32 page-add events
…age-add

Address cubic review on #6352:
- P2: WebglRenderer's deviceMaxTextureSize also feeds the oversized-glyph
  overflow page (TextureAtlas allocates it at that size), so an unclamped
  16384 still allowed a 1GiB page; clamp it to 4096 like the glyph renderer
- P3: subscribe to onAddTextureAtlasCanvas before loadAddon so the page
  added during WebGL activation is counted toward the reset threshold
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

…n bumps

patchedDependencies keys are version-pinned, so bumping @xterm/addon-webgl
silently drops the SUPER-1793 fix while everything still builds. Add
patches/README.md with the regeneration procedure and a bun test asserting
the patch markers in the installed bundles (verified to fail on pristine).

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔇 Additional comments (3)
apps/desktop/src/webgl-atlas-patch.test.ts (2)

1-24: LGTM!

Also applies to: 28-29


25-27: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

⚠️ Unverified finding
Sandbox verification was unavailable.

Cover both canvas-release paths in the guard.

Lines 25-27 only require one canvas.width = 0 match per bundle. The documented patch must release canvases in both the merge path and _evictAllPages, and must reset both width and height. A partial patch can therefore pass this test and reintroduce GPU memory growth.

Assert both cleanup sites, or count the expected width and height resets.

Suggested guard
 		test(`${name} frees merged-away/evicted atlas canvases eagerly`, () => {
-			expect(src).toMatch(/canvas\.width\s*=\s*0/);
+			expect(src.match(/canvas\.width\s*=\s*0/g) ?? []).toHaveLength(2);
+			expect(src.match(/canvas\.height\s*=\s*0/g) ?? []).toHaveLength(2);
 		});

Verify the installed bundles with:

patches/README.md (1)

1-57: LGTM!


ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9617f777-dd9e-4aab-b25d-2df2a3440af5

📥 Commits

Reviewing files that changed from the base of the PR and between ced5afb and 4bf519f.

📒 Files selected for processing (2)
  • apps/desktop/src/webgl-atlas-patch.test.ts
  • patches/README.md

@Kitenite
Kitenite merged commit b7dc3da into main Aug 12, 2026
12 of 13 checks passed
@Kitenite
Kitenite deleted the fix-gpu-atlas-clamp branch August 12, 2026 22:44

This branch was previously deployed

1 inactive deployment
preview — 7db22a94 Deployed Aug 12, 2026 by Kitenite via Deploy API #7561
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant