feat(tui): render agent-generated visualizations in chat #2088

Problem

RustCode renders assistant messages through the TUI's Markdown pipeline. Fenced blocks are displayed as source/code, and tool results have a separate text-oriented transcript renderer. There is no message representation or rendering path for an agent to present a useful visual interaction directly in the conversation. As a result, charts, diagrams, exploratory controls, and other task-specific views fall back to prose, ASCII art, or Markdown tables.

T3 Code recently shipped agent-created visualizations in its thread. This is a useful reference for the interaction we want—not a requirement to copy its web stack or execute arbitrary HTML in RustCode.

Reference implementation research

• T3 Code #7220 — render Codex visualizations inline (closed): parses Codex visualization references in assistant messages, validates that the referenced bounded temporary HTML artifact belongs to that thread/message, issues content-revision-bound signed URLs, and runs the HTML in a sandboxed iframe with no network access. Fenced examples remain ordinary Markdown. This is a file-backed artifact path, not arbitrary HTML embedded in the transcript.
• T3 Code #11460 — render themed HTML visualizations inline (closed before merge during the provider/orchestration V2 rewrite): proposes a provider-agnostic t3-html fence rendered inline, with theme synchronization, resizing, source/reset controls, and sandboxing. The proposal sanitizes markup, blocks generated scripts/navigation/external resources, and uses narrowly authorized bridge scripts for measurement/theme. Treat it as design evidence, not as the shipped PR.
• T3 Code #19654 — skeleton and cached HTML renders (merged Oct. 15): illustrates follow-on work for layout-aware loading and caching; extracting a bounded page outline also makes renders easier to preview before loading the full page.
• T3 Code #18897 — HTML render links follow chat link behavior (merged Oct. 13) and #18898 — preserve app shortcuts when an HTML render has focus (merged Oct. 14): demonstrate that embedding interactive content also requires deliberate navigation and keyboard/focus integration.
• T3 Code #19029 — show pages published inside Code Mode scripts (closed) and #19796 — recover Antigravity html_render pages (open when checked Oct. 16): highlight a separate protocol concern: a page must remain attached to the originating thread even when an agent wraps tool calls or a provider omits the tool result from its stream.

RustCode codebase observations

• rustcode/tui/src/ui/assistant_render.rs owns assistant-message presentation and currently routes fenced blocks through code/plain-text rendering; it already has careful handling for partial streams, fence length, and copy behavior.
• rustcode/tui/src/ui/markdown.rs is the structured Markdown renderer, with bounded caching and explicit table rendering.
• rustcode/tui/src/ui/history_cell.rs caches committed assistant rendering by width/theme and keeps an incremental Markdown cache for streaming responses.
• rustcode/tui/src/ui/scrollback.rs deliberately holds incomplete fences and unstable Markdown blocks until they are safe to commit. Any visual block must preserve this streaming/scrollback contract.
• rustcode/tui/src/ui/conversation_render.rs dispatches finalized assistant and tool messages through separate paths. rustcode/tui/src/ui/tool_result.rs bounds tool-result transcript output; visualization output should not bypass its own size/resource limits or be smuggled in as a tool-result side effect.
• The repository has a ratatui/crossterm terminal frontend and currently has no HTML, iframe, or WebView rendering path. T3's iframe implementation cannot be transplanted into the TUI as-is.

Proposal

Add a first-class, bounded visualization block to assistant conversation output so the agent can choose a fit-for-purpose view instead of always emitting a Markdown table. Keep the canonical assistant text/protocol backward compatible, and make the renderer capability explicit: the terminal should render supported visualizations using native terminal widgets; an HTML/WebView implementation, if desired for a future graphical frontend, must be separately designed and sandboxed rather than silently executing HTML in the terminal.

A practical first slice is a small allowlisted schema with a few high-value widgets (for example, chart, key/value or sortable data table, and diagram), rendered natively in ratatui with keyboard-operable controls and textual fallback. The wire format and block boundaries should be structured and versioned, not inferred from arbitrary prose. The agent should be able to provide labels, data, accessible text/summary, and a clear fallback. Unsupported, malformed, oversized, or incomplete blocks remain visible as source or a concise fallback instead of disappearing.

Do not make arbitrary HTML/JavaScript execution, general-purpose component code, or cross-provider artifact recovery a prerequisite for the MVP. Consider those as follow-up work after the protocol, UX, and threat model are agreed.

Acceptance criteria

• Decide and document the first supported format and widget allowlist, versioning, maximum payload/data sizes, and fallback behavior.
• Define how visual blocks are represented in assistant content/tool protocol without breaking existing providers, stored sessions, copy, or export behavior.
• Parse only complete, recognized blocks; malformed and unsupported blocks render safely as ordinary source/fallback text. Normal code fences and Markdown tables are unchanged.
• Render the initial supported widgets in the TUI using native ratatui primitives; provide keyboard navigation/selection where interactive and a useful non-interactive textual representation.
• Integrate with live streaming and finalized transcript paths without duplicate rows, flicker, unbounded re-rendering, or committing a partial visual block to terminal scrollback.
• Keep assistant-only rendering semantics explicit. Tool output or user-provided content cannot cause executable content to run.
• Set hard limits for dimensions, item counts, text lengths, and render cost; truncate with an explicit indication rather than monopolizing the viewport.
• Add tests for incremental/partial blocks, malformed payloads, oversized inputs, unknown schema versions/widgets, width changes, theme changes, and parity between live and committed rendering.
• Document the format and add a small example prompt/result demonstrating when the agent should choose a visualization over Markdown.
• If a future GUI renderer executes HTML, require a separate security review and implement strict origin/path authorization, size limits, CSP/sandbox isolation, no ambient filesystem/network access, safe bridge messages, and explicit navigation/focus handling. Do not treat sanitization alone as a security boundary.

Open design questions

1. Should the first milestone be only native structured widgets (cross-platform and safe by construction), or should it include a desktop-only sandboxed HTML artifact viewer?
2. Should visualizations arrive as a typed assistant-content block, a dedicated tool result/artifact referenced by an assistant message, or both? The rendering layer must be able to associate an artifact with the correct thread/message and reject unreferenced paths.
3. Which initial widget types materially help coding tasks beyond the existing Markdown table, code, diff, and diagram conventions?
4. What should copy/export do for an interactive block: copy its source/schema, its textual summary, or offer both?

Related RustCode areas

• rustcode/tui/src/ui/assistant_render.rs
• rustcode/tui/src/ui/markdown.rs
• rustcode/tui/src/ui/history_cell.rs
• rustcode/tui/src/ui/scrollback.rs
• rustcode/tui/src/ui/conversation_render.rs
• rustcode/tui/src/ui/tool_result.rs