Skip to content

feat(tui): Hidden Ctrl+T HUD showing running/final LLM token cost - #783

Merged
Twixes merged 9 commits into
mainfrom
feat/token-cost-hud
Jul 1, 2026
Merged

Twixes merged 9 commits into
mainfrom
feat/token-cost-hud

Conversation

@Twixes

@Twixes Twixes commented Jul 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Not a big problem, but in development, I'm curious about the costs of a wizard run. Haven't found a way to see it.

Changes

Proposing a token cost bar at the top, default on in development, hidden in prod. Although in both cases toggleable with Ctrl+T toggle. Shows the current wizard run's accumulated token usage and USD cost estimate.

Screenshot 2026-07-01 at 13 27 57@2x

Verification

Had Claude do a bunch of tests. Ran this locally.

Adds an undiscoverable Ctrl+T toggle (not registered as a keyboard
hint) that shows the current wizard run's accumulated token usage and
USD cost estimate, updating live as the agent works and settling to
the SDK's exact total_cost_usd once the run completes.

Note: Ctrl+M (as originally asked) can't work here -- in terminal raw
input Ctrl+M and Enter/Return are the same byte (0x0D), and this
codebase never enables the Kitty keyboard protocol that could
disambiguate them (checked start-tui.ts's render() call). Ctrl+T has
no such collision and nothing else in the repo uses it.

- token-pricing.ts: pricing table + cost formula extracted from the
  benchmark's CostTrackerPlugin into a shared module, so the live HUD's
  per-turn estimate and the --benchmark report can't drift apart.
  Behavior-identical refactor (cost-tracker.ts re-exports it locally).
- WizardStore: tokenUsage/tokenHudVisible observation state.
  addTokenUsage accumulates each assistant turn's usage (mirrors
  TokenTrackerPlugin/CacheTrackerPlugin's extraction, minus SDK-retry
  dedup -- approximate by design for a debug HUD, not a billing
  record). setFinalTokenCostUsd reconciles to the SDK's authoritative
  total_cost_usd on completion, same trick CostTrackerPlugin.onFinalize
  uses.
- agent-interface.ts: wires both into the existing assistant-message
  loop and completeWithSuccess -- no new code paths, just two more
  getUI() calls alongside the existing analytics capture.
- ScreenContainer: the Ctrl+T useInput + TokenCostHud render live here
  (not on any screen), since it's the one component mounted for the
  whole process. contentHeight is budgeted for the HUD's one row so it
  can't push content past the terminal height.
- Guarded all 6 existing 'any key dismisses/exits this screen'
  handlers (OutroScreen and friends) against key.ctrl/key.meta -- Ink
  calls every mounted useInput for a keypress with no stopPropagation,
  so without this, toggling the HUD on e.g. the outro screen would
  also dismiss it (and OutroScreen's dismissal chains into
  process.exit()).
- exit-line.ts: the final tally is also echoed into post-exit
  scrollback unconditionally, so it's guaranteed visible even if the
  HUD was never opened or a screen transition raced the toggle.
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci error-tracking-upload-source-maps
  • /wizard-ci mcp-analytics
  • /wizard-ci misc
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci error-tracking-upload-source-maps/android
  • /wizard-ci error-tracking-upload-source-maps/cicd-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-nested-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-single-stage-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-gitlab-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-pnpm-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-raw-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-ssh-vps-node-raw
  • /wizard-ci error-tracking-upload-source-maps/flutter
  • /wizard-ci error-tracking-upload-source-maps/ios
  • /wizard-ci error-tracking-upload-source-maps/next
  • /wizard-ci error-tracking-upload-source-maps/next-no-posthog
  • /wizard-ci error-tracking-upload-source-maps/node-raw
  • /wizard-ci error-tracking-upload-source-maps/node-rollup
  • /wizard-ci error-tracking-upload-source-maps/node-rollup-typescript-plugin
  • /wizard-ci error-tracking-upload-source-maps/node-webpack
  • /wizard-ci error-tracking-upload-source-maps/nuxt-3-6
  • /wizard-ci error-tracking-upload-source-maps/nuxt-4-3
  • /wizard-ci error-tracking-upload-source-maps/react-native
  • /wizard-ci error-tracking-upload-source-maps/react-vite
  • /wizard-ci error-tracking-upload-source-maps/rust
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci misc/quack-quack
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

Twixes added 3 commits July 1, 2026 12:26
Separates the HUD from the screen content below it. Bumped
contentHeight's budget from 1 to 2 rows to account for the extra line.
Two changes to the hidden Ctrl+T HUD:

1. Defaults visible for local/dev/test runs (WizardStore's
   $tokenHudVisible now inits to IS_DEV instead of false), so
   contributors see it without needing to know the shortcut. Still
   defaults hidden in the published build. The panel now also names
   its own shortcut ('Ctrl+T to hide'), so it's self-documenting
   either way -- bumped ScreenContainer's contentHeight budget from 2
   to 3 rows for the extra hint line.

2. Fixes a real pricing gap: the cost formula was a single flat table
   (Sonnet rates) applied to every token regardless of which model
   actually produced it. Two ways that's wrong in practice --
   AgentConfig.modelOverride switches specific programs to Haiku (e.g.
   source-map detection), and the SDK reports a model string on every
   individual assistant turn because a subagent dispatched via the
   Agent tool can run on a different model than the main session. A
   Haiku turn priced as Sonnet would show a running estimate ~3-15x
   too high.

   token-pricing.ts now has Haiku/Sonnet/Opus price tables and
   pricePerMtokForModel(), matched by substring so a dated version
   suffix never breaks the match. agent-interface.ts reads
   message.message.model (the BetaMessage's own model field) per turn
   -- not the run's nominal model -- and threads it through a new
   shared TokenUsageDelta type (WizardUI/InkUI/LoggingUI/store) into
   computeTokenCostUsd. Unknown/missing model falls back to Sonnet
   pricing, the same behavior as before this change.

   Deliberately NOT threaded into the benchmark's CostTrackerPlugin --
   MiddlewareContext has no model field to pass, and --benchmark always
   measures the default flow, never a Haiku-overridden program. Left a
   comment there; both paths still reconcile to the SDK's authoritative
   total_cost_usd at the end regardless.
…seInput calls

The 6 'any key dismisses/exits this screen' handlers each had their own
inline if (key.ctrl || key.meta) return; guard, copy-pasted with the
same comment. Centralizes that into a shared hook so:

- The guard can't be forgotten on the next screen that needs this
  pattern (AuditAreaPane.tsx's unrelated 'o' key handler shows this
  mistake is easy to make -- it checks input but not key.ctrl, so a
  hypothetical Ctrl+O would trigger it; not fixed here, out of scope,
  but it's a live example of the risk).
- isIgnoredForDismiss is a pure, directly unit-tested predicate,
  matching how keyboard-hints-utils.ts's matchesKey is tested --
  Ink-wiring itself stays untested, consistent with the rest of this
  hooks directory.

No behavior change for any of the 6 screens.
@Twixes Twixes changed the title feat(tui): hidden Ctrl+T HUD showing running/final LLM token cost feat(tui): Hidden Ctrl+T HUD showing running/final LLM token cost Jul 1, 2026
Twixes added 4 commits July 1, 2026 13:29
The exit-line tally previously showed unconditionally whenever a run
produced any usage, regardless of whether the user ever saw the HUD.
That meant a production run (where the HUD defaults hidden) would
still get a cost number printed to scrollback out of nowhere.

tokenCostLine now checks store.tokenHudVisible first -- the tally only
survives into scrollback if the panel is actually showing at exit
(which defaults on in dev/test via IS_DEV, off in production, and is
always overridable with Ctrl+T either way).

Also: 'Ctrl+T to hide' now shares the cost line's row when there's
room, right-aligned with a minimum 2-column gap, falling back to its
own row on a narrow terminal. tokenCostHudRowCount is exported so
ScreenContainer's height budget (1 HUD row + spacer, or 2 + spacer)
can never disagree with what TokenCostHud actually renders -- both
call the exact same function with the exact same inputs.
The previous pricePerMtokForModel matched by family-name substring
(model.includes('opus')/('haiku')), which silently mis-prices any
model that shares a family name but not a price: Claude Opus 4.5 is
$5/$25 per Mtok, half of what the table charged (it had Opus 4.1's
$15/$75 rate hardcoded), and Claude Sonnet 5 is $2/$10, cheaper than
Sonnet 4.6's $3/$15 -- both would have been silently mis-priced by
the substring match.

Replaced with an exact-match table keyed by the *undated* model id
(e.g. 'claude-haiku-4-5', not 'claude-haiku-4-5-20251001') --
pricePerMtokForModel strips a trailing 8-digit release-date suffix
before falling back to the table, since Anthropic's dated and undated
ids for the same version always carry the same price, verified for
every entry against https://models.dev/api.json. A model id that's
given but not in the table now returns undefined (computeTokenCostUsd
treats that as $0 for the delta) rather than guessing from a family
name -- the live estimate can undercount for an unrecognized model,
but never mis-prices one the way the substring match did; the final
number is unaffected either way since it reconciles to the SDK's
total_cost_usd regardless.

Also dropped the unused PRICE_PER_MTOK export (nothing outside this
file referenced it) and rounded pricingFromInput's derived cache rates
to 6dp to avoid binary float noise (3 * 0.1 = 0.30000000000000004).

Considered adopting tokenlens (an existing npm package wrapping
models.dev) instead of hand-maintaining this table, but its published
static catalog doesn't have any of the three model ids this wizard
actually uses -- only models.dev's *live* API does, which would mean
a network dependency for what's a hidden debug HUD. Kept the local
table; decision documented in the module's doc comment.

Also removes a handful of tests added earlier in this branch that
were verification scaffolding rather than real regression coverage:
duplicate assertions of already-covered code paths, and a couple of
near-tautological checks (0 tokens costs $0; a longer string needs
more width).
No behavior change -- four cleanups across the branch's own diff:

- agent-interface.ts: extracted the 'assistant' case's inline usage-
  extraction (type cast + field mapping) into a named
  extractTokenUsageDelta() helper, matching this file's existing
  pattern of small private helpers above handleSDKMessage. Shrinks the
  case block from ~30 lines to 2.
- TokenCostHud.tsx: tokenCostLineParts() was being computed twice per
  render (once directly, once again inside the tokenCostHudRowCount()
  call used to decide inline vs. two-row layout), each independently
  recomputing lineLength. Consolidated into one layoutTokenCostLine()
  that both the exported row-count helper and the component now share.
- store.ts / TokenCostHud.tsx / exit-line.ts: the "sum all four token
  counters" formula was duplicated verbatim in TokenCostHud and
  exit-line. Both already import from store.ts, so added
  totalTokenCount(usage) there instead of introducing a new shared
  module (which would've needed a type-only import back into
  token-pricing.ts, adding a cycle for no real benefit).
- token-pricing.ts: computeTokenCostUsd took 7 positional parameters,
  several the same type -- an easy mis-ordering hazard at call sites.
  Converted to a single options object shaped like TokenUsageDelta, so
  store.ts's addTokenUsage can now just pass computeTokenCostUsd(delta)
  directly instead of re-listing its fields in a fixed order.
  cost-tracker.ts's two call sites and the test file's calls updated to
  match; also dropped its local computeCost alias in favor of calling
  computeTokenCostUsd directly, since it only had two call sites.
Sonnet 5's $2/$10 rate isn't its permanent price -- it's a promotional
discount that reverts to Sonnet 4.6's rate ($3/$15) on 2026-09-01.
Hardcoding $2/$10 as a static table entry would have silently
undercharged every Sonnet 5 turn forever after the promo ends, exactly
the kind of staleness this table is meant to avoid.

Pulled 'claude-sonnet-5' out of the static PRICE_TABLE into a small
sonnet5Price(now) helper that checks the cutoff date, wired through
pricePerMtokForModel's new optional `now` param (defaults to the real
clock; injectable so the promo-window tests are deterministic instead
of depending on whichever date the suite happens to run on).

Also fixed a comment that had crept 1-2 characters past this repo's
80-char printWidth.

@edwinyjlim edwinyjlim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is really slick!

ty for making it so isolated and excluding it from prod build ❤️

@Twixes
Twixes enabled auto-merge (squash) July 1, 2026 22:30
@Twixes
Twixes merged commit b7ef4ba into main Jul 1, 2026
17 checks passed
@Twixes
Twixes deleted the feat/token-cost-hud branch July 1, 2026 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants