fix(grok): read the CLI's own completed-turn usage instead of estimating it

CHANGELOG.md

@@ -16,6 +16,7 @@
 - **DeepSeek Harness (`dsh`) is now a supported provider.** Reads DeepSeek's open-source agent harness from `~/.dsh/sessions` (`DSH_HOME` relocates the root), both the default zstd logs and the uncompressed `session.jsonl` variant. A `.zstd` log is a concatenation of independent zstd frames, one per write batch, so it is decoded frame by frame behind a structural frame scan and a torn trailing frame from a crashed writer is ignored rather than failing the file (needs Node 22.15+ for `zlib` zstd; below that dsh is skipped with a notice instead of counted as $0). One call per `(turn, step)`, with the step's final `assistant/message` usage superseding the streamed `assistant/chunk` sample of the same call rather than adding to it, the model taken from the message that served the step, and reasoning tokens billed at the output rate. DSH records tokens but no cost, so calls are priced from the shared tables. The events a forked session replays from its parent are skipped, since codeburn already counts the parent's own log. The session format is pinned at version 0 upstream with no compatibility implied, so a log stamped with any other version is skipped with a notice instead of read under today's assumptions.
 
 ### Changed
+- **Grok Build now reads the CLI's own completed-turn usage instead of estimating it.** Usage comes from the `turn_completed.usage` records Grok CLI already writes into `updates.jsonl` (`inputTokens`, `outputTokens`, `cachedReadTokens`, `cacheCreationTokens`, `reasoningTokens`), deduplicated by `prompt_id` and emitted as one session-level call from the top-level totals. The previous parser reconstructed an estimate from the running `_meta.totalTokens` context counter, so **existing Grok totals will change materially on upgrade** - on one real 568-session corpus cache-read went from 150K to 96.3M tokens, total tokens from 20.0M to 113.9M, and cost from $36.98 to $56.79. Cache read and cache creation are subsets of input and reasoning is a subset of output, so reasoning is clamped to the record's reported output and split back out to match this repo's exclusive-reasoning contract. `modelUsage` only selects a priced attribution id; multi-model rate attribution stays out of scope, so one session is priced at one model's rate. `costUsdTicks` is ignored because its scale is undocumented. Sessions with no usable record - older CLI versions - keep the old context-curve heuristic and stay flagged estimated. **In a session that has at least one `turn_completed` record, turns without one are not counted at all** (their tokens are dropped rather than estimated), and the session is marked estimated instead of claiming full provider coverage. Cached Grok sessions re-parse once. The daily cache re-derives once on first run after upgrade: this is a global re-derivation of every day and every provider, since the daily cache has no per-provider invalidation, but it reads the warm session cache rather than re-parsing transcripts, so it costs seconds (~3s on the corpus above), and the superseded cache file is retained on disk as the baseline for days no source can still re-derive. (#998)
 - **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now pending BYTES alone (200 MB), not file count: 250 pending files holding under a megabyte between them spawned threads that made the run ~5% slower, while a few hundred huge rollouts were being turned away. The count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, and the per-thread memory budget is derived per parse as `clamp(256 MB, 2 × average pending file + 128 MB, 1 GB)` rather than a flat 256 MB — a 260 MB rollout peaks near 430 MB in its worker and scales linearly with the pool, so the flat figure over-subscribed exactly the workload this adds. The decision is per provider, and at most one pool is alive at a time.
 - **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with under 200 MB behind the pending whole-file re-parses, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why.
 - **A warm launch rewrites only the month that changed, and a ranged query reads only the months it can report on.** Per-provider shards still meant one appended session republished that provider's entire history — 95 MB for Claude on a 6 GB corpus. Each provider's shard is now split again by the UTC month of the cached session's FIRST turn, a bucket that never moves as a session grows, so an append rewrites one month. Every shard records the newest month it holds, which lets `--period today/week` skip the shards that cannot contribute a turn to the range; the skipped months stay on disk untouched across the save, and providers whose cache is the only surviving record (durable) or whose parse fingerprint moved are always read in full. Remaining shards are read concurrently. Existing v8 and v7 caches are re-laid-out losslessly on first load and the old layout removed once the new one is published: nothing re-parses.
@@ -29,6 +30,7 @@
 - **The resident `codeburn serve` child.** The first real panel request is also the cache warm-up, so startup never runs an artificial warm-up query beside a duplicate one-shot child; each served command carries its own read-only option allowlist, and anything outside it falls back to a normal spawn; the child exits when its stdin closes, so it can never outlive the app. Requests whose response exceeds the 16 MiB frame limit still replace the child, but that deliberate kill no longer spends the resident's unexpected-death budget. (#972)
 
 ### Fixed
+- **Context-bloat detection now counts reasoning tokens as generated output.** `detectContextBloat` divided context by `totalOutputTokens` alone, but reasoning is stored beside output rather than inside it, so for every reasoning-bearing provider the detector saw a fraction of the tokens actually generated and invented findings - a session whose real ratio was 20:1, under the 25:1 threshold, was reported as 133:1 and "high impact". It now uses the same `output + reasoning` sum the reports use, which corrects grok, codex, kiro, hermes, qwen and cursor-agent alike.
 - **The unpriced-models warning in the dashboard is now readable at every terminal width.** It lived in a fixed-width panel with an inline model list and a fix command, so it clipped mid-name at 80 columns and clipped *earlier* at 200, where the three-column layout narrows each panel - neither the affected models nor a runnable command survived. The panel line is now a pointer, `! N unpriced: codeburn models --unpriced` (shortened to `! N: codeburn models --unpriced` below 45 columns of panel), and the model list moves to that command's plain output, which is full width, copyable, and lists every model rather than the first two. The command's hint no longer reads as an unconditional instruction to alias: a subscription or flat-rate model is correctly $0, and mapping it onto another model's per-token rate would invent spend that was never billed. Provider-supplied model IDs are now stripped of terminal control characters in every human-readable report rather than only on the unpriced path, and `--unpriced` shows raw IDs instead of friendly names because `model-alias` keys on the raw ID. (#969)
 - **`codeburn models --unpriced --top N` returned nothing for a `--top N` smaller than the number of priced models.** `--top` is applied inside `aggregateModels`, before the unpriced filter, on rows sorted cost-first — and unpriced rows are $0 on both, so they sorted last and the slice removed exactly the rows the flag exists to show. A user with unpriced models was told they had none. The slice now runs after the filter — and after ranking, because unpriced rows tie at $0 on both keys, so slicing them in aggregate order kept whichever models happened to appear earliest in the transcript rather than the largest. The order now matches the one the unpriced-models warning shows. (#969)
 - **Old durable sources remain visible while they still exist.** The 90-day session-cache age-out now applies only after a durable source disappears from discovery, so an unchanged older Copilot source keeps reporting usage and reuses its persisted fingerprint instead of being reparsed and immediately discarded. (#987) On long-lived machines this makes previously dropped history reappear, so lifetime totals can jump once after upgrading.

docs/providers/grok.md

@@ -4,7 +4,7 @@ Grok Build, xAI's coding CLI. Sessions use the `grok-build` model by default.
 
 - **Source:** `src/providers/grok.ts`
 - **Loading:** eager (`src/providers/index.ts`)
-- **Test:** `tests/providers/grok.test.ts`
+- **Test:** `tests/grok-parser-pipeline.test.ts`, `tests/providers/grok.test.ts`
 
 ## Where it reads from
 
@@ -13,33 +13,38 @@ Grok Build, xAI's coding CLI. Sessions use the `grok-build` model by default.
 
 ## Storage format
 
-JSON + JSONL. `summary.json` holds the session id, cwd, timestamps, and `current_model_id`. `signals.json` holds `modelsUsed`, `toolsUsed`, and `contextTokensUsed`. `updates.jsonl` is the ACP log: each streamed chunk carries `params._meta.totalTokens` (running context size) and `params._meta.promptId` (one per turn).
+JSON + JSONL. `summary.json` holds the session id, cwd, timestamps, and `current_model_id`. `signals.json` holds `modelsUsed`, `toolsUsed`, and `contextTokensUsed`. `updates.jsonl` is the ACP log: streamed chunks carry `params._meta.totalTokens` (running context size) and `params._meta.promptId` (one per turn); newer CLI versions also append `params.update.sessionUpdate: "turn_completed"` with snake_case `prompt_id` and a provider-recorded `usage` object.
 
 ## Token model
 
-**Estimated.** Grok does not log billable input/output tokens. It only records the running context fill (`totalTokens` per chunk, and `contextTokensUsed` in signals). The parser reconstructs a rough estimate from the per-turn `totalTokens` curve: input is the context entering each turn, output is the context growth during it. The result is flagged `costIsEstimated` and re-priced with `calculateCost`.
+**Authoritative when available.** A `turn_completed` record reports the whole input side in `inputTokens` and the whole output side in `outputTokens`. `cachedReadTokens` is treated as a subset of input; `cacheCreationTokens` is treated as another input subset by analogy because the record exposes no separate fresh-input field, with any per-record violation clamped locally. The top-level totals are the accounting basis. `modelUsage` is retained only as a model-attribution signal; multi-model attribution is deliberately out of scope, so one session uses one selected model's rate. `reasoningTokens` is clamped to that record's reported output before the parser emits exclusive output plus reasoning, preserving the reported total through the cache pipeline. These counts are provider-recorded, so `costIsEstimated` is false for fully covered sessions while CodeBurn applies its own pricing table. `costUsdTicks` is ignored because its scale is undocumented.
+
+**Estimated fallback.** Older sessions without any valid `turn_completed.usage` record use the running context fill (`totalTokens` per chunk) and the existing compaction-aware per-turn curve. That path remains flagged `costIsEstimated`; a completed record is never blended with the heuristic.
+
+**Mixed sessions undercount.** The choice between the two paths is per session, not per turn. If a session has at least one usable `turn_completed` record, the whole session is billed from the summed records and any turn WITHOUT a record contributes nothing at all - its tokens are dropped, not estimated, so such a session reads low. The row is marked `costIsEstimated: true` rather than claiming full provider coverage. This is deliberate: blending the heuristic into real records would reintroduce the roughly 5x output over-count this parser exists to remove. It happens when a session straddles a CLI upgrade or a run dies before writing its last record; an open turn is filled by a later parse once it writes one, pre-upgrade turns never are. Measured on a 568-session corpus, 1 turn out of 566 was uncovered.
 
 ## Pricing
 
-`grok-build` is aliased to `grok-build-0.1` in `src/models.ts`, so it prices off the bundled LiteLLM fallback. Note that xAI's published API rate and the LiteLLM fallback figure differ, so treat the cost as an estimate and verify against your xAI usage console.
+`grok-build` is aliased to `grok-build-0.1` in `src/models.ts`, so it prices off the bundled LiteLLM fallback. If `usage.modelUsage` contains a model id that CodeBurn can price, that id is preferred; when the real id is not priced yet, the existing summary/signals model is retained so a known alias does not become a $0 row. This is a single attribution choice for the session, not a per-model accounting split; multi-model rate attribution is a follow-up. CodeBurn still does not use Grok's undocumented `costUsdTicks`.
 
 ## Caching
 
-None.
+Authoritative records expose cache-read and cache-creation token counts. The legacy estimate has no cache-creation signal and keeps its inferred cache-read count.
 
 ## Deduplication
 
 Per `grok:<session-dir>:<updated_at>:<id>`.
 
 ## Quirks
 
-- **No cache or output/tool-token split.** Only context fill is available, so cache fields are `0` and the cost is an estimate (likely an upper bound, since re-sent context is cached server-side and not exposed in the session files).
+- **Two token paths.** Completed turns carry provider usage; sessions from older CLI versions have only the context curve and therefore remain estimates (likely an upper bound, since re-sent context is cached server-side and not exposed in those files).
+- **A turn with no `turn_completed` record is dropped inside an otherwise-covered session** (see Token model). The session still reports, marked estimated, but reads low by those turns.
 - **No bash-command capture.** Tool names come from `signals.toolsUsed`; per-command bash text is not extracted, so `bashCommands` is empty.
 - **Whole-session timestamp.** Spend is attributed to `updated_at`, since the context curve is cumulative.
 - **Subscription vs API.** Grok Build runs via either a metered xAI API account (tiered) or a SuperGrok subscription; the session files do not record which.
 
 ## When fixing a bug here
 
 1. Discovery: check the `sessions/<cwd>/<uuid>/` walk and the `GROK_HOME` resolution.
-2. Token estimate: see `estimateTokens` (groups `updates.jsonl` by `promptId`).
+2. Token accounting: see `parseUpdates` (deduplicates `turn_completed` by snake_case `prompt_id`, then falls back to grouping streamed chunks by camelCase `_meta.promptId`).
 3. Add a fixture-format session under `tests/providers/grok.test.ts`; do not mock the filesystem.

src/daily-cache.ts

@@ -6,6 +6,20 @@ import { join } from 'path'
 import { getCodeburnCacheDir } from './cache-dir.js'
 import type { DateRange, ProjectSummary } from './types.js'
 
+// Bumped to 19: Grok authoritative usage now keeps one session-level rollup
+// from top-level totals, clamps reasoning to reported output, and labels mixed
+// authoritative/heuristic coverage. Every day finalized under the previous
+// accounting carries the old Grok totals, and the daily cache has no
+// per-provider invalidation, so raising MIN_SUPPORTED_VERSION is the only
+// lever: it forces a one-time re-derivation of ALL days, for every provider,
+// not just Grok. That pass reads the warm session cache (CACHE_VERSION is
+// unchanged and only PROVIDER_PARSE_VERSIONS.grok moved), so it costs seconds
+// rather than a full re-parse, and adoptOlderDailyCaches keeps the superseded
+// file as the baseline for days no source can still re-derive.
+//
+// The shipped predecessor is v17; v18 was an unreleased draft of this change
+// and only exists in pre-release checkouts. 19 clears both.
+//
 // Bumped to 17: copilot CLI sessions were misclassified as VS Code transcripts
 // (#944), so days finalized at v16 or earlier carry output-only copilot costs —
 // the session.shutdown rollup's input/cache tokens were dropped. Raising
@@ -74,8 +88,8 @@ import type { DateRange, ProjectSummary } from './types.js'
 // that older binaries skipped. v8 added local-model savings to the daily
 // rollup; the `savingsConfigHash` field is invalidated separately when the
 // user changes their `localModelSavings` mapping.
-export const DAILY_CACHE_VERSION = 17
-const MIN_SUPPORTED_VERSION = 17
+export const DAILY_CACHE_VERSION = 19
+const MIN_SUPPORTED_VERSION = 19
 // Version-suffixed so different binaries each own a distinct file and never
 // clobber an incompatible schema. Bumping the version mints a fresh filename;
 // adoptOlderDailyCaches then unions days out of every previous file (including

src/optimize.ts

@@ -3195,7 +3195,9 @@ export function findContextBloatCandidates(projects: ProjectSummary[]): ContextB
 
     for (const session of sessions) {
       const inputTokens = sessionEffectiveContextTokens(session)
-      const outputTokens = session.totalOutputTokens
+      // Reasoning is stored separately from ordinary output, but both are
+      // generated tokens for this detector. Reports already use their sum.
+      const outputTokens = session.totalOutputTokens + session.totalReasoningTokens
       const ratio = inputTokens / Math.max(outputTokens, 1)
       const currentMs = new Date(session.firstTimestamp).getTime()
       const gapMs = previousTimestampMs !== null ? currentMs - previousTimestampMs : null