copilot provider: cache hit rate is always 0.0% when using GitHub Copilot CLI (sessions misidentified as VS Code transcripts) #944

When using GitHub Copilot CLI as the LLM harness (sessions stored in ~/.copilot/session-state/), codeburn always reports 0.0% cache hit rate and 0 cached / 0 written tokens, even though cache activity is clearly occurring (confirmed by inspecting the raw JSONL files).

Cache hit  0.0%

Tokens
┌─────────────┬─────────────┬─────────┐
│  Type       │     Tokens  │  Share  │
├─────────────┼─────────────┼─────────┤
│  Input      │  2,132,219  │    78%  │
│  Output     │    605,462  │    22%  │
│  Cache in   │          0  │     0%  │
│  Cache out  │          0  │     0%  │
└─────────────┴─────────────┴─────────┘

From the raw JSONL, a single session shows massive cache usage (e.g. cacheReadTokens: 1,404,178, cacheWriteTokens: 45,101), so the data is clearly there — codeburn is just not reading it.

Root Cause

File: src/providers/copilot.ts, createJsonlParser(), first-pass detection loop (~line 727)

if (ev.type === 'session.start') {
  const data = ev.data as SessionStartData & { producer?: string }
  if (data.producer === 'copilot-agent') {
    isTranscript = true   // ← BUG: this also matches Copilot CLI sessions
  }
  break
}

The producer: 'copilot-agent' value is used to detect VS Code transcript format (which does NOT emit a session.shutdown rollup with token data). However, the GitHub Copilot CLI also sets producer: 'copilot-agent' in its session.start events:

{
  "type": "session.start",
  "data": {
    "producer": "copilot-agent",
    "copilotVersion": "1.0.74",
    "context": { "cwd": "/...", "gitRoot": "/..." }
  }
}

Every single one of the 189 ~/.copilot/session-state/*/events.jsonl files on the test system has producer: 'copilot-agent'.

Because isTranscript is set to true for all of them, the shutdown handler is skipped:

if (event.type === 'session.shutdown') {
  if (isTranscript) continue  // ← ALWAYS hits for Copilot CLI sessions
  // ...reads cacheReadTokens, cacheWriteTokens, inputTokens from modelMetrics
}

The session.shutdown event is the only place the Copilot CLI records input tokens, cache-read tokens, and cache-write tokens. Skipping it means all three values are permanently 0 for every Copilot CLI session.

Evidence from Raw JSONL

A real session.shutdown event from a Copilot CLI session (~/.copilot/session-state/*/events.jsonl):

{
  "type": "session.shutdown",
  "data": {
    "shutdownType": "routine",
    "modelMetrics": {
      "claude-sonnet-5": {
        "usage": {
          "inputTokens": 1449317,
          "outputTokens": 3431,
          "cacheReadTokens": 1404178,
          "cacheWriteTokens": 45101,
          "reasoningTokens": 473
        }
      }
    },
    "tokenDetails": {
      "cache_read":  { "tokenCount": 1486816 },
      "cache_write": { "tokenCount": 973672 }
    }
  }
}

Expected cache hit calculation for this session:

• net input = 1,449,317 − 1,404,178 − 45,101 = 38
• cache read = 1,404,178
• cache hit rate = 1,404,178 / (38 + 1,404,178) ≈ ~100%

Actual reported: 0.0%

How to Distinguish Copilot CLI from VS Code Transcripts

The two formats can be differentiated by additional fields in session.start.data:

Proposed Fix Options

Option A – Check for copilotVersion in session.start.data (least invasive):

if (data.producer === 'copilot-agent' && !data.copilotVersion) {
  isTranscript = true
}

Option B – Do not skip session.shutdown even in transcript mode; only process it if modelMetrics is present (VS Code transcripts won't have it):

if (event.type === 'session.shutdown') {
  // VS Code transcripts don't write modelMetrics; Copilot CLI does.
  if (isTranscript && !isRecord(shutdownData.modelMetrics)) continue
  ...
}

Option C – Two-pass: detect CLI format if the first assistant.message event has non-zero outputTokens (VS Code transcripts have outputTokens: 0).

Impact

• Cache read/write tokens are always 0 for Copilot CLI users
• Cache hit rate is always 0.0%
• Input token counts are also 0 (only output tokens from assistant.message are captured in transcript mode)
• Cost is significantly underreported — in the example session above, cost from output-only was $0 for input vs. actual cost including ~1.4M cache-read tokens

Reproduction Steps

1. Use GitHub Copilot CLI (not VS Code) — sessions appear in ~/.copilot/session-state/
2. Run several agentic sessions so session.shutdown events are written
3. Run codeburn overview --period week
4. Observe Cache in: 0 / Cache out: 0 / Cache hit: 0.0%
5. Confirm the raw JSONL has non-zero cacheReadTokens in session.shutdown.data.modelMetrics.<model>.usage

# Quick verification command:
grep "session.shutdown" ~/.copilot/session-state/*/events.jsonl \
  | python3 -c "
import json, sys
for line in sys.stdin:
    path, _, data = line.partition(': ')
    d = json.loads(data)
    mm = d.get('data', {}).get('modelMetrics', {})
    for model, metrics in mm.items():
        usage = metrics.get('usage', {})
        print(f'{model}: cacheRead={usage.get(\"cacheReadTokens\",0):,}')
"

Environment

• codeburn version: 0.9.19
• GitHub Copilot CLI version: 1.0.74
• OS: macOS Darwin (arm64)
• Node.js: v22.18.0
• OTel DB: Not present (VS Code not configured with github.copilot.chat.otel.dbSpanExporter.enabled)