Skip to content

feat(cache): cost-saved telemetry on cache hit #88

Description

@moonming

Background

The DP currently logs cache_status=hit|miss on every chat request. The dashboard /usage page can show "N hits today" but cannot answer the only question operators actually care about:

"How much did caching save me?"

There is no way today to attribute saved tokens or saved USD to cache hits. Without that number every other piece of cache work (TTL tuning, applies_to scoping, semantic cache, etc.) is unjustifiable to finance / leadership.

Proposed fix

On a cache hit, the DP already has the cached ChatResponse, which carries usage.prompt_tokens / completion_tokens (and the four detail counters added for cache pricing). Emit two extra fields on the cache-hit telemetry event:

  • tokens_saved_input = cached usage.prompt_tokens
  • tokens_saved_output = cached usage.completion_tokens
  • cost_saved_usd = price_table.lookup(model) applied to the saved tokens

The price table already exists in cp-api (internal/dpmgr/dpstore/pricing.go); the DP either receives a snapshot via the existing config watcher or computes from the resolved aisix_core::Model pricing fields, whichever is already wired up for non-cache cost reporting.

Where the fields land

  • DP structured log: hit event gets tokens_saved_input, tokens_saved_output, cost_saved_usd fields.
  • Existing telemetry exporters (OTel, Langfuse) receive them automatically as part of the same span / event.
  • Usage rollup table (requests / usage_daily in cp-api) gains a cost_saved_usd column populated from the hit events; a follow-up dashboard PR adds the "Saved this week: $X" tile to /usage.

Why this delivers value

  1. Direct ROI dashboard — converts the abstract "hits" counter into a dollar number that sells the product internally.
  2. Operations decisions become data-driven — TTL tuning, policy applies_to scope, semantic-cache adoption all become "did $ saved go up?" instead of guesswork.
  3. Per-tenant cost attribution — group cost_saved_usd by api_key_id and the dashboard can show "we saved you $Y this month" on customer-facing billing pages.

Acceptance criteria

  • On a cache hit, the chat dispatch path emits tokens_saved_input, tokens_saved_output, cost_saved_usd (computed from the cached usage + the model's price entry).
  • On a cache miss, those fields are absent / zero (no double-counting).
  • Pricing lookup falls back gracefully (zero, not error) when the model has no price entry.
  • Unit test: cache hit dispatch produces the expected three fields with values matching the cached usage × pricing.
  • Existing cache-hit path still serves the cached response (no perf regression).

Out of scope

  • Dashboard wiring of the new field into a "Saved this week" tile — separate cp-api + dashboard PR after the DP fields land.
  • Per-token-class differential pricing (cached_prompt vs reasoning) — first cut uses base prompt + completion rates, refinement is a follow-up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-value differentiatorenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions