feat(cache): cost-saved telemetry — cache_hit_saved_*_tokens on hits - #92
Merged
Conversation
…hits Closes #88. Adds two new fields to UsageEvent (and threads them through Success + UsageExtras + emit_usage_event): cache_hit_saved_input_tokens = cached usage.prompt_tokens (on hit) cache_hit_saved_output_tokens = cached usage.completion_tokens (on hit) Both are zero on miss / disabled / error. cp-api derives cost_saved_usd on ingest by multiplying these counters by its model-pricing catalog — matching the existing pattern where the DP records tokens and cp-api owns USD (see chat.rs:629 comment on cost_usd). Keeping pricing on the cp-api side means a price change is a single deploy and the dashboard gets "tokens saved" + "USD saved" off the same column. Existing prompt_tokens / completion_tokens behavior on hit rows is unchanged (still mirrors the cached usage). The new field is additive so cp-api / dashboard rollups can choose to either (a) keep using the existing prompt/completion column with a cache_status='hit' filter or (b) sum the dedicated saved counters without a filter — the explicit column is unambiguous and avoids the "tokens that didn't actually cost anything are in the same bucket as tokens that did" trap. Regression test in aisix-proxy: cache_hit_emits_saved_token_counters_on_telemetry_event exercises the live router + capturing UsageSink, asserts saved=0 on the miss event and saved=(7,11) on the hit event mirroring the mock upstream's usage block.
There was a problem hiding this comment.
Pull request overview
This PR extends DP-side per-request telemetry (aisix_obs::UsageEvent) with explicit “tokens saved by cache hits” counters, and threads them through the chat completion success path so cp-api can compute cost_saved_usd on ingestion.
Changes:
- Add
cache_hit_saved_input_tokens/cache_hit_saved_output_tokensfields toUsageEvent(omitted from JSON when zero). - Populate these fields on the cache-hit path (and ensure non-hit paths default to zero) by threading through
Success -> UsageExtras -> emit_usage_event. - Add a regression test asserting miss vs hit telemetry behavior for the new counters.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| crates/aisix-proxy/src/lib.rs | Adds a regression test verifying miss/hit events and the new saved-token counters. |
| crates/aisix-proxy/src/chat.rs | Threads and emits the new saved-token counters from dispatch success paths into UsageEvent. |
| crates/aisix-obs/src/usage.rs | Introduces the new UsageEvent fields with serde defaults/omission when zero. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+559
to
+562
| // filter) lets cp-api compute `cost_saved_usd` | ||
| // without joining on the status enum. | ||
| cache_hit_saved_input_tokens: prompt.try_into().unwrap_or(u32::MAX), | ||
| cache_hit_saved_output_tokens: completion.try_into().unwrap_or(u32::MAX), |
Comment on lines
787
to
+791
| guardrail_blocked, | ||
| guardrail_bypassed_reason: extras.bypass_reason, | ||
| cache_status: extras.cache_status, | ||
| cache_hit_saved_input_tokens: extras.cache_hit_saved_input_tokens, | ||
| cache_hit_saved_output_tokens: extras.cache_hit_saved_output_tokens, |
Comment on lines
828
to
+832
| /// fired. Goes onto `dpmgr_usage_events.cache_status`. | ||
| cache_status: String, | ||
| /// On a cache HIT, the cached response's prompt + completion | ||
| /// tokens. Zero otherwise. cp-api derives `cost_saved_usd` on | ||
| /// ingest from these + its pricing catalog (see #88). |
Comment on lines
+155
to
+161
| /// cp-api derives `cost_saved_usd` server-side by multiplying these | ||
| /// counters by the model's pricing (same pattern as `cost_usd` on | ||
| /// non-cache rows — the DP doesn't own the pricing catalog). | ||
| /// Surfacing tokens (not USD) here keeps pricing changes a cp-api- | ||
| /// only deploy and lets the dashboard show "tokens saved" too. | ||
| #[serde(default, skip_serializing_if = "is_zero_u32")] | ||
| pub cache_hit_saved_input_tokens: u32, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #88.
Summary
Adds two new fields to the per-request
UsageEvent:cache_hit_saved_input_tokens— cachedusage.prompt_tokenson hit, zero otherwisecache_hit_saved_output_tokens— cachedusage.completion_tokenson hit, zero otherwiseThreaded through
Success→UsageExtras→emit_usage_eventso every hit-path code path populates them and every miss / disabled / streaming / error path defaults them to zero.Why tokens, not USD
cp-api already owns the pricing catalog and recomputes
cost_usdserver-side on ingestion (see the existingchat.rs:629comment: "cp-api recomputes cost server-side from its pricing catalog when ingesting telemetry; the DP just records 0.0 on the wire"). Keepingcost_saved_usdderivation on the cp-api side matches that pattern:cost_usdcp-api will derive
cost_saved_usd = cache_hit_saved_input_tokens × input_price + cache_hit_saved_output_tokens × output_pricein a follow-up PR; that work is out of scope for this DP-side change.Why a dedicated column instead of reusing prompt_tokens
prompt_tokens/completion_tokensare still populated on hit rows (matches existing dashboard rollups, no wire-format break). But adding the explicitcache_hit_saved_*pair means:SUM(cache_hit_saved_*_tokens × price)with nocache_status='hit'filterExisting fields are untouched — this is purely additive.
Tests
New regression test in
aisix-proxy/src/lib.rs:cache_hit_emits_saved_token_counters_on_telemetry_eventexercises the live router with a capturing
UsageSinkand asserts:cache_status="miss", both saved counters = 0cache_status="hit", saved counters = (7, 11) mirroring the mocked upstream's usage blockprompt_tokens/completion_tokenson the hit event also still mirror the cached usage (additive contract)Test plan
cargo test -p aisix-proxy --lib cache_hit_emits(1/1 green)cargo test --workspace --lib(469 / 469 green; was 468 + 1 new)cargo clippy --workspace --all-targets -- -D warningscargo fmt --all -- --check(rustfmt 1.8.0)