Skip to content

feat(ai): Classify LLM cache usage - #127015

Merged
vgrozdanic merged 2 commits into
masterfrom
feat/llm-cache-classification
Oct 12, 2026
Merged

vgrozdanic merged 2 commits into
masterfrom
feat/llm-cache-classification

Conversation

@vgrozdanic

@vgrozdanic vgrozdanic commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Adds provisional classification for the LLM prompt-cache shadow detector. Call sites with enough traffic and stored evidence are classified as healthy, not caching, or repeatedly writing cache entries without reading them back.

Classification uses cache hit rate, cache creation volume, prompt size, call volume, and stored-span count. All models use a 4,096-token prompt threshold because savings below that size are not material enough to report. This avoids model-specific policy and maintenance.

call-site aggregates
→ eligibility checks
→ healthy | not caching | cache thrash
→ validation probes in the next stack layer

These results remain provisional. This layer does not query telemetry, schedule detection, emit logs, or create issues. Later stack layers validate traffic spacing and instrumentation before treating a classification as a shadow finding.

@vgrozdanic
vgrozdanic added this pull request to stack #127016 October 9, 2026 11:46
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 9, 2026
Comment on lines +16 to +18
# https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
# https://platform.openai.com/docs/guides/prompt-caching
# https://ai.google.dev/gemini-api/docs/caching

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in order to keep this up to date, we'll set up Junior periodic task for now, there's no other way of having this information at the moment

@vgrozdanic
vgrozdanic marked this pull request as ready for review October 9, 2026 11:52
@vgrozdanic
vgrozdanic requested a review from a team as a code owner October 9, 2026 11:52
Base automatically changed from feat/llm-cache-domain to master October 12, 2026 15:27
@vgrozdanic
vgrozdanic force-pushed the feat/llm-cache-classification branch from 92b310e to 6db179a Compare October 12, 2026 15:28
@vgrozdanic
vgrozdanic merged commit d0becf1 into master Oct 12, 2026
65 checks passed
@vgrozdanic
vgrozdanic deleted the feat/llm-cache-classification branch October 12, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants