Skip to content

runtime: dynamic outcome_unknown system notice breaks prompt caching; drop it, keep only the tool result #5924

Closed

Description

@me2seeks

What happened

When a run seals with an unknown tool outcome (dispatch without response), the next explicit user turn rebuilds the provider request with two additions (ai-sdk-turn.ts:1502-1510):

  1. a synthetic outcome_unknown tool result beside the unmatched call — request-local, and required: an unmatched tool_use is a provider 400, verified live (an assistant message with 'tool_calls' must be followed by tool messages… on one provider, No tool output found for function call on another);
  2. a dynamic system notice appended to the stable system prompt (generated in inspectPriorUnknownToolOutcomes, model-history.ts:249-257), with per-event rows (- bash (operation …)).

Because the notice changes per event and sits in the system prompt — the first segment of the request — it moves the divergence point to the very front. On strict-prefix prompt-caching providers (Anthropic per its documentation; OpenAI-family measured below), the entire conversation history behind it loses its cache on exactly the turns that are already paying to replay that history.

Measured against live providers (~22k-token requests, warm cache, then one variant):

request variant Kimi k3-256k GPT-family via OpenAI-compatible relay
identical repeat 100% cached ~100% cached
system prompt + appended notice (current design) 86% 0%
same change carried in the tool result instead 89% 87%

Behavior A/B (does the notice change what the model does?), three arms — (A) synthetic tool result only, (B) result + dynamic notice (current), (C) result + fixed generic rule:

  • Neutral "please continue" scenario: all arms 5/5 verify-state-first on both k3-256k and gpt-6-luna.
  • Pressured "just rerun the delete" scenario (k3-256k, N=10/arm): A 1/10, B 0/10, C 5/10.

The dynamic notice showed no measurable behavioral benefit on either model, so its cache cost buys nothing.

How to reproduce

  1. Run a turn whose tool dispatch is admitted, then kill the host before the result lands (recovery seals it outcome_unknown).
  2. Send a fresh explicit user message.
  3. The provider request's system prompt now carries the per-event notice; on a strict-prefix provider the cached prefix of the whole history is invalidated (observable via provider usage cached-token fields).

Environment

  • Maka version or commit: 1e80e3b (main)
  • Surface: Runtime Host (provider request assembly)

Logs, screenshots, or additional context

Proposed change: keep appendPriorUnknownToolResponses exactly as-is (required for well-formed history), drop the system notice:

  • remove the systemNotice fragment from the system-prompt join in ai-sdk-turn.ts;
  • narrow PriorUnknownToolOutcomeProjection's projected variant (drop systemNotice) and stop generating it in model-history.ts;
  • update the tests that assert the notice (session-manager.test.ts, ai-sdk-backend.test.ts), keeping coverage that the synthetic tool result is still projected.

Refs #5671 — that broader redesign (abandon the old branch, fork fresh turns) would subsume this; this issue is the smaller, independently shippable step that keeps current projection semantics.

Retirement semantics (unknownOutcomeRetired) are intentionally out of scope.

Activity

  1. self-assigned this
    on Oct 2, 2026
  2. me2seeks commented on Oct 2, 2026

    @me2seeks
    ContributorAuthor

    Claiming this — I ran the measurements cited above and will implement the change (drop the notice join, narrow the projection type, update tests) in a PR shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions