Skip to content

fix(buzz-agent): Responses reasoning summary, Anthropic display:summarized, ACP v2 messageId - #5195

Merged
wpfleger96 merged 4 commits into
mainfrom
duncan/thinking-summary-mode
Aug 7, 2026
Merged

fix(buzz-agent): Responses reasoning summary, Anthropic display:summarized, ACP v2 messageId#5195
wpfleger96 merged 4 commits into
mainfrom
duncan/thinking-summary-mode

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 7, 2026

Copy link
Copy Markdown
Member

Three pre-existing gaps in the buzz-agent observer feed fixed together per Will's ruling ("all 3 in the current PR"):

  1. OpenAI/DBv2-GPT routeresponses_body never requested reasoning.summary; GPT-family models billed thinking tokens but returned summary: [].
  2. Anthropic/DBv2-Claude routeanthropic_thinking_config() never sent thinking.display; newest Claude models (Opus 5, Sonnet 5, Fable 5, Mythos 5, Opus 4.7/4.8, Mythos Preview) default to display:"omitted", returning thinking blocks with an empty thinking field — observer rendered nothing.
  3. ACP v2 compliance — buzz-agent negotiates ACP v2 but emitted agent_thought_chunk and agent_message_chunk without messageId, which ACP v2's ContentChunk requires (messageId + content both required at schema head d13d1baa).

Changes

crates/buzz-agent/src/config.rs

  • New ThinkingSummary enum (Auto/Concise/Detailed) with BUZZ_AGENT_THINKING_SUMMARY env var (default Auto); mirrors BUZZ_AGENT_THINKING_EFFORT pattern
  • anthropic_thinking_config() now emits "display": "summarized" in both the adaptive shape and the manual-budget shape whenever thinking is enabled
  • Rewrote is_adaptive_thinking_model and anthropic_thinking_config doc comments to match Anthropic's exact three-way per-model terminology (doc: https://platform.claude.com/docs/en/build-with-claude/thinking-troubleshooting#supported-models):
    • Opus 4.6/4.7/4.8, Sonnet 4.6: Off — thinking OFF by default; type:"adaptive" required to enable
    • Opus 5, Sonnet 5: On — thinking on by default, can be disabled; we still send type:"adaptive" to activate output_config.effort
    • Fable 5, Mythos 5, Mythos Preview: Always on — thinking cannot be disabled; we still send type:"adaptive" to activate output_config.effort

crates/buzz-agent/src/llm.rs

  • responses_body emits reasoning.summary alongside reasoning.effort when effort is set (gated — no bare reasoning:{summary} without effort)
  • Covers both the pure-OpenAI Responses path and the DBv2 GPT-family Responses path

crates/buzz-agent/src/agent.rs

  • agent_thought_chunk carries "messageId": format!("{run_id}-thought-{round}")
  • agent_message_chunk carries "messageId": format!("{run_id}-message-{round}")
  • The two IDs are distinct (thought and assistant are two logical messages per the ACP v2 Message ID RFD)
  • run_id is a fresh random token per session/prompt invocation so IDs are session-unique across multiple prompts

crates/buzz-agent/src/lib.rs

  • run_id plumbed into RunCtx (was already generated in run_prompt, just not threaded through)

crates/buzz-agent/tests/golden_transcripts.rs

  • test_acp_v2_chunks_carry_message_id — negotiates v2, drives two consecutive session/prompt calls, asserts: both chunk types carry non-empty messageId; thought and message IDs are distinct; IDs do not recur across the two prompts in the same ACP session

desktop/src-tauri/src/managed_agents/env_vars.rs

  • BUZZ_AGENT_THINKING_SUMMARY added to is_safe_to_reveal allowlist

desktop/src-tauri/src/commands/agent_config_tests.rs

  • Tests for BUZZ_AGENT_THINKING_SUMMARY allowlist entry (case-insensitive)

Tests added

  • parse_thinking_summary_round_trips_all_values
  • parse_thinking_summary_unset_and_empty_yield_auto
  • parse_thinking_summary_is_case_insensitive
  • parse_thinking_summary_rejects_unknown_value
  • thinking_summary_as_str_mapping
  • responses_body_summary_present_iff_effort_set
  • responses_body_emits_configured_summary_mode
  • responses_body_concise_summary_mode
  • anthropic_thinking_config_adaptive_emits_display_summarized
  • anthropic_thinking_config_manual_budget_emits_display_summarized
  • test_acp_v2_chunks_carry_message_id (integration test — two-prompt cross-session case)

Notes

  • DBv2 gateway parity for display: unverified — the DBv2 Claude route proxies Anthropic Messages shape, but whether the gateway passes thinking.display through is not confirmed. Flagged here rather than blocking on it.
  • buzz-acp and Desktop TS are unchanged — they already parse messageId as optional and will pick it up from the wire automatically.
  • Chat Completions and OpenRouter paths: untouched.

… reasoning summaries

The OpenAI Responses API only populates reasoning summary arrays when a
summary mode is explicitly requested via `reasoning.summary`. Without it,
GPT-family models through buzz-agent bill thinking tokens but return
`summary: []`, leaving the observer feed empty even with effort set.

Changes:
- config.rs: new `ThinkingSummary` enum (Auto/Concise/Detailed) and
  `parse_thinking_summary` fn mirroring `parse_thinking_effort`; new
  `thinking_summary` field on `Config` (default Auto); wired via
  BUZZ_AGENT_THINKING_SUMMARY env var.
- llm.rs: `responses_body` now emits `reasoning.summary` alongside
  `reasoning.effort` when effort is set. No summary sent when effort is
  None — avoids 400s on non-reasoning models.
- env_vars.rs: BUZZ_AGENT_THINKING_SUMMARY added to is_safe_to_reveal
  allowlist (non-secret enum, same treatment as THINKING_EFFORT).
- agent_config_tests.rs: extended allowlist tests with THINKING_SUMMARY.

Also resolves the doc contradiction in is_adaptive_thinking_model vs
anthropic_thinking_config: the roster comment previously implied adaptive
models reason without any config ("always-on"), conflicting with the
anthropic_thinking_config doc saying `thinking:{type:"adaptive"}` is
required. Clarified: some models reason by default (Fable 5, Mythos 5,
Mythos Preview per the extended-thinking support table), but we always
send type:"adaptive" explicitly so that output_config.effort is honoured
and thinking token depth is predictable. Source:
https://platform.claude.com/docs/en/build-with-claude/extended-thinking

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 requested a review from a team as a code owner August 7, 2026 15:50
…accuracy

Three pre-existing gaps resolved per Thufir review findings on PR #5195.

Fix A — config.rs comment accuracy (Thufir finding 1a):
Rewrite the anthropic_thinking_config doc and is_adaptive_thinking_model
roster to match the per-model table at:
  https://platform.claude.com/docs/en/build-with-claude/thinking-troubleshooting#supported-models

Correct split:
- Opus 4.6/4.7/4.8, Sonnet 4.6: thinking OFF by default; type:adaptive required.
- Opus 5, Sonnet 5, Fable 5, Mythos 5, Mythos Preview: thinking ON with no config
  (always-on); we still send type:adaptive to activate output_config.effort.
Previous comments claimed all adaptive families needed the field to think —
true only for the first sub-bucket.

Fix B — Anthropic thinking display (Thufir finding 1b / root cause of observer gap):
Anthropic defaults thinking.display to omitted on newest models (Fable 5,
Mythos 5, Opus 5, Sonnet 5, Opus 4.8, Opus 4.7, Mythos Preview), returning
thinking blocks with an empty thinking field — parse_anthropic read nothing.
Fix: send display:summarized in anthropic_thinking_config() for both the
adaptive shape and the manual-budget shape whenever thinking is enabled.
Tests: adaptive and manual-budget families each assert display:summarized
present; unknown model path asserts thinking absent.
DBv2 gateway parity for display unverified — flagged in PR body.

Fix C — ACP v2 ContentChunk messageId compliance (Thufir ACP addendum):
buzz-agent negotiates ACP v2 but agent.rs emitted agent_thought_chunk and
agent_message_chunk without messageId. ACP v2 ContentChunk requires both
messageId and content (schema/v2/schema.json @d13d1baa); v1 allows the field,
so this addition is backwards-safe and buzz-acp/Desktop already parse it.
Fix: assign a stable round-scoped ID (format round-{n}) shared by the thought
and assistant message chunks from the same provider round. A provider round
produces at most one of each, so one ID per round is the right granularity.
Integration test: negotiates v2, drives agent through a Responses-route
reasoning response, asserts both chunk types carry identical non-empty
messageId plus correct content text.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96 wpfleger96 changed the title feat(buzz-agent): add BUZZ_AGENT_THINKING_SUMMARY for Responses-route reasoning summaries fix(buzz-agent): Responses reasoning summary, Anthropic display:summarized, ACP v2 messageId Aug 7, 2026
…ys-on terminology, lint clean

Three targeted repairs per Thufir round-2 review of PR #5195.

Fix 1 — ACP v2 messageId contract (Thufir IMPORTANT, corrects round-2 implementation):
The prior round-2 gave both the thought and assistant chunks from one provider
round the same messageId ('round-{n}'), which is wrong in two ways: (a) they are
two distinct logical messages per the ACP v2 Message ID RFD; (b) 'round' resets
per run() so 'round-1' recurred across session/prompt calls in the same ACP
session, violating the session-uniqueness requirement.

Fix: derive two IDs per round from the existing per-run run_id (already random
per session/prompt, lib.rs:842): '<run_id>-thought-<round>' and
'<run_id>-message-<round>'. Added run_id to RunCtx so agent.rs can access it
without a new dependency. Plumbed from lib.rs run_prompt (run_id was already in
scope, just not threaded through RunCtx).

Updated test_acp_v2_chunks_carry_message_id to assert the corrected invariants:
thought and message IDs are distinct, both are non-empty, and neither recurs
across a second session/prompt in the same ACP session (the cross-prompt case
the old test did not exercise).

Fix 2 — Three-way On/Always-on/Off terminology (Thufir IMPORTANT, completes Fix A):
Anthropic's support table uses three distinct statuses, not two. The round-2
pass collapsed all non-Off models into 'always-on', incorrectly calling Opus 5
and Sonnet 5 'always-on' (their status is 'On': default-on, can be disabled) and
calling Mythos Preview 'default-on' (its status is 'Always on': cannot be
disabled). Also, the inline branch comment at config.rs:167 still said 'thinking
must be explicitly enabled', contradicting On/Always-on models.

Corrected all four locations: config.rs top-of-function doc (sub-bucket list),
is_adaptive_thinking_model doc (sub-bucket list), inline branch comment, and
inline code comments at the Fable 5/Mythos 5/Mythos Preview match arms.

Fix 3 — doc_lazy_continuation lint (Thufir IMPORTANT, fixes failing CI gate):
Three clippy::doc_lazy_continuation errors at the continuation lines after the
nested list in the anthropic_thinking_config doc comment. Added a blank doc line
after the list items to terminate the list context before the continuation prose.

Source: Anthropic thinking-troubleshooting#supported-models table (On/Always-on
distinction); ACP Message ID RFD at d13d1baa lines 63-67, 115-124, 256-263.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on Wes's behalf at exact head 47ef696d6094d5fa4adcdc5a0a3dbe63de0a2727. No blocking findings.

I traced all three changes through request construction, provider-response parsing, and ACP emission:

  • OpenAI Responses: reasoning.summary is emitted only alongside configured reasoning.effort; parse_responses already consumes every summary_text item. Default auto is the provider-recommended way to select the most detailed summary supported by the model. Explicit concise/detailed remain operator choices because OpenAI documents model-dependent support.
  • Anthropic: display: "summarized" is valid in both type: "adaptive" and type: "enabled" shapes. Anthropic's current thinking documentation explicitly supports both modes and explains that newer-model defaults otherwise return an empty thinking field. The field remains absent when this code does not enable/configure thinking.
  • ACP v2: each emitted thought and assistant message gets a distinct ID, stable across its chunks and unique across rounds and prompt invocations because the prefix is the existing fresh per-prompt run_id. The two-prompt golden test pins the collision boundary that the earlier implementation missed.

The new env value is non-secret and its reveal allowlist/config parsing are covered. git diff --check passed. Full relevant CI is green (Rust Lint, Unit Tests, Desktop Core/build/smoke, Mobile, Windows Rust, cross-compiles, Security); two Desktop integration shards were still running when reviewed. The author-disclosed DBv2 Anthropic gateway passthrough uncertainty may affect whether that route gains observer text, but it does not create a regression in the native Anthropic path and is not a merge blocker.

Verdict: 9/10, safe to merge once required CI completes. This is a comment-only review, not an approval.

wesbillman
wesbillman previously approved these changes Aug 7, 2026
generate run_id before s.busy = true in acquire_session(). if
session_token() fails, propagate a static error and reject the prompt
without mutating session state. previously unwrap_or_else fell back to
"x" making every prompt in the session share the same run_id namespace,
recreating the within-session messageId collisions this PR eliminates.

three comment fixes ride along:
- config.rs:114: "both sub-buckets" -> "all three sub-buckets"
- config.rs:2333: "default-on" -> "Always on" for Mythos Preview test
- golden_transcripts.rs:822: remove overclaiming stability assertion
  (test collects one chunk per kind per prompt, not repeated chunks)

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 merged commit 742e8d1 into main Aug 7, 2026
26 checks passed
@wpfleger96
wpfleger96 deleted the duncan/thinking-summary-mode branch August 7, 2026 17:45
atishpatel added a commit that referenced this pull request Aug 7, 2026
…-log-harness

* origin/main:
  feat(desktop): adding rich link previews to messages (#3818)
  fix(buzz-agent): Responses reasoning summary, Anthropic display:summarized, ACP v2 messageId (#5195)
  fix(desktop): retain distinct agent instances in autocomplete (#5202)
  fix(desktop): defer channel visibility change to Save (#5203)
  feat(desktop): Projects follow-ups — access restrictions, fast loading, activity feed polish (#5073)
  refactor(cli): replace probe/decider/detail split with single typed extractor (#5191)
  fix(desktop): drop unhandled rejection from throwing window.Notification (#5143)
  fix(desktop): fence localStorage SecurityError from killing the React tree (#5142)
  fix(desktop): make terminal output selectable (#4980)
  fix(desktop): use WEBKIT_DMABUF_RENDERER_FORCE_SHM for NVIDIA/AppImage (#3654) (#4505)
  Make public starter channels best effort (#5192)
  Mobile: add anchored reaction popover (#5025)
  feat(mobile): add bee pull-to-refresh (#5059)

Signed-off-by: Atish Patel <atish@squareup.com>
wpfleger96 pushed a commit that referenced this pull request Aug 7, 2026
* origin/main: (32 commits)
  Recover from max-token response truncation (#5223)
  chore(release): release Buzz Desktop version 0.5.6 (#5214)
  fix(mobile): keep latest messages above composer (#4981)
  fix(sdk): preserve self-mention p tags in message and forum event builders (#4975)
  bump @tauri-apps/cli to ~2.11.4 to fix linux app icon issue (#4858)
  feat(desktop): adding rich link previews to messages (#3818)
  fix(buzz-agent): Responses reasoning summary, Anthropic display:summarized, ACP v2 messageId (#5195)
  fix(desktop): retain distinct agent instances in autocomplete (#5202)
  fix(desktop): defer channel visibility change to Save (#5203)
  feat(desktop): Projects follow-ups — access restrictions, fast loading, activity feed polish (#5073)
  refactor(cli): replace probe/decider/detail split with single typed extractor (#5191)
  fix(desktop): drop unhandled rejection from throwing window.Notification (#5143)
  fix(desktop): fence localStorage SecurityError from killing the React tree (#5142)
  fix(desktop): make terminal output selectable (#4980)
  fix(desktop): use WEBKIT_DMABUF_RENDERER_FORCE_SHM for NVIDIA/AppImage (#3654) (#4505)
  Make public starter channels best effort (#5192)
  Mobile: add anchored reaction popover (#5025)
  feat(mobile): add bee pull-to-refresh (#5059)
  Remove agent creation success modal (#5063)
  fix(buzz-agent): escalate LLM timeouts per retry and log per-call latency (#5130)
  ...

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>

# Conflicts:
#	desktop/src/shared/api/tauri.ts
tlongwell-block pushed a commit that referenced this pull request Aug 7, 2026
Brings the bench branch up from 13c9e90 to current main (c7b6636) as
a merge commit — no rebase, no history rewrite. Clean auto-merge, zero
conflicts. Notably picks up:

- #5248: budget summarizer reasoning separately so it cannot starve
  the handoff summary
- #5223: recover from max-token response truncation
- #5195: Responses reasoning summary + ACP v2 messageId fixes
- #5130: escalate LLM timeouts per retry, log per-call latency

Originating Buzz thread:
buzz://message?channel=c3252dd2-0142-4e01-88c7-a2183c3960a5&id=9e60a8dfa64a59339d7b357c509e231f6bcf76e5507a92078a6723378f9d1e95

Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
wpfleger96 pushed a commit that referenced this pull request Aug 7, 2026
…format

* origin/main: (60 commits)
  feat(desktop): unify add agent flows (#5015)
  fix(buzz-agent): budget summarizer reasoning separately so it cannot starve the handoff summary (#5248)
  infra: bind development services to loopback (#4871)
  chore(release): release Buzz Desktop version 0.5.7 (#5252)
  fix(desktop): isolate relay admission tests (#5221)
  fix(desktop): externalize boot <style> to prevent Tauri CSP nonce override (#5242)
  fix(desktop): let imported and recovered identities finish onboarding (#5228)
  Recover from max-token response truncation (#5223)
  chore(release): release Buzz Desktop version 0.5.6 (#5214)
  fix(mobile): keep latest messages above composer (#4981)
  fix(sdk): preserve self-mention p tags in message and forum event builders (#4975)
  bump @tauri-apps/cli to ~2.11.4 to fix linux app icon issue (#4858)
  feat(desktop): adding rich link previews to messages (#3818)
  fix(buzz-agent): Responses reasoning summary, Anthropic display:summarized, ACP v2 messageId (#5195)
  fix(desktop): retain distinct agent instances in autocomplete (#5202)
  fix(desktop): defer channel visibility change to Save (#5203)
  feat(desktop): Projects follow-ups — access restrictions, fast loading, activity feed polish (#5073)
  refactor(cli): replace probe/decider/detail split with single typed extractor (#5191)
  fix(desktop): drop unhandled rejection from throwing window.Notification (#5143)
  fix(desktop): fence localStorage SecurityError from killing the React tree (#5142)
  ...

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants