fix(runtime): RuntimeEvent ledger is recovery-grade (WAL), not source-of-truth — write-path truncation blocks counterfactual replay #615

Activity

1. Astro-Han commented on Jul 7, 2026

   

   Astro-Han

   on Jul 7, 2026

   ContributorAuthor

   More actions

   Independent verification pass (Codex, read-only, per-claim against source). Central thesis CONFIRMED: write path has pre-ledger loss (F1/F2/F4 confirmed at cited lines; ledger append path verified through ai-sdk-flow.ts:227 → agent-run.ts:441), read path depends on unsnapshotted live inputs, projection core purity confirmed, read-time prune/compact mechanisms confirmed non-mutating with SHA-verified archives.

   Five statements need narrowing before this is used as a fix spec:

   1. F3 is too broad. Only bare-string Pi tool results get the 8192-char cap (pi-agent-backend.ts:543-553); results already shaped as {kind: ...} objects go through redactUnknown() with no per-field cap (:544,556).
   2. toolSchemaHash is NOT persisted in model_stream_started. The trace event carries systemPromptHash and requestShapeHash only (run-trace.ts:104, agent-run.ts:928); toolSchemaHash is computed inside the request-shape diagnostic but never emitted. Replay test step 1 must drop that assertion or the capture must be added first.
   3. History projection hash is not uniformly shape-only. Plain text is hashed as role + char count, but tool-result output payloads DO get a content hash via stableHash(value) (request-shape.ts:282-318). Tool outputs are therefore content-verifiable today; only message text is not.
   4. Injectable fetch only reaches the claude-subscription/codex-subscription branches (model-factory.ts:15,52,71); plain openai/google/openai-compatible providers don't receive it. Byte-diff test step 3 requires wiring fetch through the remaining provider branches first.
   5. "System prompt live-read each turn" is not uniform. resolveSystemPrompt() is invoked per send, but the desktop local-memory fragment is snapshotted at backend creation (apps/desktop/src/main/main.ts:631). Conclusion unchanged (resolved prompt body is still unrecorded), mechanism corrected.

   One adjacent write-path finding the audit missed: SpawnChildAgent parent-side tool result stores at most a 4000-char summary (session-manager.ts:532,1032). Softer than F1/F3 since the full child run persists in the child's own ledger (recoverable via AgentOutput), but strictly the parent run's ledger is not self-sufficient.
2. Astro-Han

   assigned

   likun666661

   on Jul 7, 2026
3. Astro-Han commented on Jul 7, 2026

   

   Astro-Han

   on Jul 7, 2026

   ContributorAuthor

   More actions

   @likun666661 This audit touches the ledger/evidence surfaces you own (AHE target protocol #411, evidence export #413) — please review both the findings and the verification corrections above. Two points most relevant to AHE: (1) F1/F3 mean trace-index.json evidence currently references pre-truncated tool outputs, so AHE's failure evidence is lossy at the source; (2) the fix plan's request_captured event would give the evidence export a content-exact request record instead of hash-only. Push back if any finding mischaracterizes the design intent.
4. likun666661 commented on Jul 8, 2026

   

   likun666661

   on Jul 8, 2026

   Member

   More actions

   补充 review 结论：我按当前 upstream/main@a6d99e0d 过了一遍，#615 的核心判断是成立的。

   这不是普通的 resume/runtime correctness bug，而是 RuntimeEvent 当前更像 recovery-grade WAL：它足够恢复“当时已经投给模型的 lossy view”，但还不能作为 AHE / RSI / counterfactual replay 的 source of truth。关键原因有两类：一类是写入 RuntimeEvent 之前已经把事实截断/投影掉；另一类是 provider request 的若干输入仍在发送时 live resolve，只记录 hash/诊断，不记录 byte-exact request body。

   受影响模块主要是这几组：

   1. Shell/tool 输出捕获与投影：packages/runtime/src/shell-exec.ts, bash-tail-buffer.ts, shell-run-manager.ts, shell-tools.ts, builtin-tools.ts, tool-output.ts。
   2. Tool result 到 RuntimeEvent 的落盘链路：tool-runtime.ts, ai-sdk-flow.ts, agent-run.ts, runtimeEventStore, session-manager.ts。
   3. Provider request 构造与 request shape：ai-sdk-backend.ts, model-adapter.ts / model-factory.ts, request-shape.ts。
   4. Live prompt/context 输入：system-prompt/session-environment-prompt.ts, project-context.ts, workspace-instructions.ts, shell run context summary / turn tail prompt。
   5. 后续 projection/archive/evidence surface：context-budget.ts, active-tool-result-prune.ts, semantic-compact.ts, packages/headless/src/harbor-cell.ts, desktop tool-result archive artifacts, result-export.ts, AHE docs。

   一个具体的 write-path 丢失序列是：

   1. 用户启动 run，AgentRun.begin() 只把 user text / attachments 这一层记录下来。
   2. LLM 调 Bash，Bash 产生很长输出，例如 5MB。
   3. shell-exec.ts / BashTailBuffer 只保留约 1MB tail，head 已经丢掉。
   4. shapeTerminalResult() 再通过 truncateToolOutput(..., { direction: 'tail' }) 投影到模型可见预算，默认约 2000 lines / 50KB。
   5. ToolRuntime 发出的是已经 shaped 过的 tool_result session event。
   6. ai-sdk-flow.ts 把它映射成 RuntimeEvent 的 function_response.result。
   7. AgentRun.recordRuntimeEvents() 再把这个 RuntimeEvent 落到 ledger。
   8. 后面的 replay / archive / AHE evidence 只能看到这份已经截断后的结果。想用更宽松 policy 做 counterfactual replay 时，原始 head bytes 已经不存在。

   Pi/tool-result 这边也有类似边界，不过需要说精确一点：Pi backend 对 bare string result 会走 redactBoundedText()，默认 8192 chars；object-shaped result 走的是 redactUnknown()，不是同一个 8192 cap。synthetic tool error 还有 TOOL_ERROR_RESULT_MAX_CHARS = 4000 的单独 cap。

   一个具体的 read-path 非纯序列是：

   1. t0 发送模型请求时，ai-sdk-backend.ts live resolve system prompt、turn tail prompt、shell run summary、workspace instructions、git/project context、tool schema、context budget policy 等。
   2. provider 实际收到的是这些 live bytes 组合出的 { system, messages, tools, providerOptions... }。
   3. 当前 trace/ledger 记录了 user text、部分 prompt/request shape hash 和诊断，但没有记录完整 canonical request body，也没有把 tool schema / effective context policy 作为可重建 snapshot 落下来。
   4. t1 用同一 RuntimeEvent ledger replay，如果日期、git branch、workspace instructions、tool registry 或 env policy 变了，要么 rebuild 出不同 provider bytes，要么只能靠 hash 发现 drift，却不能 byte-exact reconstruct 原请求。

   几个边界/修正也建议写进方案里：

   • request-shape.ts 里确实计算了 toolSchemaHash，但 model_stream_started 目前只持久化 systemPromptHash / requestShapeHash 等字段，没有把独立 toolSchemaHash 作为事件字段发出来。
   • historyProjectionHash 对 tool-result output payload 会做 content-ish stable hash；普通 text 主要是 shape/length 级别，不是完整文本 hash。
   • injectable fetch 当前不是所有 provider branch 都贯通，wire-level replay test 需要先把 provider wiring 补齐。
   • write-time secret redaction 仍然应该保留；secret 不应为了 replay 以 raw 形式持久化。这里需要的是“full redacted artifact before model-facing budget”，不是 raw secret ledger。
   • SpawnChildAgent 父 run 里的工具结果 summary 是 lossy 的，但 child ledger 本身还可以另查；这点不要和父 tool result 的 lossy summary 混成同一个不可恢复面。

   所以我会把修复方向拆成两层：

   1. Tool/output source-of-truth：在任何 model-facing budget/truncation 之前，先持久化 full redacted output artifact，RuntimeEvent 里记录 content-addressed ref、size/hash、redaction metadata、projection policy。50KB/8KB/4000-char 这些应该是“给模型看的 projection 参数”，不是 source-of-truth 的存储上限。如果底层捕获仍然只有 1MB tail，也要显式记录 dropped-byte / uncertainty metadata。
   2. Provider request snapshot：在 provider send 边界新增类似 request_captured 的事件，记录 canonical {system,messages,tools,providerOptions,effectiveContextBudgetPolicy}。大体积字段可以 artifact 化，但 replay 必须能从 ledger/artifact 还原 byte-exact request，不能再依赖 live workspace/git/env/tool registry。

   测试也建议分层做：先做 hash-level regression，确认当前 hash drift 能被稳定检测；有 request_captured 后做 content-level replay，确认从 ledger/artifact 重建出的 request bytes 和原请求一致；等 provider fetch 注入贯通所有分支后，再补 wire-level capture/replay 测试。
5. Astro-Han

   added

   bugSomething isn't workingSomething isn't working

   on Aug 20, 2026
6. Astro-Han

   added the

   Bug

   issue type on Aug 29, 2026
7. Astro-Han

   added a commit that references this issue on Oct 11, 2026

   docs: verify the runtime core docs against main (#3522) (#6054)

   ...

   95c5d28