diff --git a/AGENTS.md b/AGENTS.md index 25aa17440b..688cc9eb70 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -232,9 +232,21 @@ hook's JSON decision channel, and Codex only reads a Stop hook's stderr on exit code 2, which this hook never uses). Fail-closed cases are unchanged: a payload that carries `turn_id` as a blank -string is truncated or foreign, not an old Codex, and is refused **before the -rollout is read**; a rollout with no id-carrying turn marker at all skips too. -Both skips stay silent unless `PLANNOTATOR_DEBUG` is set. +string — or as anything that is not a string — is truncated or foreign, not an +old Codex, and is refused **before the rollout is read**; a rollout with no +id-carrying turn marker at all skips too. Both skips stay silent unless +`PLANNOTATOR_DEBUG` is set. + +The deny→resubmit guard follows the same two shapes. When `stop_hook_active` is +set, the plan is served only if it changed across the boundary the previous +blocking Stop left in the turn. Codex >= 0.117 records that boundary as a +`` **user** message (`build_hook_prompt_message`); `rust-v0.114.0`, +`v0.115.0` and `v0.116.0` record it as a **developer** message +(`DeveloperInstructions::new(continuation_prompt)`), so the rollout-fallback +path — and only that path, which keeps every `turn_id`-carrying Codex +byte-identical — also accepts the last developer message in the turn as the +boundary. Without it the guard is inert on exactly the versions the fallback +enables, and an unrevised denied plan is re-served on every Stop of the turn. ## Code Review Flow diff --git a/apps/hook/server/codex-session.test.ts b/apps/hook/server/codex-session.test.ts index 50db4430cc..f45ec8cb3d 100644 --- a/apps/hook/server/codex-session.test.ts +++ b/apps/hook/server/codex-session.test.ts @@ -742,7 +742,7 @@ describe("getLatestCodexPlan", () => { logCodexStopSkip("missing-turn-id", { debug: "1", write }); logCodexStopSkip("missing-turn-marker", { debug: "1", write }); expect(messages).toEqual([ - "[DEBUG] Codex Stop plan review skipped: missing Stop payload turn_id.", + "[DEBUG] Codex Stop plan review skipped: unusable Stop payload turn_id (present but blank or not a string).", "[DEBUG] Codex Stop plan review skipped: missing id-carrying rollout turn marker.", ]); }); @@ -1073,3 +1073,176 @@ describe("getLatestCodexPlan", () => { }); }); }); + +// Codex rust-v0.114.0/0.115.0/0.116.0 record a blocking Stop hook's reason as a +// DEVELOPER-role message and continue the same turn with stop_hook_active; the +// `` user message the de-duplication above keys on only arrived in +// rust-v0.117.0 (`build_hook_prompt_message`). Without a pre-0.117 boundary the +// guard is inert on exactly the versions the rollout turn-id fallback enables. +describe("deny/resubmit de-duplication on the rollout-fallback path", () => { + test("does not re-serve an unchanged denied plan after a developer-role continuation", () => { + const turnId = "turn-fallback-denied"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + completedPlanItem("Original plan", turnId), + developerMessage("YOUR PLAN WAS NOT APPROVED.\n\nPlease revise."), + assistantMessage("Understood, I will start implementing."), + ), + ); + + expect(resolveCodexStopPlan(path, { stopHookActive: true })).toEqual({ + plan: null, + skipReason: null, + fallbackTurnId: turnId, + }); + }); + + test("does not re-serve a plan the model resubmitted unchanged", () => { + const turnId = "turn-fallback-unchanged"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + completedPlanItem("Original plan", turnId), + developerMessage("YOUR PLAN WAS NOT APPROVED.\n\nPlease revise."), + completedPlanItem("Original plan", turnId), + ), + ); + + expect(resolveCodexStopPlan(path, { stopHookActive: true }).plan).toBeNull(); + }); + + test("serves the revised plan after a developer-role continuation", () => { + const turnId = "turn-fallback-revised"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + completedPlanItem("Original plan", turnId), + developerMessage("YOUR PLAN WAS NOT APPROVED.\n\nPlease revise."), + completedPlanItem("Revised plan", turnId), + ), + ); + + expect(resolveCodexStopPlan(path, { stopHookActive: true })).toEqual({ + plan: { text: "Revised plan", source: "plan-item" }, + skipReason: null, + fallbackTurnId: turnId, + }); + }); + + test("anchors on the LAST developer message, so turn-setup instructions are not a boundary", () => { + // Codex records policy/personality developer instructions when it builds the + // turn. Those precede the plan, so only the continuation recorded after it + // may act as the boundary. + const turnId = "turn-fallback-setup"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + developerMessage("Approval policy: on-request."), + completedPlanItem("Original plan", turnId), + developerMessage("YOUR PLAN WAS NOT APPROVED.\n\nPlease revise."), + assistantMessage("Acknowledged."), + ), + ); + + expect(resolveCodexStopPlan(path, { stopHookActive: true }).plan).toBeNull(); + }); + + test("still serves the plan on a first Stop of the turn", () => { + // stop_hook_active is false: nothing has been reviewed yet, so a developer + // message in the turn must not suppress anything. + const turnId = "turn-fallback-first-stop"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + developerMessage("Approval policy: on-request."), + completedPlanItem("Original plan", turnId), + ), + ); + + expect(resolveCodexStopPlan(path).plan).toEqual({ + text: "Original plan", + source: "plan-item", + }); + }); + + test("a payload-named turn keeps the 0.117+ behaviour: no developer boundary", () => { + // Same rollout, but the Stop payload carried a turn_id, which is the modern + // shape. There the `` item is authoritative and a developer + // message must NOT be treated as a boundary. + const turnId = "turn-payload-developer"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + completedPlanItem("Original plan", turnId), + developerMessage("Approved command prefix saved:\n[\"git\", \"status\"]"), + assistantMessage("Running the plan now."), + ), + ); + + expect( + resolveCodexStopPlan(path, { turnId, stopHookActive: true }), + ).toEqual({ + plan: { text: "Original plan", source: "plan-item" }, + skipReason: null, + }); + }); + + test("a payload-named turn still de-duplicates across its hook prompt", () => { + const turnId = "turn-payload-hook-prompt"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + completedPlanItem("Original plan", turnId), + hookPrompt("Please revise the plan."), + assistantMessage("Understood."), + ), + ); + + expect( + resolveCodexStopPlan(path, { turnId, stopHookActive: true }).plan, + ).toBeNull(); + }); + + test("prefers the hook prompt over a later developer message on the fallback path", () => { + // A rollout that somehow carries both shapes: the boundary + // wins, so a developer message recorded after the revised plan cannot + // suppress it. + const turnId = "turn-fallback-both"; + const path = writeTempRollout( + buildRollout( + sessionMeta(), + turnStarted(turnId), + completedPlanItem("Original plan", turnId), + hookPrompt("Please revise the plan."), + completedPlanItem("Revised plan", turnId), + developerMessage("Approved command prefix saved:\n[\"git\", \"status\"]"), + ), + ); + + expect(resolveCodexStopPlan(path, { stopHookActive: true }).plan).toEqual({ + text: "Revised plan", + source: "plan-item", + }); + }); + + test("fails closed on a non-string turn id instead of throwing", () => { + // The Stop payload is untrusted JSON; a foreign or malformed turn_id must + // be refused before the rollout is read, from inside the module rather than + // only at the CLI call site. + for (const raw of [42, true, null, {}, []]) { + expect( + resolveCodexStopPlan("not-read.jsonl", { + turnId: raw as unknown as string, + }), + ).toEqual({ plan: null, skipReason: "missing-turn-id" }); + } + }); +}); diff --git a/apps/hook/server/codex-session.ts b/apps/hook/server/codex-session.ts index c75530464b..cd37f17f60 100644 --- a/apps/hook/server/codex-session.ts +++ b/apps/hook/server/codex-session.ts @@ -317,6 +317,41 @@ function findLastHookPromptIndex( return -1; } +/** + * The pre-`` shape of the same boundary. + * + * Codex only started recording a blocking Stop hook's continuation as a + * `` USER message in rust-v0.117.0 + * (`codex_protocol::items::build_hook_prompt_message`). On rust-v0.114.0, + * v0.115.0 and v0.116.0 — the versions the rollout turn-id fallback exists to + * support — `codex-rs/core/src/codex.rs` records it as + * `DeveloperInstructions::new(continuation_prompt).into()`, i.e. a + * `ResponseItem::Message { role: "developer" }`, and then continues the SAME + * turn with `stop_hook_active = true`. + * + * Nothing in that message identifies the hook that wrote it (the text is the + * hook's own `reason`, which Plannotator lets the user re-template), so the + * role is all there is to key on. The LAST one in the turn is the right + * anchor: Codex records the continuation and immediately re-runs the model, so + * a later developer message can only come from a turn that kept going after + * the block. + */ +function isDeveloperMessage(entry: RolloutEntry): boolean { + if (entry.type !== "response_item") return false; + if (entry.payload?.type !== "message") return false; + return entry.payload?.role === "developer"; +} + +function findLastDeveloperMessageIndex( + entries: RolloutEntry[], + startIndex: number +): number { + for (let i = entries.length - 1; i >= Math.max(startIndex, 0); i--) { + if (isDeveloperMessage(entries[i])) return i; + } + return -1; +} + function getPlanItemText( entry: RolloutEntry, turnId: string @@ -353,7 +388,7 @@ export function logCodexStopSkip( if (!opts.debug) return; const detail = reason === "missing-turn-id" - ? "missing Stop payload turn_id." + ? "unusable Stop payload turn_id (present but blank or not a string)." : "missing id-carrying rollout turn marker."; (opts.write ?? console.error)(`[DEBUG] Codex Stop plan review skipped: ${detail}`); } @@ -512,9 +547,12 @@ export function getRecentCodexMessages( * Fallback source: raw assistant response_item messages that still contain a * block in the rollout transcript. * - * When stopHookActive is true, this only returns a changed post-feedback plan: - * - no plan after the last hook prompt => null - * - identical plan after the last hook prompt => null + * When stopHookActive is true, this only returns a changed post-feedback plan, + * measured across the boundary the previous blocking Stop left in the turn (a + * `` user message on Codex >= 0.117, a developer-role message on + * the rollout-fallback path that serves Codex < 0.117): + * - no plan after that boundary => null + * - identical plan after that boundary => null * * The turn comes from options.turnId, or from the rollout's own turn markers * when the Stop payload carried no `turn_id` field, and must be anchored in the @@ -536,12 +574,21 @@ export function resolveCodexStopPlan( rolloutPath: string, options: GetLatestCodexPlanOptions = {} ): CodexStopPlanLookup { - // A payload that NAMES a turn but names it blank is truncated or foreign, not - // an old Codex: fail closed before the file is read, so stale plan content is - // never even loaded. Keyed on the field being present, because an absent - // field is the old-Codex shape handled below. - const payloadTurnId = options.turnId?.trim(); - if (options.turnId !== undefined && !payloadTurnId) { + // A payload that NAMES a turn but names it unusably — blank, or any + // non-string a foreign/untyped caller hands through — is truncated or + // foreign, not an old Codex: fail closed before the file is read, so stale + // plan content is never even loaded. Keyed on the field being present, + // because an absent field is the old-Codex shape handled below. The + // non-string coercion lives here rather than only at the call site so the + // module fails closed for every caller instead of throwing at `.trim()`. + const rawTurnId: unknown = options.turnId; + const payloadTurnId = + rawTurnId === undefined + ? undefined + : typeof rawTurnId === "string" + ? rawTurnId.trim() + : ""; + if (rawTurnId !== undefined && !payloadTurnId) { return { plan: null, skipReason: "missing-turn-id" }; } @@ -566,7 +613,13 @@ export function resolveCodexStopPlan( } return { - plan: findPlanInTurn(entries, turnStartIndex, turnId, !!options.stopHookActive), + plan: findPlanInTurn(entries, turnStartIndex, turnId, { + stopHookActive: !!options.stopHookActive, + // Only the rollout-fallback path (a Stop payload with no `turn_id`, i.e. + // Codex < 0.117) accepts the developer-role continuation boundary, so + // every Codex that sends `turn_id` keeps byte-identical behaviour. + allowDeveloperBoundary: payloadTurnId === undefined, + }), skipReason: null, ...(payloadTurnId ? {} : { fallbackTurnId: turnId }), }; @@ -576,54 +629,75 @@ function normalizePlan(text: string): string { return text.replace(/\r\n/g, "\n").trim(); } +function toPlanResult( + candidate: CodexPlanCandidate | null +): CodexPlanResult | null { + return candidate ? { text: candidate.text, source: candidate.source } : null; +} + +/** + * The deny→resubmit guard: across the boundary the previous blocking Stop left + * in the turn, return the plan only when it is genuinely NEW. + * + * - no plan after the boundary => the plan on screen is the one already denied + * - identical plan after it => the model resubmitted it unchanged + */ +function planChangedAcrossBoundary( + candidates: CodexPlanCandidate[], + boundaryIndex: number +): CodexPlanResult | null { + const plansAfter = candidates.filter( + (candidate) => candidate.index > boundaryIndex + ); + if (plansAfter.length === 0) return null; + + const latestAfter = pickLatestPreferredPlan(plansAfter); + if (!latestAfter) return null; + + const latestBefore = pickLatestPreferredPlan( + candidates.filter((candidate) => candidate.index < boundaryIndex) + ); + + if ( + latestBefore && + normalizePlan(latestBefore.text) === normalizePlan(latestAfter.text) + ) { + return null; + } + + return toPlanResult(latestAfter); +} + function findPlanInTurn( entries: RolloutEntry[], turnStartIndex: number, turnId: string, - stopHookActive: boolean + options: { stopHookActive: boolean; allowDeveloperBoundary: boolean } ): CodexPlanResult | null { const candidates = collectPlanCandidates(entries, turnStartIndex, turnId); if (candidates.length === 0) return null; - if (!stopHookActive) { - const latestPlan = pickLatestPreferredPlan(candidates); - return latestPlan - ? { text: latestPlan.text, source: latestPlan.source } - : null; + if (!options.stopHookActive) { + return toPlanResult(pickLatestPreferredPlan(candidates)); } const lastHookPromptIndex = findLastHookPromptIndex(entries, turnStartIndex); - - if (lastHookPromptIndex === -1) { - const latestPlan = pickLatestPreferredPlan(candidates); - return latestPlan - ? { text: latestPlan.text, source: latestPlan.source } - : null; + if (lastHookPromptIndex !== -1) { + return planChangedAcrossBoundary(candidates, lastHookPromptIndex); } - const plansAfterHookPrompt = candidates.filter( - (candidate) => candidate.index > lastHookPromptIndex - ); - if (plansAfterHookPrompt.length === 0) return null; - - const latestAfterHookPrompt = pickLatestPreferredPlan(plansAfterHookPrompt); - if (!latestAfterHookPrompt) return null; - - const plansBeforeHookPrompt = candidates.filter( - (candidate) => candidate.index < lastHookPromptIndex - ); - const latestBeforeHookPrompt = pickLatestPreferredPlan(plansBeforeHookPrompt); - - if ( - latestBeforeHookPrompt && - normalizePlan(latestBeforeHookPrompt.text) === - normalizePlan(latestAfterHookPrompt.text) - ) { - return null; + // Codex < 0.117 has no `` item at all: it records the blocking + // Stop hook's reason as a developer-role message and continues the same turn + // with `stop_hook_active = true`. Without this the guard above is inert on + // exactly the versions the rollout turn-id fallback newly enables, and a + // denied plan the model did not revise gets re-served for review on every + // Stop of that turn. + const lastDeveloperIndex = options.allowDeveloperBoundary + ? findLastDeveloperMessageIndex(entries, turnStartIndex) + : -1; + if (lastDeveloperIndex !== -1) { + return planChangedAcrossBoundary(candidates, lastDeveloperIndex); } - return { - text: latestAfterHookPrompt.text, - source: latestAfterHookPrompt.source, - }; + return toPlanResult(pickLatestPreferredPlan(candidates)); } diff --git a/apps/marketing/src/content/docs/guides/troubleshooting.md b/apps/marketing/src/content/docs/guides/troubleshooting.md index 2806f48d4f..bd46c56b73 100644 --- a/apps/marketing/src/content/docs/guides/troubleshooting.md +++ b/apps/marketing/src/content/docs/guides/troubleshooting.md @@ -97,17 +97,17 @@ If a Codex plan turn completes without opening Plannotator: ### Minimum Codex version The `Stop` hook Plannotator uses arrived in Codex **rust-v0.114.0**, so anything -older has no Codex plan review at all. Two later changes matter: +older has no Codex plan review at all. Three later changes matter: | Codex version | What changed | |---------------|--------------| | `rust-v0.114.0` | Hooks engine ships; the feature key is `codex_hooks` and it is off by default | | `rust-v0.117.0` (`0.116.0-alpha.12`) | The Stop payload starts carrying `turn_id` | -| `rust-v0.125.0` | Hooks become enabled by default | -| `rust-v0.131.0` | The feature key is renamed to `hooks`, with `codex_hooks` kept as a legacy alias | +| `rust-v0.124.0` (`0.124.0-alpha.3`) | Hooks become enabled by default | +| `rust-v0.129.0` (`0.129.0-alpha.5`) | The feature key is renamed to `hooks`, with `codex_hooks` kept as a legacy alias | -Step 3 above assumes `rust-v0.131.0` or newer, which is the version this -installer targets. On `rust-v0.114.0`–`rust-v0.130.x` the key Codex reads is +Step 3 above assumes `rust-v0.129.0` or newer, which is the version this +installer targets. On `rust-v0.114.0`–`rust-v0.128.x` the key Codex reads is `codex_hooks = true`, so a config that says `hooks = true` alone has no effect there — write `codex_hooks = true` (both keys together are fine) and restart Codex. @@ -115,9 +115,12 @@ Codex. You do **not** need a Codex new enough to send `turn_id`. On `rust-v0.114.0`, `v0.115.0` and `v0.116.0` the Stop payload omits that field, and Plannotator resolves the turn from the session rollout instead, which is why plan review -opens there as well. When that fallback runs it prints one line to stderr -naming the turn it used, so a hook run captured with stderr visible shows -exactly what happened. +opens there as well. Requesting changes behaves the same there too: Codex +records the feedback as a developer message rather than the `` +item newer versions use, and Plannotator reads either shape, so a plan the +model did not revise is not put back in front of you. When the fallback runs it +prints one line to stderr naming the turn it used, so a hook run captured with +stderr visible shows exactly what happened. Codex hooks are currently disabled on Windows in the official Codex docs, so the Windows installer prints manual guidance instead of changing Codex config automatically.