Skip to content

feat: question answers reach the agent — plan.answered, skill, PFM, archive, WebMCP (PR 3) - #1639

Merged
backnotprop merged 1 commit into
mainfrom
feat/question-blocks-agent
Sep 30, 2026
Merged

backnotprop merged 1 commit into
mainfrom
feat/question-blocks-agent

Conversation

@backnotprop

Copy link
Copy Markdown
Owner

PR 3 (agent side) of the approved question/answer components feature (.product/approved/qa-components/: SPEC §8 PR 3, owner decisions in approval.md). Builds on #1637 and #1638. No publish, no release.

What this adds

  1. plan.answered prompt. Send answers already posts /api/deny with answersOnly: true. The agent now gets:

    The user answered the questions in your plan and asked for no other changes. Their answers are below. Revise the plan so each answered question becomes a decision, then call {{toolName}} again. Rules: {{planFileRule}}- remove the :::question block and write the decision into the plan, or keep it with the chosen choice - [x] … Keep every question under "Unanswered" as it is …

    It replaces "YOUR PLAN WAS NOT APPROVED".

    • The prompt is configurable as prompts.plan.answered, with runtime overrides like the other keys. A custom plan.denied does not replace it.
    • Every consumer uses one selector, composePlanDeniedMessage (packages/shared/prompts.ts, vendored to Pi):
      • the hook outputs in apps/hook/server/index.ts (Claude Code's message, Codex, Gemini, Copilot, Vibe)
      • the opencode-plan CLI JSON
      • OpenCode 1 and 2 (submit-plan-executor.ts)
      • Pi's index.ts
    • Both plan servers pass the flag on only when it is the boolean true. Without the flag, the message is byte-identical to before.
    • The client drops PR 2's interim framing paragraph (planDenyFeedbackFields), so the framing is not said twice.
    • Compatibility: an old server ignores the flag and wraps the unframed export in its denied prompt, which still reads as "revise with these answers". An old UI never sends the flag.
  2. /plannotator skill. A new section, "Asking the reviewer questions", is a byte copy of QUESTION_AUTHORING_GUIDE.

    • The guide now also says how answers come back: ### Q2. …, Answer: …, (your recommendation), Other:, quoted free text, Skipped, Note:, and settled questions.
    • A freshness test in plannotator-skill-reference.test.ts fails on any drift. Mutation-checked: changing "about 8" to "about 9" in the SKILL fails it.
    • Pi ships the same SKILL.md through vendor.sh.
  3. PFM reminder. A short "Questions for the reviewer" paragraph and an example.

    • It is one static constant, used by all three runtimes, so prompt caching is not affected.
    • A test parses the example with the real question parser, and another checks the text is static.
    • Length is 2633, under the 3000 cap.
  4. Feedback archive. A new additive field, questionAnswer, on each annotation record. It is validated by parseQuestionAnswer, and malformed values are dropped.

    • Pi now vendors core/question-block.
  5. WebMCP (read-only).

    • read_document returns questions: number, key, kind, prompt, context, section, options (recommended/settled), status, the answer, and its annotation id. Orphaned answers have q: null.
    • The key is present only when the document has questions or answers.
    • Answer rows in annotations carry answersQuestion.
    • No tool writes an answer. Tests cover add_comments (including a reply to an answer), update_comment and remove_comments. External POST/PATCH drop questionAnswer (now tested).
    • The read_document description and the include enum are updated.
  6. Docs.

    • AGENTS.md (CLAUDE.md): a new "Question blocks" section, the /api/deny row, and the pfmReminder note.
    • New marketing guide: guides/questions.md.
    • Also updated: custom-feedback.md (answered row and default), api-endpoints.md (answersOnly), webmcp-tools.md, and the HANDOFF line.

Evidence

  • Unit tests:
    • prompts: selection, the byte-identical denied prompt, config and runtime overrides, and a custom denied not leaking
    • the Bun server (packages/server/plan-answered.test.ts) and the Pi server (apps/pi-extension/server/plan-answered.test.ts): a real /api/deny → decision → message. A non-boolean flag is ignored.
    • the OpenCode executor
    • the client deny body
    • the archive field
    • the PFM example
    • WebMCP questions (documentTools.questions.test.ts)
    • the external API dropping the field
    • the skill freshness test
  • bun test: 5612 pass, 0 fail. bun run typecheck passes.
  • CI DOM list: 1312 pass, 4 fail. The 4 failures are DocBadges.test.tsx, which passes 8/8 when run alone. It is an order-dependent flake in my local ordering and unrelated to this PR.
  • bun run build:review && build:hook && build:opencode && build:pi succeed.
  • bun pm pack --dry-run in apps/pi-extension includes generated/question-block.ts, generated/prompts.ts and the updated skills/plannotator/SKILL.md.
  • End to end, in headless Chromium (--use-mock-keychain --password-store=basic) with a temp data dir. The plan had 3 questions (single with recommendation, multi, free text):
    • Bun hook, Send answers. The real apps/hook/server/index.ts ran with a PermissionRequest on stdin. The primary read "Send answers". Hook stdout:
      {"hookSpecificOutput":{"hookEventName":"PermissionRequest","decision":{"behavior":"deny","message":"The user answered the questions in your plan and asked for no other changes. Their answers are below.\n\nRevise the plan so each answered question becomes a decision, then call ExitPlanMode again.\n\nRules:\n- For each answered question, either remove its `:::question` block and write the decision into the plan, or keep the block and mark the chosen choice `- [x]` (a checked choice reads as settled).\n- Keep every question listed under \"Unanswered\" as it is.\n- Treat a note on an answer as guidance from the user.\n- Do NOT change the plan title (first # heading) unless the user explicitly asks you to.\n\n# Plan Feedback\n\n## Answers to your questions\n\n3 of 3 questions answered.\n\n### Q1. Where should losing conflict versions be kept? (line 12)\nAnswer: Local only, purged after 30 days (your recommendation)\n\n### Q2. Which sync indicators ship in v1? (line 26)\nAnswer:\n- Per-document synced / pending / failed dot\n- Global offline banner\n\n### Q3. Describe the manual test you'd run before calling offline sync shippable. (line 36)\nAnswer:\n> Two phones, one offline, same note.\n\n---\n"}}}
      
    • Bun hook, comment deny (a global comment; the primary read "Send Feedback"). The message is the unchanged denied prompt: YOUR PLAN WAS NOT APPROVED.\n\nYou MUST revise … ## 1. General feedback about the plan\n> Why 60s? ….
    • Pi server, both flows. The built apps/pi-extension/plannotator.html ran on startPlanReviewServer, and the message was composed exactly as Pi's index.ts does it. Send answers gave answersOnly: true and the answered prompt, with plannotator_submit_plan and the plan-file rule. The comment deny gave the unchanged denied prompt.

Deviations

  • Framing paragraph dropped, not trimmed. The UI and server ship together in every bundle, so the flag and the prompt always arrive together. A trimmed line would repeat the prompt's first sentence.
  • The plan archive has no structured answers. Plan /api/approve and /api/deny never sent an annotations array (this predates this PR), so plan answers are archived only inside feedback. Annotate surfaces carry the new questionAnswer field. Sending plan annotations to the archive would be a separate change.
  • The guide text changed (one bullet on how answers come back). This is in core 0.25.8, which is still unpublished.
  • E2E "unchanged from main" for the comment deny is shown by the message equalling getPlanDeniedPrompt (also unit-tested) and by the client comment path being unchanged. I did not build a separate main binary for a side-by-side run.
  • WebMCP read_document description was reworded to fit the 500-character budget with questions included.

Release checklist

… PFM, archive, WebMCP (PR 3)

- plan.answered prompt (prompts.plan.answered, DEFAULT_PLAN_ANSWERED_PROMPT)
  selected by composePlanDeniedMessage when /api/deny carries answersOnly:
  true; wired through the Bun plan server, every hook output, the
  opencode-plan CLI JSON, OpenCode 1/2 (submit-plan-executor) and Pi.
  Ordinary denies are byte-identical. The client drops its interim
  framing paragraph (planDenyFeedbackFields).
- plannotator skill: 'Asking the reviewer questions' is a byte copy of
  QUESTION_AUTHORING_GUIDE, guarded by plannotator-skill-reference.test.ts;
  the guide gains how answers come back.
- PFM reminder: a static question-block paragraph + example.
- Feedback archive: additive questionAnswer per annotation (validated);
  Pi vendors core/question-block.
- WebMCP: read_document returns read-only questions; answer rows carry
  answersQuestion; external POST/PATCH drop questionAnswer (tested).
- Docs: AGENTS.md 'Question blocks', marketing guide, custom-feedback,
  api-endpoints, webmcp-tools.
@backnotprop
backnotprop merged commit 32b825d into main Sep 30, 2026
40 of 41 checks passed
@backnotprop
backnotprop deleted the feat/question-blocks-agent branch September 30, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant