Skip to content

fix(agent): corrected read-before-write prompt, install_skill capture-and-continue - #816

Merged
gewenyu99 merged 8 commits into
mainfrom
posthog-code/pi-prompt-yara-guard
Jul 9, 2026
Merged

gewenyu99 merged 8 commits into
mainfrom
posthog-code/pi-prompt-yara-guard

Conversation

@gewenyu99

@gewenyu99 gewenyu99 commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator

Correct the read-before-write agent prompt and make install_skill capture the failure and continue rather than aborting the run.

…_skill halt

Three field fixes from the 2026-07-07 triage, all aimed at agents spiraling instead of adapting:

1. YARA repeat-block guard (code, not prompt): the runtime notes already say "retrying the same edit will just block again", but post-2.38.1 data shows ~4.2 blocks per affected run — the prompt alone doesn't hold. Both fences now track the exact payloads they block (tool + content hash). An identical retry gets an escalated reason ("this exact content was ALREADY blocked — change the code, not the retry"), and from the third attempt on the agent is told to note it in the setup report and move on. Wired into the pi security extension's pre-execution scan (bash/write/edit) and the anthropic PreToolUse Bash hook; the tracker lives in yara-hooks so both share one implementation.

2. PI_RUNTIME_NOTES: tell the agent not to delete .posthog-events.json (the host removes it after the run; `rm` is the #1 bash denial at ~31/week) and to `write` new files directly instead of `read`ing a missing path first (the fence errors on reads of nonexistent files and the read doesn't count).

3. posthog-integration prompt STEP 2: if install_skill fails, emit [ERROR-RESOURCE-MISSING] and halt — same pattern STEP 1 already has for load_skill_menu — instead of freelancing the integration without the skill.

Generated-By: PostHog Code
Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci error-tracking-upload-source-maps
  • /wizard-ci mcp-analytics
  • /wizard-ci misc
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci error-tracking-upload-source-maps/android
  • /wizard-ci error-tracking-upload-source-maps/cicd-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-nested-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-single-stage-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-gitlab-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-pnpm-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-raw-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-ssh-vps-node-raw
  • /wizard-ci error-tracking-upload-source-maps/flutter
  • /wizard-ci error-tracking-upload-source-maps/ios
  • /wizard-ci error-tracking-upload-source-maps/next
  • /wizard-ci error-tracking-upload-source-maps/next-no-posthog
  • /wizard-ci error-tracking-upload-source-maps/node-raw
  • /wizard-ci error-tracking-upload-source-maps/node-rollup
  • /wizard-ci error-tracking-upload-source-maps/node-rollup-typescript-plugin
  • /wizard-ci error-tracking-upload-source-maps/node-webpack
  • /wizard-ci error-tracking-upload-source-maps/nuxt-3-6
  • /wizard-ci error-tracking-upload-source-maps/nuxt-4-3
  • /wizard-ci error-tracking-upload-source-maps/react-native
  • /wizard-ci error-tracking-upload-source-maps/react-vite
  • /wizard-ci error-tracking-upload-source-maps/rust
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci misc/quack-quack
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

gewenyu99 added 2 commits July 7, 2026 11:32
… capture-and-continue on skill install failure

Reworks three pieces of this PR per review:

1. rm .posthog-events.json shouldn't be fence-blocked (it isn't on anthropic): Bash sits in BASE_ALLOWED_TOOLS, so the claude-agent-sdk auto-approves it and canUseTool is never consulted — the install/build allowlist only ever bites on pi, where this rm was the #1 denial. wizardCanUseTool now allows exactly `rm [-f] [./].posthog-events.json` (after the operator checks, so chaining still dies), and the "don't delete it" prompt note is gone. The pi host-side cleanup stays as a safety net.

2. The read-before-write pi note treated a self-inflicted problem as a harness fence: pi's write/edit have NO read requirement (verified in pi 0.79.8 source — write is mkdir+writeFile, edit fuzzy-matches oldText; no read tracking exists). The wizard's own shared prompt line ("You must read a file immediately before attempting to write it, even if you have previously read it") is what sent agents to read nonexistent paths. That line now states the accurate rule — read existing files before editing/overwriting; never read a path that doesn't exist yet — and the pi note is deleted.

3. install_skill failure no longer halts. A freestyled integration beats an outright failure, but the run must be measurable: new [SKILL-INSTALL-FAILED] signal, STEP 2 tells the agent to emit it and continue from its own knowledge (noting the missing skill in the setup report), and both harnesses capture a `wizard: agent continued without skill` event with the detail — complementing the tool-level `wizard: skill install failed` event from #807.

Generated-By: PostHog Code
Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
The repeat-block escalation touched the same pi/security.ts + yara-hooks.ts scanning paths that #804 (pi scans through warlock, legacy scanner deleted) rewrites — it belongs on top of that PR, not racing it from main. Reverted the tracker, its fence wiring, and its tests to main here; this PR keeps the non-YARA fixes (plan-file rm allowance, corrected read-before-write prompt, install_skill capture-and-continue). The guard follows as a stacked PR on fix/pi-harness-use-warlock.

Generated-By: PostHog Code
Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
@gewenyu99 gewenyu99 changed the title fix(agent): escalate repeated YARA blocks + pi prompt notes + install_skill halt fix(agent): plan-file rm allowance, corrected read-before-write prompt, install_skill capture-and-continue Jul 7, 2026
gewenyu99 added a commit that referenced this pull request Jul 7, 2026
…(stacked on #804)

The YARA guard split out of #816, rebuilt on top of #804's warlock-based pi fence, plus the one surviving piece of the closed #818:

1. Repeat-block escalation: agents retry identically-blocked payloads (~4.2 blocks per affected run) despite the prompt saying not to. Both fences (pi pre-execution scan, anthropic PreToolUse Bash hook) now hash the payloads they block; an identical retry gets an escalated reason ("this exact content was ALREADY blocked — change the code, not the retry"), and from the third attempt the agent is told to note it in the setup report and move on. The escalation decorates #804's remediation-bearing block message, never replaces it. Tracker lives in yara-hooks, shared by both fences; policy denies are unaffected.

2. Edit scanning scoped to replacement text: #804 still scans JSON.stringify(input.edits), which includes each edit's oldText — pre-existing file content. That blocked edits whose surroundings contained a violation and even edits REMOVING one (field FPs: capture edits adjacent to existing identify() PII; pre-existing violations blocking their own replacement). Now scans only the joined newText, matching the anthropic path which scans new_string only.

Generated-By: PostHog Code
Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
#834 allows scoped `rm [-f] <relative-file>` in the pi fence (where the
allowlist actually bites), which covers the plan file and more, without
touching the shared policy. This PR keeps the read-before-write prompt
correction and the install_skill capture-and-continue.

Generated-By: PostHog Code
Task-Id: 9b33e996-4d8c-4906-8405-63fb4bcd7d07
@gewenyu99
gewenyu99 requested a review from a team as a code owner July 8, 2026 19:33
@gewenyu99 gewenyu99 changed the title fix(agent): plan-file rm allowance, corrected read-before-write prompt, install_skill capture-and-continue fix(agent): corrected read-before-write prompt, install_skill capture-and-continue Jul 8, 2026
@gewenyu99

Copy link
Copy Markdown
Collaborator Author

Full TUI run — express-todo (default · anthropic · claude-sonnet-4-6 · no flag overrides)

Local smoke test of PR #816, driven through the real wizard TUI on the one-file Express app.

Outcome: runPhase: completed · skillsComplete: true · deps added: posthog-node
Screens: intro → auth → run → outro → mcp → slack-connect → keep-skills

All 23 rendered frames

01-intro.ans (+0s)

01-intro.ans

02-auth.ans (+1s)

02-auth.ans

03-run.ans (+2s)

03-run.ans

04-run.ans (+3s)

04-run.ans

05-run.ans (+30s)

05-run.ans

06-run.ans (+33s)

06-run.ans

07-run.ans (+37s)

07-run.ans

08-run.ans (+40s)

08-run.ans

09-run.ans (+44s)

09-run.ans

10-run.ans (+46s)

10-run.ans

11-run.ans (+1m29s)

11-run.ans

12-run.ans (+1m30s)

12-run.ans

13-run.ans (+1m42s)

13-run.ans

14-run.ans (+1m42s)

14-run.ans

15-run.ans (+1m59s)

15-run.ans

16-run.ans (+2m00s)

16-run.ans

17-run.ans (+2m19s)

17-run.ans

18-run.ans (+2m20s)

18-run.ans

19-run.ans (+3m51s)

19-run.ans

20-outro.ans (+4m04s)

20-outro.ans

21-mcp.ans (+4m05s)

21-mcp.ans

22-slack-connect.ans (+4m06s)

22-slack-connect.ans

23-keep-skills.ans (+4m07s)

23-keep-skills.ans

@gewenyu99

Copy link
Copy Markdown
Collaborator Author

Full TUI run — express-todo (override · pi · openai/gpt-5.4 · low effort)

Local smoke test of PR #816, driven through the real wizard TUI on the one-file Express app.

Outcome: runPhase: completed · skillsComplete: true · deps added: posthog-node
Screens: intro → auth → run → outro → mcp → slack-connect → keep-skills

All 31 rendered frames

01-intro.ans (+0s)

01-intro.ans

02-auth.ans (+1s)

02-auth.ans

03-run.ans (+2s)

03-run.ans

04-run.ans (+3s)

04-run.ans

05-run.ans (+15s)

05-run.ans

06-run.ans (+16s)

06-run.ans

07-run.ans (+17s)

07-run.ans

08-run.ans (+18s)

08-run.ans

09-run.ans (+18s)

09-run.ans

10-run.ans (+19s)

10-run.ans

11-run.ans (+20s)

11-run.ans

12-run.ans (+21s)

12-run.ans

13-run.ans (+22s)

13-run.ans

14-run.ans (+36s)

14-run.ans

15-run.ans (+37s)

15-run.ans

16-run.ans (+41s)

16-run.ans

17-run.ans (+41s)

17-run.ans

18-run.ans (+1m11s)

18-run.ans

19-run.ans (+1m11s)

19-run.ans

20-run.ans (+1m17s)

20-run.ans

21-run.ans (+1m18s)

21-run.ans

22-run.ans (+1m19s)

22-run.ans

23-run.ans (+1m22s)

23-run.ans

24-run.ans (+1m23s)

24-run.ans

25-run.ans (+1m39s)

25-run.ans

26-run.ans (+1m40s)

26-run.ans

27-run.ans (+1m51s)

27-run.ans

28-outro.ans (+2m00s)

28-outro.ans

29-mcp.ans (+2m00s)

29-mcp.ans

30-slack-connect.ans (+2m01s)

30-slack-connect.ans

31-keep-skills.ans (+2m02s)

31-keep-skills.ans

@gewenyu99
gewenyu99 merged commit c12cdb4 into main Jul 9, 2026
16 checks passed
@gewenyu99
gewenyu99 deleted the posthog-code/pi-prompt-yara-guard branch July 9, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants