Skip to content

fix(agent): host-side run-artifact cleanup + new-file exception in read-before-write rule - #833

Closed
gewenyu99 wants to merge 1 commit into
mainfrom
posthog-code/host-side-artifact-cleanup
Closed

gewenyu99 wants to merge 1 commit into
mainfrom
posthog-code/host-side-artifact-cleanup

Conversation

@gewenyu99

Copy link
Copy Markdown
Collaborator

Problem

End-of-run agent remarks from the pi rollout (tracking: #793) are dominated by two frictions — together they account for ~35 of the last ~49 remarks on flag-driven basic-integration runs:

  1. .posthog-events.json cleanup (~25 remarks): the skill tells the agent to remove its event-plan file at the end of the run, but rm is fence-blocked, so every run burns turns on delete-vs-empty deliberation and write-tool workarounds. The pi harness already deleted the file host-side, but only pi, and the agent was still being asked to do it anyway.
  2. Read-before-write ambiguity (~7 remarks): the commandment required a Read "before writing to any file", with no carve-out for files that don't exist yet — agents hit ENOENT and couldn't tell whether that "counts".

Changes

  • ProgramConfig.cleanupArtifacts — programs declare the bookkeeping files their agent run leaves behind. The runner removes them at the runProgram seam on every termination path, for every harness (same dual registerCleanup/finally coverage as flushScanReport). Product knowledge stays in the program config; the runner just executes it.
  • posthog-integration declares EVENT_PLAN_FILE; the pi-harness-only inline cleanup is removed.
  • New commandment: leave wizard bookkeeping files in place — skip skill cleanup steps that ask to remove them (the host deletes them after the run).
  • Read-before-write commandment now scopes to files that already exist, with an explicit brand-new-file exception.
  • Unit + contract tests for the cleanup helper and the new commandment text; commandments snapshot updated so the exact prompt change is visible in this diff.

Test plan

pnpm build && pnpm test && pnpm fix — 96 files / 1,277 tests pass (snapshot updated), lint clean.

Also verified against a 1-file example app (/tmp/wz-demo/example-app with just index.js), driving the real code paths on main vs this branch:

Before (main):

=== 1. file-handling rules in the agent system prompt ===
  > Before writing to any file, you MUST read that exact file immediately beforehand using the Read tool, even if you have already read it earlier in the run. This avoids tool failures and stale edits.
=== 2. end-of-run cleanup on the 1-file example app ===
  app dir BEFORE end-of-run: [ '.posthog-events.json', 'index.js' ]
  cleanupArtifacts declared by posthog-integration: (field does not exist in this checkout)
  no shared cleanup seam in this checkout — an anthropic-arm run ends and nothing removes the file
  app dir AFTER end-of-run:  [ '.posthog-events.json', 'index.js' ]

After (this branch):

=== 1. file-handling rules in the agent system prompt ===
  > Before writing to any file that already exists, you MUST read that exact file immediately beforehand using the Read tool, even if you have already read it earlier in the run. This avoids tool failures and stale edits. Brand-new files are the exception: create them directly — a read of a missing file can only fail and is never required first.
  > Never spend turns deleting or emptying wizard bookkeeping files such as `.posthog-events.json`. If a skill's cleanup step asks you to remove one, skip that step and move on — the wizard removes its own artifacts host-side after the run.
=== 2. end-of-run cleanup on the 1-file example app ===
  app dir BEFORE end-of-run: [ '.posthog-events.json', 'index.js' ]
  cleanupArtifacts declared by posthog-integration: [ '.posthog-events.json' ]
  running shared seam: removeProgramArtifacts(session, posthogIntegrationConfig)
  app dir AFTER end-of-run:  [ 'index.js' ]

Follow-up (context-mill, not this repo): drop the skill's "delete the plan file" conclusion step so the instruction and the commandment stop conflicting.

LLM context

Authored from an audit of wizard remark telemetry on the wizard-switchboard dashboard; the before/after logs above were produced by a throwaway vitest spec driving getWizardCommandments() and the real removeProgramArtifacts seam on both checkouts.


Created with PostHog Code

…ad-before-write rule

End-of-run agent remarks from the wizard#793 rollout showed two dominant
frictions: agents burning turns figuring out how to delete
.posthog-events.json with rm fenced (delete vs empty, write-tools-only
workarounds), and ambiguity over whether brand-new files need a prior
Read before writing.

- ProgramConfig.cleanupArtifacts: programs declare the bookkeeping files
  their run leaves behind; the runner removes them at the runProgram
  seam on every termination path, for every harness (same dual
  registerCleanup/finally coverage as flushScanReport). Replaces the
  pi-harness-only inline cleanup.
- posthog-integration declares EVENT_PLAN_FILE.
- New commandment: leave wizard bookkeeping files in place; skip skill
  cleanup steps that ask to remove them.
- Read-before-write commandment now scopes to files that already exist,
  with an explicit brand-new-file exception.
- Contract/unit tests for both; commandments snapshot updated.

Generated-By: PostHog Code
Task-Id: 9b33e996-4d8c-4906-8405-63fb4bcd7d07
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci error-tracking-upload-source-maps
  • /wizard-ci mcp-analytics
  • /wizard-ci misc
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci error-tracking-upload-source-maps/android
  • /wizard-ci error-tracking-upload-source-maps/cicd-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-nested-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-single-stage-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-gitlab-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-pnpm-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-raw-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-ssh-vps-node-raw
  • /wizard-ci error-tracking-upload-source-maps/flutter
  • /wizard-ci error-tracking-upload-source-maps/ios
  • /wizard-ci error-tracking-upload-source-maps/next
  • /wizard-ci error-tracking-upload-source-maps/next-no-posthog
  • /wizard-ci error-tracking-upload-source-maps/node-raw
  • /wizard-ci error-tracking-upload-source-maps/node-rollup
  • /wizard-ci error-tracking-upload-source-maps/node-rollup-typescript-plugin
  • /wizard-ci error-tracking-upload-source-maps/node-webpack
  • /wizard-ci error-tracking-upload-source-maps/nuxt-3-6
  • /wizard-ci error-tracking-upload-source-maps/nuxt-4-3
  • /wizard-ci error-tracking-upload-source-maps/react-native
  • /wizard-ci error-tracking-upload-source-maps/react-vite
  • /wizard-ci error-tracking-upload-source-maps/rust
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci misc/quack-quack
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

Copy link
Copy Markdown
Collaborator Author

Closing — wrong approach: this papered over the harness behavior with prompt rules and host-side cleanup. The real fix is making the pi harness's tool-use semantics match what Anthropic ships in Claude Code (new-file writes need no prior read; file deletion works). Reworking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant