Repository navigation
Description
Summary
End-of-run agent remarks from flag-driven pi runs (wizard-use-pi-harness=true, gpt-5.4/low, build=prod, posthog.com/dev/ci runs excluded, July 9–14) cluster around three causes. All three trace to the same asymmetry: pi got the Anthropic path's restrictions without its affordances, and the pi prompt has drifted behind pi enforcement.
Remark rates over the last 48h of prod runs:
| Theme | pi (48 runs) | anthropic (409 runs) |
|---|---|---|
"can't delete .posthog-events.json" |
40% | 2.7% |
new events missing from read-data-schema |
40% | 4% |
| blocked shell commands | 29% | 10% |
Data: insight mTiRZTvj (Remarks: flag-driven pi runs).
Root causes
1. Scoped rm is allowed since #834, but the prompt still says it isn't
PR #834 (7e030af5, released v2.41.0 on 2026-07-10) added isScopedFileRemoval() (src/lib/agent/runner/harness/pi/security.ts:109, carve-out at :312) so a plain rm [-f] <file inside workdir> passes the fence. But no prompt line was added: PI_RUNTIME_NOTES (src/lib/agent/runner/harness/pi/index.ts:62) still says bash is "ONLY for install/build/typecheck/lint/format commands the project itself defines" and "If a bash command is blocked, do NOT retry it." The context-mill conclude step asks the agent to remove .posthog-events.json; pi has no delete tool; so the model concludes deletion is impossible without ever trying rm and remarks about it nearly every run.
The stale comment at pi/index.ts:641 ("pi's rm is fence-blocked, so the agent can't") predates #834 and is now wrong.
No user-visible damage today — host-side cleanup (pi/index.ts:646) removes the plan file after the run. The cost is wasted agent turns and remark noise.
2. Schema-lag guidance is missing from the shared prompt
Agents don't know that newly instrumented events won't appear in read-data-schema until ingested, so they stall/hedge on dashboard + insight creation. This is harness-agnostic (it shows up on anthropic too, just 10x less often in remarks) and belongs in shared guidance, not pi-only notes.
3. Fence false positives (pi-only pain)
wizardCanUseTool (src/lib/agent/agent-interface.ts:459) blocks:
- any command containing
(or)viaDANGEROUS_OPERATORS = /[;$()]/— kills every Next.js App Router path (app/(main)/page.tsx`) in lint/format args pnpm --filter client typecheck—--filtersits where the script name is parsednpx expo install—expoisn'trun/exec, and"expo install"doesn't start with an allowlisted scriptnpm test/npm start— not inSAFE_SCRIPTSxcodebuild— not inPACKAGE_MANAGERS
The Anthropic path never feels this: Bash is in allowedTools with permissionMode: 'acceptEdits', so the SDK auto-approves it (OS sandbox + YARA hooks protect instead) and canUseTool is only consulted on permission escalation. Pi checks every call through the fence (pi/security.ts:299).
Proposed fixes
- Fix 1 (small, high value): add a
PI_RUNTIME_NOTESline stating plainrm <relative-file-inside-project>is allowed (call out.posthog-events.jsoncleanup as the intended use); update the stale comment atsrc/lib/agent/runner/harness/pi/index.ts:641. - Fix 2 (shared): add a schema-lag rule to
getWizardCommandments()(src/lib/agent/commandments.ts): newly instrumented events won't be inread-data-schemayet — proceed with dashboard/insight creation anyway. - Fix 3 (fence quality, follow-ups in
src/lib/agent/agent-interface.ts):- make the
DANGEROUS_OPERATORScheck ignore(/)inside quoted path arguments - recognize
pnpm --filter <pkg> <script>(validate the script name after the filter) - recognize
npx expo install - decide deliberately on
test/start(likely keep blocked — but say so in the prompt so agents don't burn turns discovering it)
- make the
Created with PostHog Code
Activity