Repository navigation
fix(pi): scan through warlock, delete the resurrected legacy scanner - #804
Conversation
The pi harness security fence called the embedded yara-scanner.ts, a pre-warlock snapshot of the rules that was deleted in #597 and re-added in #701. Its stale rules blocked the canonical $set / $set_once person-property pattern, looping agents on retries. pi now scans through @posthog/warlock via yara-hooks scanAndTriage – same engine, rules, chunking, and LLM triage as the anthropic harness – and block messages include the rule remediation so a blocked agent can self-correct. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
pi auths the LLM gateway programmatically and never sets ANTHROPIC_BASE_URL/AUTH_TOKEN, so the env-based triage provider returned undefined on the pi path: fail-closed, but every flagged match acted on with no false-positive filtering. The provider now accepts explicit gateway auth and pi passes its boot credentials. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gewenyu99
left a comment
There was a problem hiding this comment.
Looks like a clean switch over. We should run a few tests in CI and it should be okay to merge
Resolve conflicts from #793 (pi telemetry): keep warlock-based scanning (scanAndTriage) while folding in recordExternalScan so pi scans still reach the run report. Convert the report violation shape from the deleted yara-scanner's YaraMatch to warlock's ScanMatch.
The repeat-block escalation touched the same pi/security.ts + yara-hooks.ts scanning paths that #804 (pi scans through warlock, legacy scanner deleted) rewrites — it belongs on top of that PR, not racing it from main. Reverted the tracker, its fence wiring, and its tests to main here; this PR keeps the non-YARA fixes (plan-file rm allowance, corrected read-before-write prompt, install_skill capture-and-continue). The guard follows as a stacked PR on fix/pi-harness-use-warlock. Generated-By: PostHog Code Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
…(stacked on #804) The YARA guard split out of #816, rebuilt on top of #804's warlock-based pi fence, plus the one surviving piece of the closed #818: 1. Repeat-block escalation: agents retry identically-blocked payloads (~4.2 blocks per affected run) despite the prompt saying not to. Both fences (pi pre-execution scan, anthropic PreToolUse Bash hook) now hash the payloads they block; an identical retry gets an escalated reason ("this exact content was ALREADY blocked — change the code, not the retry"), and from the third attempt the agent is told to note it in the setup report and move on. The escalation decorates #804's remediation-bearing block message, never replaces it. Tracker lives in yara-hooks, shared by both fences; policy denies are unaffected. 2. Edit scanning scoped to replacement text: #804 still scans JSON.stringify(input.edits), which includes each edit's oldText — pre-existing file content. That blocked edits whose surroundings contained a violation and even edits REMOVING one (field FPs: capture edits adjacent to existing identify() PII; pre-existing violations blocking their own replacement). Now scans only the joined newText, matching the anthropic path which scans new_string only. Generated-By: PostHog Code Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
|
/wizard-ci basic-integration/android |
|
/wizard-ci basic-integration/next-js |
|
/wizard-ci basic-integration/fastapi |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
Problem
pi harness needs warlock, it's running stale rules
pii_in_capture_callrule explicitly blocks$set/$set_onceperson properties insidecapture()– the recommended remediation – so an agent that corrects itself gets blocked again, looping until the run dies.Changes
pi/security.tsscans through warlock. The fence now callsscanAndTriage()fromyara-hooks– the same engine, rules, content chunking, and LLM triage policy as the anthropic harness. Surface mapping matches the anthropic hooks: bash commands →command-context rules with no triage (a flagged command always blocks); write/edit content →output-context rules with triage and the same wizard-docposthog_piisuppression; read/bash output →input-context rules with triage, latching a critical violation.ExtensionHandlertype is(event, ctx) => Promise<R | void> | R | void, so the WASM scan awaits inline; the sync-only signatures in our localPiExtensionApiLikewere narrower than the SDK and are widened to allow promises.remediation(e.g. "use posthog.identify(), setPersonProperties(), or $set / $set_once"), so a blocked agent can self-correct instead of thrashing.yara-scanner.tsis deleted (again) andscanAndTriageis exported fromyara-hooksso any future harness reuses the one scan path instead of resurrecting a fork.ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKENfrom the env, which onlyinitializeAgent(anthropic path) sets – pi auths the gateway programmatically, so on pi the provider would come backundefinedand triage would silently no-op (fail-closed, but every flagged match acted on). pi now passes its boot credentials throughToolGateContext.triageAuth; the env fallback keeps the anthropic path unchanged. The triage model (Haiku) is independent of the agent's model – it's a classifier judging scan matches, routed by the gateway on model id, so it works the same under pi's GPT-class agent models.false_positiveverdict unblocking a write, and fail-closed latching on scanner errors (full suite green).Intentional behavior change: the legacy file carried three product-policy rules –
autocapture_disabled,session_recording_disabled,opt_out_capturing– that were deliberately never ported to warlock (it is security-only). The anthropic path has run without them since #597 (June 25). Deleting the file brings pi to the same baseline, which is the pi harness's stated goal ("reuses the EXACT anthropic policy"). If we want those product guardrails back, that's a both-harnesses discussion, not a pi-only patch – happy to open an issue.Interplay with #793: that PR adds
recordExternalScantelemetry calls to this same file. Whichever lands second rebases; the merge is mechanical (feed the telemetry helper warlock-shaped match fields).Test plan
npx vitest run– 91 files, 1186/1186 passcapture('event', { $set: { email } })writes pass, (b) a top-level-PII write is blocked with remediation text in the agent-visible reason, (c) the YARA scan report still shows scans/violationsBefore un-drafting
@posthog/warlockhere from 0.2.2 to that release🤖 Generated with Claude Code