Skip to content

fix(pi): scan through warlock, delete the resurrected legacy scanner - #804

Merged
sarahxsanders merged 4 commits into
mainfrom
fix/pi-harness-use-warlock
Jul 7, 2026
Merged

sarahxsanders merged 4 commits into
mainfrom
fix/pi-harness-use-warlock

Conversation

@sarahxsanders

@sarahxsanders sarahxsanders commented Jul 6, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

⚠️ Draft until warlock #59 lands and releases – see the checklist at the bottom.

pi harness needs warlock, it's running stale rules

  1. The stale pii_in_capture_call rule explicitly blocks $set / $set_once person properties inside capture() – the recommended remediation – so an agent that corrects itself gets blocked again, looping until the run dies.
  2. The pi block message shows only the rule description, never the remediation, so the agent is told what's wrong but not what to do instead.

Changes

  • pi/security.ts scans through warlock. The fence now calls scanAndTriage() from yara-hooks – the same engine, rules, content chunking, and LLM triage policy as the anthropic harness. Surface mapping matches the anthropic hooks: bash commands → command-context rules with no triage (a flagged command always blocks); write/edit content → output-context rules with triage and the same wizard-doc posthog_pii suppression; read/bash output → input-context rules with triage, latching a critical violation.
  • Handlers are async now. pi's real ExtensionHandler type is (event, ctx) => Promise<R | void> | R | void, so the WASM scan awaits inline; the sync-only signatures in our local PiExtensionApiLike were narrower than the SDK and are widened to allow promises.
  • Block messages include the rule's remediation (e.g. "use posthog.identify(), setPersonProperties(), or $set / $set_once"), so a blocked agent can self-correct instead of thrashing.
  • yara-scanner.ts is deleted (again) and scanAndTriage is exported from yara-hooks so any future harness reuses the one scan path instead of resurrecting a fork.
  • Warlock's WASM engine is prewarmed at pi session setup, off the tool-call path, mirroring the anthropic harness.
  • The triage provider takes explicit gateway auth. It previously read ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN from the env, which only initializeAgent (anthropic path) sets – pi auths the gateway programmatically, so on pi the provider would come back undefined and triage would silently no-op (fail-closed, but every flagged match acted on). pi now passes its boot credentials through ToolGateContext.triageAuth; the env fallback keeps the anthropic path unchanged. The triage model (Haiku) is independent of the agent's model – it's a classifier judging scan matches, routed by the gateway on model id, so it works the same under pi's GPT-class agent models.
  • Tests: the pi security corpus is async now; new cases cover remediation in the block reason, wizard-doc PII suppression, scan-context filtering, a triage false_positive verdict unblocking a write, and fail-closed latching on scanner errors (full suite green).

Intentional behavior change: the legacy file carried three product-policy rules – autocapture_disabled, session_recording_disabled, opt_out_capturing – that were deliberately never ported to warlock (it is security-only). The anthropic path has run without them since #597 (June 25). Deleting the file brings pi to the same baseline, which is the pi harness's stated goal ("reuses the EXACT anthropic policy"). If we want those product guardrails back, that's a both-harnesses discussion, not a pi-only patch – happy to open an issue.

Interplay with #793: that PR adds recordExternalScan telemetry calls to this same file. Whichever lands second rebases; the merge is mechanical (feed the telemetry helper warlock-shaped match fields).

Test plan

  • npx vitest run – 91 files, 1186/1186 pass
  • Typecheck error set diffed against main – identical (zero new errors; the pre-existing errors are the optional pi SDK deps not installed locally)
  • Field verification after un-drafting: run a pi integration against an auth-heavy target and confirm (a) capture('event', { $set: { email } }) writes pass, (b) a top-level-PII write is blocked with remediation text in the agent-visible reason, (c) the YARA scan report still shows scans/violations

Before un-drafting

🤖 Generated with Claude Code

The pi harness security fence called the embedded yara-scanner.ts, a
pre-warlock snapshot of the rules that was deleted in #597 and
re-added in #701. Its stale rules blocked the canonical $set /
$set_once person-property pattern, looping agents on retries.

pi now scans through @posthog/warlock via yara-hooks scanAndTriage –
same engine, rules, chunking, and LLM triage as the anthropic
harness – and block messages include the rule remediation so a
blocked agent can self-correct.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci error-tracking-upload-source-maps
  • /wizard-ci mcp-analytics
  • /wizard-ci misc
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci error-tracking-upload-source-maps/android
  • /wizard-ci error-tracking-upload-source-maps/cicd-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-nested-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-single-stage-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-gitlab-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-pnpm-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-raw-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-ssh-vps-node-raw
  • /wizard-ci error-tracking-upload-source-maps/flutter
  • /wizard-ci error-tracking-upload-source-maps/ios
  • /wizard-ci error-tracking-upload-source-maps/next
  • /wizard-ci error-tracking-upload-source-maps/next-no-posthog
  • /wizard-ci error-tracking-upload-source-maps/node-raw
  • /wizard-ci error-tracking-upload-source-maps/node-rollup
  • /wizard-ci error-tracking-upload-source-maps/node-rollup-typescript-plugin
  • /wizard-ci error-tracking-upload-source-maps/node-webpack
  • /wizard-ci error-tracking-upload-source-maps/nuxt-3-6
  • /wizard-ci error-tracking-upload-source-maps/nuxt-4-3
  • /wizard-ci error-tracking-upload-source-maps/react-native
  • /wizard-ci error-tracking-upload-source-maps/react-vite
  • /wizard-ci error-tracking-upload-source-maps/rust
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci misc/quack-quack
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

pi auths the LLM gateway programmatically and never sets
ANTHROPIC_BASE_URL/AUTH_TOKEN, so the env-based triage provider
returned undefined on the pi path: fail-closed, but every flagged
match acted on with no false-positive filtering. The provider now
accepts explicit gateway auth and pi passes its boot credentials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a clean switch over. We should run a few tests in CI and it should be okay to merge

Comment thread src/lib/agent/triage-provider.ts Resolved
Resolve conflicts from #793 (pi telemetry): keep warlock-based scanning
(scanAndTriage) while folding in recordExternalScan so pi scans still
reach the run report. Convert the report violation shape from the
deleted yara-scanner's YaraMatch to warlock's ScanMatch.
gewenyu99 added a commit that referenced this pull request Jul 7, 2026
The repeat-block escalation touched the same pi/security.ts + yara-hooks.ts scanning paths that #804 (pi scans through warlock, legacy scanner deleted) rewrites — it belongs on top of that PR, not racing it from main. Reverted the tracker, its fence wiring, and its tests to main here; this PR keeps the non-YARA fixes (plan-file rm allowance, corrected read-before-write prompt, install_skill capture-and-continue). The guard follows as a stacked PR on fix/pi-harness-use-warlock.

Generated-By: PostHog Code
Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
gewenyu99 added a commit that referenced this pull request Jul 7, 2026
…(stacked on #804)

The YARA guard split out of #816, rebuilt on top of #804's warlock-based pi fence, plus the one surviving piece of the closed #818:

1. Repeat-block escalation: agents retry identically-blocked payloads (~4.2 blocks per affected run) despite the prompt saying not to. Both fences (pi pre-execution scan, anthropic PreToolUse Bash hook) now hash the payloads they block; an identical retry gets an escalated reason ("this exact content was ALREADY blocked — change the code, not the retry"), and from the third attempt the agent is told to note it in the setup report and move on. The escalation decorates #804's remediation-bearing block message, never replaces it. Tracker lives in yara-hooks, shared by both fences; policy denies are unaffected.

2. Edit scanning scoped to replacement text: #804 still scans JSON.stringify(input.edits), which includes each edit's oldText — pre-existing file content. That blocked edits whose surroundings contained a violation and even edits REMOVING one (field FPs: capture edits adjacent to existing identify() PII; pre-existing violations blocking their own replacement). Now scans only the joined newText, matching the anthropic path which scans new_string only.

Generated-By: PostHog Code
Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
@sarahxsanders
sarahxsanders marked this pull request as ready for review July 7, 2026 18:31
@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration/android

@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration/next-js

@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration/fastapi

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 7, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: 1ecbb94
Workflow: View run

App Confidence PR YARA
basic-integration/android/Jetchat 3/5 #2452 (logs) ✓

Configuration

Setting Value
Wizard ref fix/pi-harness-use-warlock
Context Mill ref main
PostHog ref master

Search for trigger ID 1ecbb94 in wizard-workbench PRs.

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 7, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: 5f6bc1d
Workflow: View run

App Confidence PR YARA
basic-integration/next-js/15-app-router-saas 4/5 #2455 (logs) ✓
basic-integration/next-js/15-app-router-todo 4/5 #2450 (logs) ✓
basic-integration/next-js/15-pages-router-saas 4/5 #2454 (logs) ✓
basic-integration/next-js/15-pages-router-todo 4/5 #2451 (logs) ✓

Configuration

Setting Value
Wizard ref fix/pi-harness-use-warlock
Context Mill ref main
PostHog ref master

Search for trigger ID 5f6bc1d in wizard-workbench PRs.

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 7, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: 4ac81e3
Workflow: View run

App Confidence PR YARA
basic-integration/fastapi/fastapi3-ai-saas 5/5 #2453 (logs) ✓

Configuration

Setting Value
Wizard ref fix/pi-harness-use-warlock
Context Mill ref main
PostHog ref master

Search for trigger ID 4ac81e3 in wizard-workbench PRs.

@sarahxsanders
sarahxsanders merged commit 383ac5d into main Jul 7, 2026
16 checks passed
@sarahxsanders
sarahxsanders deleted the fix/pi-harness-use-warlock branch July 7, 2026 19:00
gewenyu99 added a commit that referenced this pull request Jul 8, 2026
…(stacked on #804) (#820)

Co-authored-by: sarahxsanders <sarah@posthog.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants