Repository navigation
refactor(security): replace warlock kill-switch flag with a release-gate smoke test - #828
Conversation
0.2.3 ships warlock #59: the PII capture-rule bleed fix (a propless capture() no longer false-positives a following identify()) and remediation text naming the $set / $set_once / setPersonProperties escape hatches. #804 migrated pi onto warlock but stayed on 0.2.2, so main currently runs the pi fence on the un-fixed rules; this closes that. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ate smoke test Remove the remote wizard-warlock-disabled feature flag. A network-flippable switch that disables a security control is itself an attack surface, and its value only held because we had no way to catch a bad warlock release before shipping. Replace it with two things we control: - Version-lock: @posthog/warlock is pinned exactly (already the case). - A release gate: scripts/warlock-smoke-test.ts imports the REAL warlock (the unit suite mocks it) and asserts the security-critical verdicts on a few fixtures — PII fires, $set/$set_once allowed, no cross-call bleed, engine actually loaded. Wired into postbuild, so pnpm build (and therefore every CI build + publish) fails if a pinned bump regresses a rule. The local POSTHOG_WIZARD_WARLOCK_DISABLED env override stays — it's a dev/CI escape hatch, not a remote production bypass. pi never checked the flag, so only the anthropic path changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
The gate was PII-biased (incident tunnel-vision) — a bump that gutted exfiltration, destructive, supply-chain, or the secret rules would have passed green. Now one representative fixture per warlock CATEGORY (prompt_injection, exfiltration, destructive_operations, supply_chain, posthog_pii, posthog_hardcoded_key, hardcoded_secret), each asserting its category fires, plus the wizard-specific FP contracts (/$set_once allowed, no cross-call bleed). A coverage guard imports warlock's CATEGORIES and fails if any lacks a fixture, so an upstream category addition forces a conscious wizard decision. Per-rule exhaustive coverage stays warlock's job; this gate only proves each threat class the wizard acts on still detects. Secret-shaped fixtures are assembled at runtime so no scannable literal is committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gewenyu99
left a comment
There was a problem hiding this comment.
I'm generally good with these changes. I think if we run CI, that would also catch any unexpected explosions, too in real context.
This should be safe
Co-authored-by: Vincent (Wen Yu) Ge <29069505+gewenyu99@users.noreply.github.com>
|
/wizard-ci basic-integration |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
|
looks like a bunch of CI failed for API overloaded errors? I haven't seen this one yet @gewenyu99 https://github.com/PostHog/wizard-workbench/actions/runs/28899252126/job/85732247289 |
|
/wizard-ci basic-integration/react-router |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
|
/wizard-ci basic-integration/javascript-node |
|
/wizard-ci basic-integration/astro |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
|
/wizard-ci basic-integration/flask |
🧙 Wizard CI ResultsTrigger ID:
Configuration
|
🧙 Wizard CI ResultsTrigger ID:
Configuration
|

Problem
The wizard had a remote kill switch for Warlock/YARA scanning: the
wizard-warlock-disabledPostHog feature flag, which disables the security scanner for a run when it resolves to'true'. Two problems with relying on it:@posthog/warlock(ESM + WASM can't load under the test runner), so nothing actually exercises warlock's rules — a regressed rule would ship, and the flag was the only recourse.We'd rather not be able to ship a bad scanner in the first place than be able to remotely switch off a good one.
Changes
scripts/warlock-smoke-test.tsimports the real package and asserts one representative fixture per warlock category still fires (prompt_injection, exfiltration, destructive_operations, supply_chain, posthog_pii, posthog_hardcoded_key, hardcoded_secret), plus the wizard-specific false-positive contracts ($set/$set_onceallowed, no cross-call bleed). A coverage guard imports warlock's exportedCATEGORIESand fails if any category lacks a fixture, so an upstream category addition forces a conscious wizard decision. Scope is deliberately one-per-category, not per-rule — exhaustive per-rule coverage is warlock's own job (every rule ships +/- tests there); this only proves each threat class the wizard acts on still detects. Secret-shaped fixtures are assembled at runtime so no scannable literal is committed. Wired intopostbuild, sopnpm build— every CI build (build.yml) and every publish (publish.yml) — fails if a pinned bump guts a category.wizard-warlock-disabledfeature flag — the constant, the flag branch inisWarlockDisabled, and its call-site plumbing.isWarlockDisabled()now checks only the local env override.POSTHOG_WIZARD_WARLOCK_DISABLED— a dev/CI escape hatch, not a remote production bypass.The pi harness never checked the flag, so only the anthropic path changes behavior (it loses the remote off-switch; scanning is now always on there unless the local env override is set).
New safety model
Remote off-switch → version-lock + a test that refuses to ship a version whose rules misbehave. Proactive (can't release a bad scanner) instead of reactive (disable it after it's live).
Test plan
pnpm buildrunspostbuild → test:smoke → test:warlock; the warlock smoke test passes (6 cases) against the pinned 0.2.3.isWarlockDisabledtests are updated to the env-only contract.🤖 Generated with Claude Code