Skip to content

refactor(security): replace warlock kill-switch flag with a release-gate smoke test - #828

Merged
sarahxsanders merged 8 commits into
mainfrom
fix/warlock-smoke-test-remove-killswitch
Jul 8, 2026
Merged

sarahxsanders merged 8 commits into
mainfrom
fix/warlock-smoke-test-remove-killswitch

Conversation

@sarahxsanders

@sarahxsanders sarahxsanders commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #827 (the @posthog/warlock 0.2.3 bump) — merge that first. This PR's diff is only the smoke test + flag removal.

Problem

The wizard had a remote kill switch for Warlock/YARA scanning: the wizard-warlock-disabled PostHog feature flag, which disables the security scanner for a run when it resolves to 'true'. Two problems with relying on it:

  1. A network-flippable switch that turns off a security control is itself an attack surface — a bad flag eval, an accidental flip, or a compromised flag source silently disables scanning.
  2. It existed because we had no way to catch a bad warlock release before shipping. The wizard unit suite mocks @posthog/warlock (ESM + WASM can't load under the test runner), so nothing actually exercises warlock's rules — a regressed rule would ship, and the flag was the only recourse.

We'd rather not be able to ship a bad scanner in the first place than be able to remotely switch off a good one.

Changes

  • Add a real-warlock release gate — scripts/warlock-smoke-test.ts imports the real package and asserts one representative fixture per warlock category still fires (prompt_injection, exfiltration, destructive_operations, supply_chain, posthog_pii, posthog_hardcoded_key, hardcoded_secret), plus the wizard-specific false-positive contracts ($set/$set_once allowed, no cross-call bleed). A coverage guard imports warlock's exported CATEGORIES and fails if any category lacks a fixture, so an upstream category addition forces a conscious wizard decision. Scope is deliberately one-per-category, not per-rule — exhaustive per-rule coverage is warlock's own job (every rule ships +/- tests there); this only proves each threat class the wizard acts on still detects. Secret-shaped fixtures are assembled at runtime so no scannable literal is committed. Wired into postbuild, so pnpm build — every CI build (build.yml) and every publish (publish.yml) — fails if a pinned bump guts a category.
  • Remove the wizard-warlock-disabled feature flag — the constant, the flag branch in isWarlockDisabled, and its call-site plumbing. isWarlockDisabled() now checks only the local env override.
  • Keep POSTHOG_WIZARD_WARLOCK_DISABLED — a dev/CI escape hatch, not a remote production bypass.

The pi harness never checked the flag, so only the anthropic path changes behavior (it loses the remote off-switch; scanning is now always on there unless the local env override is set).

New safety model

Remote off-switch → version-lock + a test that refuses to ship a version whose rules misbehave. Proactive (can't release a bad scanner) instead of reactive (disable it after it's live).

Test plan

  • pnpm build runs postbuild → test:smoke → test:warlock; the warlock smoke test passes (6 cases) against the pinned 0.2.3.
  • Full suite green (1240). The isWarlockDisabled tests are updated to the env-only contract.
  • Typecheck clean; no dangling references to the removed flag key.

🤖 Generated with Claude Code

sarahxsanders and others added 2 commits July 7, 2026 16:01
0.2.3 ships warlock #59: the PII capture-rule bleed fix (a propless
capture() no longer false-positives a following identify()) and
remediation text naming the $set / $set_once / setPersonProperties
escape hatches. #804 migrated pi onto warlock but stayed on 0.2.2, so
main currently runs the pi fence on the un-fixed rules; this closes that.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ate smoke test

Remove the remote wizard-warlock-disabled feature flag. A network-flippable
switch that disables a security control is itself an attack surface, and its
value only held because we had no way to catch a bad warlock release before
shipping. Replace it with two things we control:

- Version-lock: @posthog/warlock is pinned exactly (already the case).
- A release gate: scripts/warlock-smoke-test.ts imports the REAL warlock
  (the unit suite mocks it) and asserts the security-critical verdicts on a
  few fixtures — PII fires, $set/$set_once allowed, no cross-call bleed,
  engine actually loaded. Wired into postbuild, so pnpm build (and therefore
  every CI build + publish) fails if a pinned bump regresses a rule.

The local POSTHOG_WIZARD_WARLOCK_DISABLED env override stays — it's a dev/CI
escape hatch, not a remote production bypass. pi never checked the flag, so
only the anthropic path changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci error-tracking-upload-source-maps
  • /wizard-ci mcp-analytics
  • /wizard-ci misc
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci error-tracking-upload-source-maps/android
  • /wizard-ci error-tracking-upload-source-maps/cicd-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-nested-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-github-actions-single-stage-docker-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-gitlab-node-raw
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-pnpm-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-monorepo-raw-node-react
  • /wizard-ci error-tracking-upload-source-maps/cicd-ssh-vps-node-raw
  • /wizard-ci error-tracking-upload-source-maps/flutter
  • /wizard-ci error-tracking-upload-source-maps/ios
  • /wizard-ci error-tracking-upload-source-maps/next
  • /wizard-ci error-tracking-upload-source-maps/next-no-posthog
  • /wizard-ci error-tracking-upload-source-maps/node-raw
  • /wizard-ci error-tracking-upload-source-maps/node-rollup
  • /wizard-ci error-tracking-upload-source-maps/node-rollup-typescript-plugin
  • /wizard-ci error-tracking-upload-source-maps/node-webpack
  • /wizard-ci error-tracking-upload-source-maps/nuxt-3-6
  • /wizard-ci error-tracking-upload-source-maps/nuxt-4-3
  • /wizard-ci error-tracking-upload-source-maps/react-native
  • /wizard-ci error-tracking-upload-source-maps/react-vite
  • /wizard-ci error-tracking-upload-source-maps/rust
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci misc/quack-quack
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

The gate was PII-biased (incident tunnel-vision) — a bump that gutted
exfiltration, destructive, supply-chain, or the secret rules would have
passed green. Now one representative fixture per warlock CATEGORY
(prompt_injection, exfiltration, destructive_operations, supply_chain,
posthog_pii, posthog_hardcoded_key, hardcoded_secret), each asserting its
category fires, plus the wizard-specific FP contracts (/$set_once
allowed, no cross-call bleed). A coverage guard imports warlock's
CATEGORIES and fails if any lacks a fixture, so an upstream category
addition forces a conscious wizard decision.

Per-rule exhaustive coverage stays warlock's job; this gate only proves
each threat class the wizard acts on still detects. Secret-shaped fixtures
are assembled at runtime so no scannable literal is committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sarahxsanders
sarahxsanders requested a review from a team July 7, 2026 20:18
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm generally good with these changes. I think if we run CI, that would also catch any unexpected explosions, too in real context.

This should be safe

Comment thread package.json Resolved
Comment thread scripts/warlock-smoke-test.ts Resolved
Co-authored-by: Vincent (Wen Yu) Ge <29069505+gewenyu99@users.noreply.github.com>
Base automatically changed from chore/bump-warlock-0.2.3 to main July 7, 2026 21:10
@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 7, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: a5131dd
Workflow: View run

App Confidence PR YARA
basic-integration/android/Jetchat 4/5 #2464 (logs) ✓
basic-integration/angular/angular-saas 5/5 #2468 (logs) ✓
basic-integration/astro/astro-hybrid-marketing 5/5 #2465 (logs) ✓
basic-integration/astro/astro-ssr-docs N/A Failed (logs) ✓
basic-integration/astro/astro-static-marketing 5/5 #2462 (logs) ✓
basic-integration/astro/astro-view-transitions-marketing 4/5 #2463 (logs) ✓
basic-integration/django/django3-saas 4/5 #2467 (logs) ✓
basic-integration/fastapi/fastapi3-ai-saas 3/5 #2466 (logs) ✓
basic-integration/flask/flask3-social-media N/A Failed (logs) ✓
basic-integration/javascript-node/express-todo 4/5 #2461 (logs) ✓
basic-integration/javascript-node/fastify-blog 4/5 #2471 (logs) ✓
basic-integration/javascript-node/hono-links N/A Failed (logs) ✓
basic-integration/javascript-node/koa-notes 4/5 #2470 (logs) ✓
basic-integration/javascript-node/native-http-contacts 4/5 #2469 (logs) ✓
basic-integration/javascript-web/saas-dashboard 4/5 #2472 (logs) ✓
basic-integration/laravel/laravel12-saas 4/5 #2476 (logs) ✓
basic-integration/next-js/15-app-router-saas 4/5 #2475 (logs) ✓
basic-integration/next-js/15-app-router-todo 4/5 #2474 (logs) ✓
basic-integration/next-js/15-pages-router-saas 4/5 #2477 (logs) ✓
basic-integration/next-js/15-pages-router-todo 4/5 #2473 (logs) ✓
basic-integration/nuxt/movies-nuxt-3-6 4/5 #2479 (logs) ✓
basic-integration/nuxt/movies-nuxt-4 4/5 #2478 (logs) ✓
basic-integration/python/meeting-summarizer 4/5 #2480 (logs) ✓
basic-integration/rails/fizzy 4/5 #2484 (logs) ✓
basic-integration/react-native/expo-react-native-hacker-news 4/5 #2482 (logs) ✓
basic-integration/react-native/react-native-saas 4/5 #2481 (logs) ✓
basic-integration/react-router/react-router-v7-project N/A Failed (logs) ✓
basic-integration/react-router/rrv7-starter N/A Failed (logs) ✓
basic-integration/react-router/saas-template 4/5 #2490 (logs) ✓
basic-integration/react-router/shopper 4/5 #2483 (logs) ✓
basic-integration/sveltekit/CMSaasStarter 4/5 #2486 (logs) ✓
basic-integration/swift/hackers-ios 5/5 #2491 (logs) ✓
basic-integration/tanstack-router/tanstack-router-code-based-saas 4/5 #2485 (logs) ✓
basic-integration/tanstack-router/tanstack-router-file-based-saas 4/5 #2488 (logs) ✓
basic-integration/tanstack-start/tanstack-start-saas 5/5 #2487 (logs) ✓
basic-integration/vue/movies 4/5 #2489 (logs) ✓

Configuration

Setting Value
Wizard ref fix/warlock-smoke-test-remove-killswitch
Context Mill ref main
PostHog ref master

Search for trigger ID a5131dd in wizard-workbench PRs.

@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

looks like a bunch of CI failed for API overloaded errors? I haven't seen this one yet @gewenyu99 https://github.com/PostHog/wizard-workbench/actions/runs/28899252126/job/85732247289
CleanShot 2026-07-08 at 08 44 29@2x

@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration/react-router

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 8, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: 6ecedec
Workflow: View run

App Confidence PR YARA
basic-integration/react-router/react-router-v7-project 4/5 #2502 (logs) ✓

Configuration

Setting Value
Wizard ref fix/warlock-smoke-test-remove-killswitch
Context Mill ref main
PostHog ref master

Search for trigger ID 6ecedec in wizard-workbench PRs.

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 8, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: ac3676a
Workflow: View run

App Confidence PR YARA
basic-integration/react-router/react-router-v7-project 4/5 #2497 (logs) ✓
basic-integration/react-router/rrv7-starter 4/5 #2504 (logs) ✓
basic-integration/react-router/saas-template 4/5 #2506 (logs) ✓
basic-integration/react-router/shopper 4/5 #2503 (logs) ✓

Configuration

Setting Value
Wizard ref fix/warlock-smoke-test-remove-killswitch
Context Mill ref main
PostHog ref master

Search for trigger ID ac3676a in wizard-workbench PRs.

@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration/javascript-node

@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration/astro

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 8, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: 37a2050
Workflow: View run

App Confidence PR YARA
basic-integration/javascript-node/express-todo 4/5 #2493 (logs) ✓
basic-integration/javascript-node/fastify-blog 4/5 #2492 (logs) ✓
basic-integration/javascript-node/hono-links 4/5 #2494 (logs) ✓
basic-integration/javascript-node/koa-notes 4/5 #2495 (logs) ✓
basic-integration/javascript-node/native-http-contacts 4/5 #2496 (logs) ✓

Configuration

Setting Value
Wizard ref fix/warlock-smoke-test-remove-killswitch
Context Mill ref main
PostHog ref master

Search for trigger ID 37a2050 in wizard-workbench PRs.

@sarahxsanders

Copy link
Copy Markdown
Collaborator Author

/wizard-ci basic-integration/flask

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 8, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: 5b5f245
Workflow: View run

App Confidence PR YARA
basic-integration/astro/astro-hybrid-marketing 4/5 #2501 (logs) ✓
basic-integration/astro/astro-ssr-docs 4/5 #2500 (logs) ✓
basic-integration/astro/astro-static-marketing 4/5 #2498 (logs) ✓
basic-integration/astro/astro-view-transitions-marketing 4/5 #2499 (logs) ✓

Configuration

Setting Value
Wizard ref fix/warlock-smoke-test-remove-killswitch
Context Mill ref main
PostHog ref master

Search for trigger ID 5b5f245 in wizard-workbench PRs.

@wizard-ci-bot

wizard-ci-bot Bot commented Jul 8, 2026 •

Copy link
Copy Markdown

🧙 Wizard CI Results

Trigger ID: a0b3063
Workflow: View run

App Confidence PR YARA
basic-integration/flask/flask3-social-media 4/5 #2505 (logs) ✓

Configuration

Setting Value
Wizard ref fix/warlock-smoke-test-remove-killswitch
Context Mill ref main
PostHog ref master

Search for trigger ID a0b3063 in wizard-workbench PRs.

@sarahxsanders
sarahxsanders merged commit f498c81 into main Jul 8, 2026
16 checks passed
@sarahxsanders
sarahxsanders deleted the fix/warlock-smoke-test-remove-killswitch branch July 8, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants