Skip to content

feat(headless): agentic monorepo detection behind basic-integration-agentic-detection flag - #884

Merged
gewenyu99 merged 22 commits into
mainfrom
posthog-code/basic-integration-agentic-detection
Jul 16, 2026
Merged

gewenyu99 merged 22 commits into
mainfrom
posthog-code/basic-integration-agentic-detection

Conversation

@gewenyu99

@gewenyu99 gewenyu99 commented Jul 13, 2026 •

Copy link
Copy Markdown
Collaborator
  • Headless and CI basic-integration runs now agent-scan the repo and install into the app they recommend instead of the repo root, so monorepos stop landing on the wrong project (or nothing).
  • The scan returns a report of every project — path, framework, whether it's a supported target, whether it already has PostHog, and exactly one flagged recommended — and we take the recommended supported one, else the first instrumentable PostHog-free one.
  • If the scan errors, times out (60s), or finds nothing, session.installDir is left untouched and the run falls back to the old root detection, exactly like flag-off.
  • It's all behind the wizard-basic-integration-agentic-detection flag, default off, so the scan never runs until we roll it out. Flag-off does move one thing: auth now happens at the top of ciPreRun rather than later, since the flag has to evaluate as the logged-in user.
  • Only the non-interactive path (ciPreRun) changes; interactive runs are untouched, and it reuses the existing agentic detector (now with an opt-in recommend mode) that self-driving already shares.
  • Every run fires one wizard: agentic detection event so we can watch the outcome mix while rolling out.
  • Why it exists, what it does, the flag, the code paths, and what's next are all in docs/runbooks/agentic-monorepo-detection.md.

…gentic-detection flag

Headless basic-integration runs detect the framework deterministically at
the repo root, which handles monorepos poorly. Mirror the self-driving
pattern: before the (unchanged) integration program runs, the agentic
project detector scans the repo and re-scopes session.installDir to the
project it labels recommended — the main client/frontend/mobile app — and
the program's own detection then runs inside that directory.

- The agentic detector gains an opt-in `recommend` mode: the prompt asks
  for exactly one recommended project and the coerce layer enforces at
  most one, stripping the field entirely for scans that didn't opt in
  (self-driving's picker is byte-identical).
- New headless-scope module owns the policy: flag gate, early idempotent
  auth, choose recommended (even with PostHog present) else first
  instrumentable, containment-checked path resolve, and fallback to the
  unchanged install dir on any failure.
- The non-interactive runner calls it only on the headless fork; CI and
  interactive paths never enter the branch.

Gated on the `basic-integration-agentic-detection` feature flag
(default off), so behavior is unchanged until rollout.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci revenue

Test an individual app:

  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
Show more apps
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci revenue/stripe

Results will be posted here when complete.

gewenyu99 added 11 commits July 13, 2026 16:28
…e self-driving

Move the scoping out of the shared non-interactive runner: instead of a
mode fork with a dynamic import inside runNonInteractive, the headless
install entry (ci-install) wraps the untouched integration program with
withHeadlessAgenticScope — a ciPreRun preamble that re-points
session.installDir, then delegates to the program's own ciPreRun. Same
shape as self-driving, which composes integrationRunStep and supplies
targetDir: the host decides the directory, the program stays a black box,
and the runner stays mechanical.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
… integration program

Replace the entry-point wrapper with a declaration inside the program
itself: ciPreRun's first phase runs the agentic scan when
session.headless is set (new explicit session field, sibling of
session.ci) and the basic-integration-agentic-detection flag is on. The
flag gate stays in the scope module, where credentials exist to
evaluate it. ci-install returns to stock; the runner's only change is
threading headless into the session it builds.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
…rop headless-scope

Replace the headless-specific detection module with a chooser-agnostic
one, @lib/detection/integration-projects: scan + classify (moved from
self-driving's detect-agentic, now with the recommended flag),
chooseProject (auto-choose policy for unattended runs), and
resolveProjectDir (containment, previously duplicated as self-driving's
integrationDir). Self-driving consumes it through a thin adapter — its
picker is just a different chooser over the same report — and a future
interactive basic-integration picker takes the same report with
recommended as the default selection.

The headless plumbing (early auth, flag gate, logging, analytics tags)
moves into the basic-integration program, whose policy it is.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
…can report

Drop integration-projects.ts and all relocation of self-driving's
mapping layer. The classified IntegrationProject vocabulary is picker-UI
machinery the non-interactive path never needed: the raw agentic report
already carries everything the pick requires (targetId = supported,
hasPostHog, recommended). The program now runs the existing detector
with registry targets, chooses via one pure two-find function, and
re-points installDir with the same inline containment check
self-driving uses. CI runs get the phase too (ciPreRun is
non-interactive by definition), so the session.headless field and
runner change are gone as well. Self-driving is untouched.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
Mirror self-driving's layout: the CI/headless-specific detection
machinery lives in posthog-integration/detect-agentic.ts (sibling of
detect.ts, same shape as self-driving/detect-agentic.ts) and index.ts
just imports and calls it — the program config's diff is one import and
one line in ciPreRun.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
Detection logic lives with the other detectors, not in the programs
folder: posthog-integration/detect-agentic.ts becomes
lib/detection/project-scope.ts (content unchanged), next to agentic.ts
and framework.ts. The program keeps only the one-line ciPreRun call.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
…driving

Three fragments were copied from self-driving into project-scope: the
registry→targets mapping, the scan purpose string, and the containment
resolve from integrationDir. Each now has one home:

- detectIntegrationProjects (project-scope.ts) owns targets + purpose;
  self-driving's detect-agentic calls it and keeps only its
  picker-facing classification.
- resolveProjectDir (agentic.ts, beside coercePath — the caller-side
  half of the same LLM-path defense) replaces both the inline copy in
  project-scope and self-driving's integrationDir body.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
- Tag agentic_detection=flag-off when the gate declines, so every run
  through the phase carries exactly one tag value. A failed flag fetch
  surfaces as an empty map, so during rollout a flaky flag endpoint no
  longer reads as a mysteriously low rollout percentage.
- Replace the naming-convention coupling on gateway flag forwarding
  with an explicit predicate: isGatewayForwardedFlag in constants.ts
  skips wizard-local flags by name, so renaming the flag to match its
  wizard-* siblings can't silently start forwarding it. Both harness
  header builders use it.
- Test the scopeInstallDirToProject orchestration: the three fallback
  exits (flag-off, error-fallback, none-fallback) and both success
  tags, with only the two network edges mocked.
- Spread options first in detectIntegrationProjects so the targets and
  purpose it exists to own can't be overridden if the options widen.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
…gging

Replace the agentic_detection session tags with one `wizard: agentic
detection` event fired on every path through the phase, so funnels can
branch on the outcome property (flag-off | error | no-project |
recommended | first-instrumentable) with no untracked exits. Success
and no-project events carry the scan facts (duration, repo type,
project/supported counts, whether the agent recommended anything, and
the chosen framework/path); the error path also reports the exception
to error tracking via captureException.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
isGatewayForwardedFlag's skip case was dead code — the flag doesn't
start with 'wizard', so the prefix filter already excludes it — and it
guarded only a hypothetical rename. If rename-safety ever matters, the
right shape is an explicit allow-list of gateway-consumed flags
(default-deny), which is its own PR. Both harness header builders are
back to stock; the flag key keeps a one-line comment noting it isn't
forwarded because it lacks the prefix.

Generated-By: PostHog Code
Task-Id: 06aaaf67-5aba-497e-bd9e-0220e2417442
@gewenyu99

Copy link
Copy Markdown
Collaborator Author

Test evidence — headless agentic detection

Tested 329f4082 (PR head) on fake fixtures and real repos, source-run via tsx so WIZARD_CI_FLAG_OVERRIDES applies, auth → US project 228144. This path is headless/CI only (no TUI), so the "snapshots" here are the detection reports + full-run logs, not TUI frames.

Result: ships clean. 1332/1332 unit tests, detection chose the right project on 18/18 live scans (deterministic on the pick), and the full binary re-scopes a monorepo to the correct sub-app with the flag on. One honest nuance on the flag-off control is written up at the bottom — not a blocker.

Test cases

# Layer Case Expected Result
1 unit agentic.test.ts + project-scope.test.ts (×4 repeats) 23/23 stable ✅
2 unit shared consumers self-driving-detect + source-maps-detect-agentic unmodified 49/49 ✅
3 unit full suite 1332/1332 ✅
4 live detect single app (fake single-next, real nuxt-single) recommend . ✅
5 live detect fake monorepo, one clear client app recommend apps/dashboard ✅
6 live detect fake monorepo, main app already has posthog recommend apps/web (main app wins over clean secondary) ✅
7 live detect real create-t3-turbo (3 client apps + 5 libs + 5 tooling) stable single pick ✅ apps/nextjs
8 live detect real cicd-monorepo (frontend+backend both have posthog) recommend frontend ✅
9 ciPreRun flag ON monorepo vs OFF monorepo vs single re-scope only when ON+monorepo ✅
10 full binary --ci flag ON, real monorepo, run to completion SDK → apps/web only, dashboard created ✅
11 full binary --ci flag OFF, same monorepo (control) scope phase skipped ✅

1 · Live agentic detection — 6 repos × 3 runs = 18 scans

Repo runs repoType chosen (all runs) targetId outcome proj/supported dur(s)
cicd-mono 3 monorepo frontend javascript_web recommended 4/2, 4/2, 4/2 12.9, 10.7, 11.0
mono-clear 3 monorepo apps/dashboard nextjs recommended 4/3, 4/3, 5/3 12.6, 12.8, 17.7
mono-posthog 3 monorepo apps/web nextjs recommended 2/2, 2/2, 3/2 10.4, 10.4, 11.3
nuxt-single 3 single . nuxt recommended 1/1, 2/2, 1/1 10.2, 9.4, 9.3
single-next 3 single . nextjs recommended 1/1, 1/1, 1/1 12.9, 10.0, 10.6
t3-turbo 3 monorepo apps/nextjs nextjs recommended 14/4, 14/7, 13/7 16.6, 16.8, 15.2

The chosen project was identical on all 3 runs of every repo. Only project_count/supported_count wobble (how many non-chosen packages get enumerated) — never the pick. Latency 9–18s.

Full classified project arrays (run 1 of the three key repos)
### mono-posthog (fake) — recommended app already has posthog-js
★ apps/web               Next.js                  target=nextjs          posthog=true
  apps/admin             Next.js                  target=nextjs          posthog=false

### cicd-monorepo (real)
  .                      pnpm Monorepo            target=null            posthog=false
  backend                Node.js                  target=javascript_node posthog=true
★ frontend               React                    target=javascript_web  posthog=true
  packages/shared        TypeScript Library       target=null            posthog=false

### t3-turbo / create-t3-turbo (real)
  .                      pnpm/Turbo               target=null            posthog=false
★ apps/nextjs            Next.js                  target=nextjs          posthog=false
  apps/expo              React Native (Expo)      target=react-native    posthog=false
  apps/tanstack-start    TanStack Start           target=tanstack-start  posthog=false
  packages/api           tRPC Backend             target=javascript_node posthog=false
  packages/auth          Better Auth              target=null            posthog=false
  packages/db            Drizzle ORM              target=null            posthog=false
  packages/ui            React Component Library  target=null            posthog=false
  packages/validators    Zod Schema Library       target=null            posthog=false
  tooling/eslint         ESLint Config            target=null            posthog=false
  tooling/github         GitHub Actions           target=null            posthog=false
  tooling/prettier       Prettier Config          target=null            posthog=false
  tooling/tailwind       Tailwind Config          target=null            posthog=false
  tooling/typescript     TypeScript Config        target=null            posthog=false

2 · ciPreRun phase — flag A/B (the exact code the PR changes)

Real posthogIntegrationConfig.ciPreRun (scope → detectFramework → gatherContext):

Fixture Flag installDir re-scoped? framework detected
mono-clear ON → apps/dashboard nextjs ✅
mono-posthog ON → apps/web nextjs ✅
single-next ON no (stays .) nextjs ✅
mono-clear OFF no javascript_node ← root mis-detection the PR fixes
single-next OFF no nextjs ✅ (no single-app regression)

3 · Full wizard binary — real monorepo, run to completion

Real Next.js app as apps/web + express services/api + shared lib. tsx bin.ts --ci --install-dir <mono> --api-key … --region us --project-id 228144.

Flag ON Flag OFF
scope phase Scanning… → Continuing with apps/web (Next.js) skipped
skill loaded Next.js App Router skill (7-step) generic skill menu (8-step)
posthog-js added to apps/web ✅ apps/web
root / api / shared touched no ✅ no
setup report written to apps/web/posthog-setup-report.md root posthog-setup-report.md
outcome complete, dashboard 1843769 complete, dashboard 1843778

Flag-on landed the SDK, instrumentation-client.ts, lib/posthog-server.ts, next.config.ts, .env.local, and event capture only in apps/web; root and siblings untouched. Exit 0.

Flag ON — full run log (transient spinner frames removed)
┌  Welcome to the PostHog setup wizard
│  Running posthog-integration in CI mode
│  Using provided API key (CI mode - OAuth bypassed)
│  Scanning the repo for projects...
│  Continuing with apps/web (Next.js).
✔  Framework: Next.js app router 📱
│  Anthropic: healthy — indicator=none
│  PostHog: healthy
│  PostHog (components): healthy
│  GitHub: healthy — indicator=none
│  npm: healthy — indicator=none
│  npm (components): healthy — indicator=none
│  npm (components) components: all operational
│  Cloudflare: degraded — indicator=minor
│  Cloudflare (components): degraded — indicator=minor
│  Cloudflare (components) components impacted: Amsterdam, Netherlands - (AMS) (degraded), Accra, Ghana - (ACC) (degraded), Africa (down), Auckland, New Zealand - (AKL) (degraded), Arica, Chile - (ARI) (down), Asia (down), Europe (down), Basra, Iraq - (BSR) (down), +47 more
│  LLM Gateway: healthy — indicator=HTTP 200
│  MCP: healthy — indicator=HTTP 302
│  GitHub Releases: healthy — indicator=HTTP 200
◇  Initializing Wizard agent...
◇  Verbose logs: /tmp/posthog-wizard.log
✔  Agent initialized. Let's get cooking!
◇  Loading integration skills for the project framework
◇  Installing the Next.js App Router skill and loading its workflow
◇  Reading the installed skill and planning the integration
◇  Planning the integration workflow and reading the first skill step
◇  Checking project structure.
◇  Checking project structure
◇  Generating events based on project.
◇  Verifying PostHog dependencies and planning tracked actions.
◇  Setting PostHog environment variables.
◇  Inserting PostHog capture code for client and server initialization.
◇  Updating Next.js configuration and initializing PostHog.
◇  Planning PostHog edits for client and server tracking.
◇  Planning PostHog initialization and server tracking edits.
◇  Finding and correcting errors in the PostHog setup.
◇  Finding and correcting errors
◇  Inspecting dashboard and insight schemas for the configured analytics artifacts.
◇  Creating the PostHog dashboard and saved insights for the newly instrumented todo events.
◇  Creating saved insights for the dashboard.
◇  Checking remaining task statuses.
●  PostHog integration complete
└  PostHog integration complete
Flag OFF — full run log (control)
┌  Welcome to the PostHog setup wizard
│  Running posthog-integration in CI mode
│  Using provided API key (CI mode - OAuth bypassed)
│  Anthropic: healthy — indicator=none
│  PostHog: healthy
│  PostHog (components): healthy
│  GitHub: healthy — indicator=none
│  npm: healthy — indicator=none
│  npm (components): healthy — indicator=none
│  npm (components) components: all operational
│  Cloudflare: degraded — indicator=minor
│  Cloudflare (components): degraded — indicator=minor
│  Cloudflare (components) components impacted: Amsterdam, Netherlands - (AMS) (degraded), Accra, Ghana - (ACC) (degraded), Africa (down), Auckland, New Zealand - (AKL) (degraded), Arica, Chile - (ARI) (down), Asia (down), Europe (down), Basra, Iraq - (BSR) (down), +47 more
│  LLM Gateway: healthy — indicator=HTTP 200
│  MCP: healthy — indicator=HTTP 302
│  GitHub Releases: healthy — indicator=HTTP 200
◇  Initializing Wizard agent...
◇  Verbose logs: /tmp/posthog-wizard.log
✔  Agent initialized. Let's get cooking!
◇  Loading the integration skill menu
◇  Reading the installed integration workflow
◇  Planning the integration stages and loading the skill workflow.
◇  Checking project structure.
◇  Checking project structure
◇  Checking project structure and reading likely business logic files.
◇  Generating events based on project.
◇  Generating events based on project and reviewing the Next.js integration example.
◇  Generating events based on project and preparing the integration plan.
◇  Setting client and server PostHog environment variables.
◇  Installing the PostHog SDK for the Next.js app workspace.
◇  Planning client and server initialization changes.
◇  Planning initialization files and server helpers.
◇  Planning application initialization and route instrumentation.
◇  Planning analytics edits for client and server tracking.
◇  Inserting PostHog capture code for client and API handlers.
◇  Initializing client and server PostHog wiring.
◇  Reviewing the current integration for build issues and event accuracy.
◇  Inserting PostHog capture code and correcting request identity handling.
◇  Revising PostHog identity flow and server event correlation.
◇  Correcting analytics identity handling and verifying the integration.
◇  Finding and correcting errors.
◇  Inspecting PostHog creation tool schemas.
●  PostHog integration complete
└  PostHog integration complete

Nuance on the flag-off control

Flag-off still instrumented apps/web — not because detection scoped there (it didn't; session.installDir stayed at root, proven by the report landing at root), but because the LLM agent explored the workspace and found the app itself, then ran the generic javascript skill against it. So both runs happened to succeed on this fixture.

What the flag actually buys is determinism + the correct framework skill: flag-on deterministically detects apps/web as Next.js and loads the curated Next.js App Router workflow; flag-off leans on generic root-level javascript_node detection plus the agent getting lucky. That luck won't hold on a messier monorepo (several client apps, deeper nesting) — which is where the stable agentic pick earns its keep.

Detection reports + logs captured by headless test harness; no PR code changed.

gewenyu99 and others added 4 commits July 14, 2026 10:32
… detection

Documents flipping the basic-integration-agentic-detection flag, its blast
radius (non-interactive runs and monorepos only, bounded fallback to root
detection, read-only scan, path clamping), the wizard: agentic detection outcome
events, and the dev/CI override.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cut to why it exists, what it does, the flag, the code paths it touches, and
what's next; drop the operational deep-dive.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gewenyu99
gewenyu99 marked this pull request as ready for review July 14, 2026 14:49
@gewenyu99
gewenyu99 requested review from a team as code owners July 14, 2026 14:49
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@sarahxsanders sarahxsanders left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks safe!

gewenyu99 and others added 5 commits July 16, 2026 10:57
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The key skipped the wizard- prefix every other flag uses, so both gateway
header builders dropped it and the rollout's LLM traces carried no
$feature tag. Renamed to wizard-basic-integration-agentic-detection; the
prefix filters now forward it like the rest.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…event

The event carried counts and a has_recommendation boolean, so a rejected
recommendation and the projects behind "supported_count: 0" were both
invisible. Adds the project list (path, framework, target, PostHog,
recommended) plus recommended_path / recommended_framework.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gewenyu99
gewenyu99 merged commit b153327 into main Jul 16, 2026
16 checks passed
@gewenyu99
gewenyu99 deleted the posthog-code/basic-integration-agentic-detection branch July 16, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants