Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ea7ff04
feat(headless): agentic monorepo detection behind basic-integration-a…
gewenyu99 Jul 13, 2026
56dc470
refactor(headless): compose agentic scoping at the install entry, lik…
gewenyu99 Jul 13, 2026
65daeb9
refactor(headless): declare agentic scoping as a flagged phase of the…
gewenyu99 Jul 13, 2026
29995ee
refactor(detection): extract shared integration-project vocabulary; d…
gewenyu99 Jul 13, 2026
0cae314
refactor: collapse the detection layer — pick straight from the raw s…
gewenyu99 Jul 13, 2026
da56d35
refactor: move the non-interactive scan phase out of the program config
gewenyu99 Jul 13, 2026
64b9e30
refactor: move non-interactive project scoping into lib/detection
gewenyu99 Jul 13, 2026
348568a
refactor: dedupe the integration scan and path containment with self-…
gewenyu99 Jul 13, 2026
d94e450
fix: address review findings on the agentic detection phase
gewenyu99 Jul 13, 2026
b84fb05
feat: fire a wizard event per agentic-detection outcome instead of ta…
gewenyu99 Jul 13, 2026
1591e6f
style: one-line comments throughout the new code
gewenyu99 Jul 13, 2026
329f408
revert: drop the gateway flag-forwarding predicate from this PR
gewenyu99 Jul 13, 2026
aa85af5
docs(runbook): add on/off + blast-radius runbook for agentic monorepo…
gewenyu99 Jul 14, 2026
2503f1a
docs(runbook): trim the agentic detection runbook to the essentials
gewenyu99 Jul 14, 2026
00fde7b
docs(runbook): bullet the sections that ran past a few sentences
gewenyu99 Jul 14, 2026
1d9cc49
docs(runbook): flesh out what's next with the real roadmap
gewenyu99 Jul 14, 2026
b8e1213
docs(runbook): spell out the fallback when the scan fails
gewenyu99 Jul 14, 2026
9dce2a4
Merge branch 'main' into posthog-code/basic-integration-agentic-detec…
gewenyu99 Jul 16, 2026
6988be7
feat(headless): time the agentic project scan out at 60s
gewenyu99 Jul 16, 2026
12286bb
fix: prefix the agentic detection flag with wizard-
gewenyu99 Jul 16, 2026
efa5cc6
feat: put the scanned projects and the recommendation on the outcome …
gewenyu99 Jul 16, 2026
fc19ce7
refactor: send the raw scan projects instead of remapping them
gewenyu99 Jul 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/runbooks/agentic-monorepo-detection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Agentic monorepo detection (headless)

## Why it exists

Headless and CI basic-integration runs detect the framework at the repo root and
install there. In a monorepo the root usually isn't an app, so we'd instrument
the wrong project or nothing at all. This finds the actual app and installs into
it.

## What it does

- Before the integration runs, an agent scans the repo and returns a report:
every project with its path, detected framework, whether it maps to a
supported target, whether it already has PostHog, and exactly one flagged
`recommended` (the main user-facing app — frontend or mobile over servers,
libs, tooling).
- We take the recommended supported project, else the first supported
PostHog-free one, and re-point the install dir there.
- On a single-repo project it recommends `.`, so nothing moves.
- If the scan errors, times out (60s, `AGENTIC_DETECTION_TIMEOUT_MS`), or finds
nothing, `session.installDir` is left untouched and the run falls back to the
old root detection, exactly like flag-off.

## What flag gates it

- `wizard-basic-integration-agentic-detection`, default off.
- On, the scan runs; off (or a failed flag fetch), it's skipped and you're back
on root detection.
- Read once at the start of each run.
- Locally: set
`WIZARD_CI_FLAG_OVERRIDES='{"wizard-basic-integration-agentic-detection":"true"}'`
on a dev/`--ci` run.

## What it affects

- Only non-interactive basic-integration runs — headless and `--ci`. Interactive
runs have their own detect step and never enter it.
- Phase: `scopeInstallDirToProject` in `src/lib/detection/project-scope.ts`,
called from the top of `ciPreRun` in
`src/lib/programs/posthog-integration/index.ts`.
- Detector: `detectProjectsWithAgent` in `src/lib/detection/agentic.ts`;
self-driving uses the same detector with its own chooser.
- Each run fires one `wizard: agentic detection` event tagged with the outcome
(`flag-off | error | timeout | no-project | recommended | first-instrumentable`).

## What's next

- The same report is meant to feed an interactive basic-integration picker
later, with `recommended` pre-selected as the default.
- Fuller monorepo support: suggesting and implementing a sane number of defaults
across the projects it finds.
- Rolling this out is just moving the flag off 0%.
6 changes: 6 additions & 0 deletions src/lib/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,9 @@ export const WIZARD_PI_MODEL_FLAG_KEY = 'wizard-pi-model';
export const WIZARD_PI_EFFORT_FLAG_KEY = 'wizard-pi-effort';
/** Feature flag key that gates the intro-screen "Tools" menu. */
export const WIZARD_TOOLS_MENU_FLAG_KEY = 'wizard-tools-menu';
/** Boolean flag: agentic project scoping for non-interactive basic-integration runs. */
export const WIZARD_BASIC_INTEGRATION_AGENTIC_DETECTION_FLAG_KEY =
'wizard-basic-integration-agentic-detection';
/** User-Agent for wizard HTTP requests and MCP server identification. */
export const WIZARD_USER_AGENT = `posthog/wizard; version: ${VERSION}`;

Expand Down Expand Up @@ -282,6 +285,9 @@ export const POSTHOG_FLAG_HEADER_PREFIX = 'X-POSTHOG-FLAG-';
/** Timeout for framework / project detection probes (ms). */
export const DETECTION_TIMEOUT_MS = 10_000;

/** Timeout for the agentic project scan (ms); past it the run falls back to root detection. */
export const AGENTIC_DETECTION_TIMEOUT_MS = 60_000;

/**
* Timeout for the OAuth authorization flow (ms).
*
Expand Down
90 changes: 89 additions & 1 deletion src/lib/detection/__tests__/agentic.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
import { coerceAgenticReport, manifestGlob } from '@lib/detection/agentic';
import {
coerceAgenticReport,
manifestGlob,
resolveProjectDir,
} from '@lib/detection/agentic';

const TARGETS = ['nextjs', 'node', 'vite'];

Expand Down Expand Up @@ -105,3 +109,87 @@ describe('coerceAgenticReport', () => {
expect(keep('ios')).toBe('ios');
});
});

describe('resolveProjectDir', () => {
it('scopes to the chosen sub-app inside the repo', () => {
expect(resolveProjectDir('/repo', 'apps/web')).toBe('/repo/apps/web');
expect(resolveProjectDir('/repo', '.')).toBe('/repo');
});

it('keeps the root for non-string values (session round-trips are unknown)', () => {
expect(resolveProjectDir('/repo', undefined)).toBe('/repo');
expect(resolveProjectDir('/repo', 42)).toBe('/repo');
});

it('falls back to the repo root when the path escapes it', () => {
// Defense-in-depth on top of coercePath: the value is LLM output.
expect(resolveProjectDir('/repo', '../../etc')).toBe('/repo');
expect(resolveProjectDir('/repo', '/etc')).toBe('/repo');
expect(resolveProjectDir('/repo', 'a/../..')).toBe('/repo');
});
});

describe('coerceAgenticReport recommendation', () => {
const projects = [
{
path: 'apps/api',
framework: 'Express',
targetId: 'node',
hasPostHog: false,
},
{
path: 'apps/web',
framework: 'Next.js',
targetId: 'nextjs',
hasPostHog: false,
recommended: true,
},
];

it('strips recommended entirely when the scan did not ask for it', () => {
// Consumers that never opted in must never see the field, even when the agent emits it anyway.
const report = coerceAgenticReport({ projects }, TARGETS);
for (const p of report.projects) {
expect('recommended' in p).toBe(false);
}
});

it('keeps the recommended label when the scan asked for it', () => {
const report = coerceAgenticReport({ projects }, TARGETS, {
recommend: true,
});
expect(report.projects.map((p) => p.recommended)).toEqual([false, true]);
});

it('keeps at most one recommended project — the first', () => {
const doubled = projects.map((p) => ({ ...p, recommended: true }));
const report = coerceAgenticReport({ projects: doubled }, TARGETS, {
recommend: true,
});
expect(report.projects.map((p) => p.recommended)).toEqual([true, false]);
});

it('coerces malformed recommended values to false', () => {
const report = coerceAgenticReport(
{
projects: [
{ path: '.', recommended: 'yes' },
{ path: 'ios', recommended: 1 },
],
},
TARGETS,
{ recommend: true },
);
expect(report.projects.map((p) => p.recommended)).toEqual([false, false]);
});

it('keeps the label while an escaping path clamps to "."', () => {
const report = coerceAgenticReport(
{ projects: [{ path: '/etc', recommended: true }] },
TARGETS,
{ recommend: true },
);
expect(report.projects[0].path).toBe('.');
expect(report.projects[0].recommended).toBe(true);
});
});
221 changes: 221 additions & 0 deletions src/lib/detection/__tests__/project-scope.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
import {
detectProjectsWithAgent,
type AgenticProject,
} from '@lib/detection/agentic';
import {
chooseIntegrationProject,
scopeInstallDirToProject,
} from '@lib/detection/project-scope';
import {
AGENTIC_DETECTION_TIMEOUT_MS,
WIZARD_BASIC_INTEGRATION_AGENTIC_DETECTION_FLAG_KEY,
} from '@lib/constants';
import { authenticate } from '@lib/agent/runner/shared/authenticate';
import { buildSession } from '@lib/wizard-session';
import { analytics } from '@utils/analytics';

// Mock only the two network edges of scopeInstallDirToProject; everything else runs real.
vi.mock('@lib/agent/runner/shared/authenticate', () => ({
authenticate: vi.fn().mockResolvedValue(undefined),
}));
vi.mock('@lib/detection/agentic', async (importOriginal) => ({
...(await importOriginal<typeof import('@lib/detection/agentic')>()),
detectProjectsWithAgent: vi.fn(),
}));

const project = (overrides: Partial<AgenticProject>): AgenticProject => ({
path: '.',
framework: 'Unknown',
targetId: null,
hasPostHog: false,
...overrides,
});

describe('chooseIntegrationProject', () => {
const api = project({
path: 'apps/api',
framework: 'Express',
targetId: 'javascript_node',
});
const web = project({
path: 'apps/web',
framework: 'Next.js',
targetId: 'nextjs',
recommended: true,
});

it('picks the recommended project when its framework is supported', () => {
expect(chooseIntegrationProject([api, web])?.path).toBe('apps/web');
});

it('picks the recommended project even when it already has PostHog', () => {
// The main app wins — skipping it would silently instrument a secondary project instead.
const withPostHog = { ...web, hasPostHog: true };
expect(chooseIntegrationProject([api, withPostHog])?.path).toBe('apps/web');
});

it('skips an unsupported recommended project for the first supported fresh one', () => {
const unsupported = project({
path: 'ios',
framework: 'Cordova',
recommended: true,
});
expect(chooseIntegrationProject([unsupported, api])?.path).toBe('apps/api');
});

it('returns undefined when no project qualifies', () => {
const unsupported = project({ path: 'crates/core', framework: 'Rust' });
const instrumented = { ...api, hasPostHog: true };
expect(chooseIntegrationProject([unsupported, instrumented])).toBe(
undefined,
);
expect(chooseIntegrationProject([])).toBe(undefined);
});
});

describe('scopeInstallDirToProject', () => {
const scan = vi.mocked(detectProjectsWithAgent);
const FLAG_ON = {
[WIZARD_BASIC_INTEGRATION_AGENTIC_DETECTION_FLAG_KEY]: 'true',
};
const web = project({
path: 'apps/web',
framework: 'Next.js',
targetId: 'nextjs',
recommended: true,
});

let flagsSpy: ReturnType<typeof vi.spyOn>;
let captureSpy: ReturnType<typeof vi.spyOn>;
let exceptionSpy: ReturnType<typeof vi.spyOn>;

beforeEach(() => {
vi.clearAllMocks();
flagsSpy = vi
.spyOn(analytics, 'getAllFlagsForWizard')
.mockResolvedValue({});
captureSpy = vi
.spyOn(analytics, 'wizardCapture')
.mockImplementation(() => undefined);
exceptionSpy = vi
.spyOn(analytics, 'captureException')
.mockImplementation(() => undefined);
});
afterEach(() => {
vi.restoreAllMocks();
});

const outcomeEvent = () => {
// Every path fires exactly one `agentic detection` event — funnels on outcome see every run.
const calls = captureSpy.mock.calls.filter(
([name]) => name === 'agentic detection',
);
expect(calls).toHaveLength(1);
return calls[0][1] as Record<string, unknown>;
};

it('fires flag-off and never scans when the flag is off (or the fetch failed)', async () => {
// A failed flag fetch surfaces as an empty map, so this path also covers "flags unavailable".
const session = buildSession({ installDir: '/repo' });
await scopeInstallDirToProject(session);

expect(vi.mocked(authenticate)).toHaveBeenCalledTimes(1);
expect(session.installDir).toBe('/repo');
expect(outcomeEvent()).toMatchObject({ outcome: 'flag-off' });
expect(scan).not.toHaveBeenCalled();
});

it('re-points installDir at the recommended project and fires recommended with scan facts', async () => {
flagsSpy.mockResolvedValue(FLAG_ON);
scan.mockResolvedValue({ repoType: 'monorepo', projects: [web] });
const session = buildSession({ installDir: '/repo' });
await scopeInstallDirToProject(session);

expect(session.installDir).toBe('/repo/apps/web');
expect(outcomeEvent()).toMatchObject({
outcome: 'recommended',
repo_type: 'monorepo',
project_count: 1,
supported_count: 1,
has_recommendation: true,
chosen_framework: 'nextjs',
chosen_path: 'apps/web',
recommended_path: 'apps/web',
projects: [web],
});
});

it('fires first-instrumentable when the fallback pick wins', async () => {
flagsSpy.mockResolvedValue(FLAG_ON);
scan.mockResolvedValue({
repoType: 'monorepo',
projects: [
project({ path: 'apps/api', framework: 'Express', targetId: 'node' }),
],
});
const session = buildSession({ installDir: '/repo' });
await scopeInstallDirToProject(session);

expect(session.installDir).toBe('/repo/apps/api');
expect(outcomeEvent()).toMatchObject({
outcome: 'first-instrumentable',
has_recommendation: false,
recommended_path: null,
});
});

it('leaves the session untouched, fires error, and reports the exception when the scan throws', async () => {
flagsSpy.mockResolvedValue(FLAG_ON);
const failure = new Error('agent unavailable');
scan.mockRejectedValue(failure);
const session = buildSession({ installDir: '/repo' });
await scopeInstallDirToProject(session);

expect(session.installDir).toBe('/repo');
expect(outcomeEvent()).toMatchObject({
outcome: 'error',
error_message: 'agent unavailable',
});
expect(exceptionSpy).toHaveBeenCalledWith(failure, {
step: 'agentic_detection',
});
});

it('leaves the session untouched and fires timeout when the scan outruns the budget', async () => {
// The scan is abandoned, not cancelled — the run must not wait on it forever.
vi.useFakeTimers();
try {
flagsSpy.mockResolvedValue(FLAG_ON);
scan.mockReturnValue(new Promise(() => undefined));
const session = buildSession({ installDir: '/repo' });
const done = scopeInstallDirToProject(session);
await vi.advanceTimersByTimeAsync(AGENTIC_DETECTION_TIMEOUT_MS);
await done;

expect(session.installDir).toBe('/repo');
expect(outcomeEvent()).toMatchObject({
outcome: 'timeout',
duration_ms: AGENTIC_DETECTION_TIMEOUT_MS,
});
} finally {
vi.useRealTimers();
}
});

it('leaves the session untouched and fires no-project when nothing qualifies', async () => {
flagsSpy.mockResolvedValue(FLAG_ON);
scan.mockResolvedValue({
repoType: 'monorepo',
projects: [project({ path: 'crates/core', framework: 'Rust' })],
});
const session = buildSession({ installDir: '/repo' });
await scopeInstallDirToProject(session);

expect(session.installDir).toBe('/repo');
expect(outcomeEvent()).toMatchObject({
outcome: 'no-project',
project_count: 1,
supported_count: 0,
});
});
});
Loading
Loading