Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ When creating your personal API key, grant it the wizard's base scope set:
```
user:read project:read organization:read llm_gateway:read query:read
dashboard:write insight:write notebook:write event_definition:write
health_issue:read wizard_session:read wizard_session:write
health_issue:read wizard_session:read wizard_session:write wizard_run:write
```

The source of truth is `WIZARD_OAUTH_SCOPES` in `src/shared/constants.ts`, which
Expand All @@ -277,8 +277,19 @@ Some programs request more on top (`PROGRAM_SCOPE_ADDITIONS` in
`integration:read` and `external_data_source:read` /
`external_data_source:write`.

The `wizard-run-sync` flag selects remote synchronization: `wizard-session`
(the default) uses WizardSession; `wizard-run` uses WizardRun. In the latter
mode, local executions synchronize tasks and terminal status. Cloud executions
require an explicit `POSTHOG_WIZARD_RUN_ID` assignment and leave terminal status
to their worker. See [WizardRun synchronization](docs/local-dev.md#wizardrun-synchronization)
for limits, shutdown behavior, migration compatibility, and deployment checks.

### OAuth app scope ceiling

Both the interactive and cloud Wizard OAuth apps must allow `wizard_run:write`
in every deployed region. Existing tokens need renewed authorization; refresh
does not add the grant. Run synchronization needs no read scope.

The wizard's OAuth app on the PostHog side caps the scopes its tokens may
carry (`OAuthApplication.scopes`). Any scope requested in this repo (see
`src/lib/oauth/program-scopes.ts`) must be grantable under that ceiling, or
Expand Down Expand Up @@ -310,7 +321,7 @@ every scope in `WIZARD_OAUTH_SCOPES`:

```
python manage.py seed_oauth_app_scopes --client-id <id> --dry-run \
--scopes "@default,llm_gateway:read,wizard_session:read,wizard_session:write,user:read,project:read,organization:read,query:read,dashboard:write,insight:write,notebook:write,event_definition:write,health_issue:read"
--scopes "@default,llm_gateway:read,wizard_session:read,wizard_session:write,wizard_run:write,user:read,project:read,organization:read,query:read,dashboard:write,insight:write,notebook:write,event_definition:write,health_issue:read"
```

then re-run without `--dry-run`. Keep `@default` in the list — dropping it
Expand Down Expand Up @@ -405,12 +416,12 @@ wizard alongside all of our other PostHog product data, and this is very
powerful. For example: we could show in-product surveys to people who have used
the wizard to improve the experience.

When the user authenticates, the wizard also streams live run state — current
With `wizard-run-sync=wizard-session`, the wizard streams live run state — current
phase, task list, planned events — to `POST /api/projects/{id}/wizard/sessions/`
so the PostHog web app can render real-time progress. Updates are debounced
(250ms) with phase changes flushed immediately; failures fall back silently to
the wizard's debug log without disturbing the TUI. Pass `--no-telemetry` (or
set `POSTHOG_WIZARD_NO_TELEMETRY=1`) to disable.
set `POSTHOG_WIZARD_NO_TELEMETRY=1`) to disable either remote transport.

## Leave rules behind

Expand Down
118 changes: 117 additions & 1 deletion docs/local-dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,9 @@ These flags are available in dev/test builds. Published builds reject them.
| `--task-stream-log[=path]` | `POSTHOG_WIZARD_TASK_STREAM_LOG` | dump every attempted task-stream sync as JSONL (default `/tmp/posthog-wizard-task-stream.jsonl`) |

`--local-posthog` is sugar over `--base-url`. It pins the API host, app host,
OAuth server, and the LLM gateway derived from them.
and OAuth server. The backend supplies the LLM gateway URL when it mints a
token. If a local PostHog API advertises `host.docker.internal`, the Wizard
uses `localhost` on the same gateway port for its host-side model calls.

`--task-stream-log` records what the run published, one JSON line per push,
truncated per run. It rides beside the PostHog destination rather than
Expand Down Expand Up @@ -175,3 +177,117 @@ reads the resolved targets.

The three service flags have no yargs default. An absent flag is `undefined`
and inherits the umbrella setting; an explicit `false` overrides it.

## WizardRun synchronization

The multivariate `wizard-run-sync` flag selects the remote transport:
`wizard-session` publishes WizardSession state; `wizard-run` publishes WizardRun
tasks and local lifecycle updates. Missing, disabled, or unknown variants use
`wizard-session`. The publisher uses the existing authenticated flag snapshot
and fixes the selection for the execution; there is no extra flag request or
polling. A synchronization failure does not switch transports. File output is
independent of the flag.

The CLI evaluates this flag in PostHog's internal flags project. A local
PostHog web app evaluates its own copy, so changing the flag in the local app
does not change the CLI's selection. To exercise WizardRun from a development
build without changing the internal flag, start an interactive workbench run
with:

```bash
WIZARD_CI_FLAG_OVERRIDES='{"wizard-run-sync":"wizard-run"}' pnpm exec tsx services/wizard-run/index.ts
```

This override is stripped from published builds. An already completed
WizardSession run is not converted; start a new execution after setting the
override.

With `wizard-run`, an authenticated interactive execution creates one local
WizardRun when the agent starts. Creation uses the selected top-level
`ProgramConfig.id`, the resolved API host and project, the target folder's
basename as its display name, and the package version. Program IDs must exist in the backend registry and
support local folders; a rejected configuration disables run synchronization
without selecting a different program. The analytics `run_id`, session
`session_id`, and cloud analytics `task_run_id` remain separate identities.

The shared task publisher sends ordered, immutable full snapshots to
`PUT /api/projects/{project_id}/wizard/runs/{run_id}/tasks/`. Only task names
and statuses are sent; PostHog owns all timestamps. Native task IDs are scoped
to their agent execution, and names are frozen at first observation. Duplicate
subjects receive stable numeric suffixes, including when names are shortened to
255 characters. The snapshot represents the shared task panel (orchestrator
queue tasks for orchestrated runs), not the session stream's extra audit-area
rollups. Completed and failed tasks from earlier composed agents remain in that
panel; omitted tasks from the same agent disappear. An empty agent list retains
earlier terminal tasks; explicitly clearing the authoritative panel sends an
empty snapshot. Unavailable state does not clear it.

Snapshots support at most 100 tasks. Oversized lists, blank names, duplicate
identities, and unexpected statuses reject that snapshot with a sanitized
file-log diagnostic; no partial list is sent. Later valid snapshots can still
sync. `pending` maps to `created`,
`in_progress` to `running`, and `skipped` to `completed`. Completed and failed
states retain their meaning. Cancellation preserves unfinished task states.
Identical snapshots are skipped; distinct transitions are queued before the
legacy session publisher's debounce, so a brief running state is retained.

At execution completion, the CLI drains run tasks and sends one local terminal
status: `completed`, `failed`, or `cancelled`. The existing signal and abort
paths share this shutdown, including Ink Ctrl-C, SIGINT and handled SIGTERM. The
two-second shutdown budget reserves its last quarter for finalization; requests
and retry timers are aborted when their budget expires. Individual requests time
out after five seconds. Task and terminal writes use at most three attempts for
network/server failures and at most one rate-limit retry, with `Retry-After`
capped at 60 seconds outside shutdown. Exhausted task delivery stops later task
writes but still permits local finalization. SIGKILL cannot flush.
Synchronization failure does not change the installation result.

Local creation includes a fresh UUID idempotency key for each execution. **POST
retries are disabled** until an integration check against the deployed backend
confirms that two identical local creation requests return the same ID. The
backend store currently applies supplied keys to local creation, while the
serializer help text describes cloud creation. No run ID is persisted for reuse.

### Cloud assignment and deployment requirements

`POSTHOG_WIZARD_RUN_ID` is the explicit UUID assignment for a WizardRun-backed
cloud execution. The strict CLI parser accepts it as the hidden `--run-id`
option. The authenticated launcher's API host and project are fixed for that
execution. Invalid assignments stop synchronization; they never trigger a local
POST or session fallback. Assigned cloud executions only publish tasks: the
worker owns terminal status after artifact publication. The assignment is not
written to project files or passed to nested agent environments.

Headless invocations without an assignment remain on the legacy WizardSession
transport. `POSTHOG_TASK_RUN_ID` is an analytics compatibility alias and is
**never** interpreted as a WizardRun assignment. Headless mode alone cannot
create a local run. The flag selects one remote transport for local and assigned
cloud executions. Legacy headless launches retain session publishing regardless
of the variant. `--no-telemetry` disables both remote transports; synthetic `--ci`
uses local output only.

The PostHog worker now supplies `POSTHOG_WIZARD_RUN_ID` alongside the existing
analytics alias and handoff path. Its default Wizard version is still 2.74.1,
which does not parse this input. The worker must use a released version that
accepts the assignment, or gate the new environment variable by version, before
the handoff can work for default cloud runs. A WizardRun launcher using a
compatible CLI must provide the assignment; absence denotes legacy mode.

The interactive and cloud Wizard OAuth apps must allow `wizard_run:write` in
each deployed region. The CLI requests this write scope without requesting
`wizard_run:read`. Existing tokens require renewed authorization to gain a new
grant; refresh does not widen permissions. Known missing grants suppress writes,
refreshable expiry uses the existing OAuth refresh path, and permanent 401/403
responses stop further run writes. Personal/project API keys cannot substitute
for a user's Wizard OAuth token on this transport. Session publishing remains
independent when run publishing is disabled.

For deployment validation, run a supported program in a synthetic workspace
using the configured OAuth app. Check one local run, task transitions and server
timestamps, task removal, intentional clear, and all three terminal outcomes.
Repeat the same creation payload/key to verify local idempotency. After
deploying the worker handoff, check that cloud tasks attach to the pre-created
ID and the worker finalizes after artifact publication. Use an authenticated
browser for the Wizard page/SSE, or a read-scoped token for task GET; general
run GET/list/SSE do not accept OAuth. Keep real credentials, paths, and customer
tasks out of validation artifacts.
30 changes: 29 additions & 1 deletion src/__tests__/cli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ vi.mock('../lib/task-stream/index', () => ({
attach() {
mockStreamAttach();
}
finishRun = vi.fn().mockResolvedValue(undefined);
shutdown() {
mockStreamShutdown();
return Promise.resolve();
Expand Down Expand Up @@ -107,7 +108,10 @@ vi.mock('../lib/detection/index', () => ({
gatherFrameworkContext: vi.fn().mockResolvedValue({}),
}));
vi.mock('@utils/analytics', () => ({
analytics: { setTag: vi.fn() },
analytics: {
setTag: vi.fn(),
shutdown: vi.fn().mockResolvedValue(undefined),
},
}));
vi.mock('@utils/wizard-abort', async (importOriginal) => ({
...(await importOriginal<typeof import('@utils/wizard-abort')>()),
Expand All @@ -119,6 +123,11 @@ vi.mock('../lib/programs/run-agent-legacy', () => ({

describe('CLI argument parsing', () => {
const originalArgv = process.argv;
const originalSignals = new Map(
(['SIGINT', 'SIGTERM'] as const).map(
(signal) => [signal, process.listeners(signal)] as const,
),
);
// eslint-disable-next-line @typescript-eslint/unbound-method
const originalExit = process.exit;

Expand All @@ -137,6 +146,7 @@ describe('CLI argument parsing', () => {
'POSTHOG_WIZARD_LOCAL_MCP',
'POSTHOG_WIZARD_LOCAL_POSTHOG',
'POSTHOG_TASK_RUN_ID',
'POSTHOG_WIZARD_RUN_ID',
'POSTHOG_TASK_ID',
];
const clearWizardEnv = () => {
Expand All @@ -158,6 +168,12 @@ describe('CLI argument parsing', () => {
});

afterEach(() => {
for (const [signal, original] of originalSignals) {
for (const listener of process.listeners(signal)) {
if (!original.includes(listener))
process.removeListener(signal, listener);
}
}
process.argv = originalArgv;
process.exit = originalExit;
clearWizardEnv();
Expand Down Expand Up @@ -628,6 +644,18 @@ describe('CLI argument parsing', () => {
expect(process.exit).not.toHaveBeenCalledWith(1);
});

test('accepts the explicit WizardRun assignment through the strict environment parser', async () => {
process.env.POSTHOG_WIZARD_CI = 'true';
process.env.POSTHOG_WIZARD_REGION = 'us';
process.env.POSTHOG_WIZARD_API_KEY = 'pha_test';
process.env.POSTHOG_WIZARD_INSTALL_DIR = '/tmp/test';
process.env.POSTHOG_WIZARD_RUN_ID =
'019edb1a-cce4-4000-8f6d-682061862da9';
await runCLI([]);
expect(process.exit).not.toHaveBeenCalledWith(1);
expect(getLastBuildSessionArgs().runId).toBeUndefined();
});

test('CLI args override CI environment variables', async () => {
process.env.POSTHOG_WIZARD_CI = 'true';
process.env.POSTHOG_WIZARD_REGION = 'us';
Expand Down
25 changes: 25 additions & 0 deletions src/__tests__/wizard-abort.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import {
wizardAbort,
WizardError,
registerCleanup,
registerShutdown,
clearCleanup,
runCleanups,
} from '@utils/wizard-abort';
Expand Down Expand Up @@ -48,6 +49,30 @@ describe('wizardAbort', () => {
vi.restoreAllMocks();
});

it.each([
[{}, 'cancelled'],
[{ error: new Error('failure') }, 'failed'],
[{ code: ErrorCodes.InternalUnhandled, status: 'cancelled' }, 'cancelled'],
] as const)(
'awaits stream shutdown before exit with outcome %s',
async (options, outcome) => {
let release!: () => void;
const shutdown = vi.fn(
() =>
new Promise<void>((resolve) => {
release = resolve;
}),
);
registerShutdown(shutdown);
const result = wizardAbort(options);
const assertion = expect(result).rejects.toThrow('process.exit called');
expect(shutdown).toHaveBeenCalledWith(outcome);
expect(process.exit).not.toHaveBeenCalled();
release();
await assertion;
},
);

it('calls analytics.shutdown, getUI().outroError, and process.exit in order', async () => {
const callOrder: string[] = [];
mockAnalytics.shutdown.mockImplementation(async () => {
Expand Down
1 change: 1 addition & 0 deletions src/agent/__tests__/agent-env-isolation.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ describe('isBlockedAgentEnvKey', () => {
// dir where that path typically sits.
expect(isBlockedAgentEnvKey('POSTHOG_HANDOFF_OUTPUT_PATH')).toBe(true);
expect(isBlockedAgentEnvKey('POSTHOG_TASK_RUN_ID')).toBe(true);
expect(isBlockedAgentEnvKey('POSTHOG_WIZARD_RUN_ID')).toBe(true);
expect(isBlockedAgentEnvKey('POSTHOG_TASK_ID')).toBe(true);
});

Expand Down
Loading
Loading