Skip to content

feat: synchronize Wizard CLI executions with WizardRun - #1323

Merged
fercgomes merged 16 commits into
mainfrom
codex/wizard-run-sync
Sep 25, 2026
Merged

fercgomes merged 16 commits into
mainfrom
codex/wizard-run-sync

Conversation

@fercgomes

@fercgomes fercgomes commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Wizard executions now publish task progress to WizardRun when the wizard-run-sync flag selects wizard-run. A local execution creates one run for its selected program, sends ordered full task snapshots, and sends one terminal outcome after draining tasks. An assigned cloud execution publishes to its existing run and leaves finalization to the worker. The wizard-session variant and legacy headless launches retain WizardSession publishing.

The CLI freezes task names across status, label, and order changes, retains completed and failed work from earlier agents, and rejects malformed snapshots without disabling later valid updates. The existing task stream owns delivery, and the shared OAuth session keeps run writes and agent gateway/MCP calls on the current login. The flag selection waits for the authenticated flag cache before sending a remote session create. Headless authentication preserves its previous analytics behavior.

The backend API shape and frontend remain unchanged. Cloud rollout still depends on the PostHog worker installing a Wizard version that accepts POSTHOG_WIZARD_RUN_ID, or gating that environment variable by version: its current default, 2.74.1, rejects the new prefixed input. Both deployed Wizard OAuth apps also need a wizard_run:write ceiling, and existing grants need reauthorization. Local creation POST retries remain disabled until deployed local idempotency is verified. Live backend integration was not run.

Checks: focused WizardRun, store, UI, abort, and agent tests passed; pnpm typecheck, pnpm test:arch, and pnpm lint passed. An isolated production bundle built and contained no CI-only overrides. Local pnpm test is affected by concurrent writes to dist and sandbox restrictions on local HTTP listeners/watchers; the PR's GitHub CI checks are the authoritative full-suite result.

@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

@fercgomes
fercgomes marked this pull request as ready for review September 24, 2026 18:05
@fercgomes
fercgomes requested a review from a team as a code owner September 24, 2026 18:05
@edwinyjlim

Copy link
Copy Markdown
Member

@fercgomes could you rebase / merge in main? it's mostly reorganizing. will review right after

@fercgomes
fercgomes force-pushed the codex/wizard-run-sync branch from 0b26cfb to eb9d87f Compare September 24, 2026 19:12
Comment on lines +7 to +15
RunPhase,
type Credentials,
type WizardSession,
} from '@lib/wizard-session';
import type { TaskItem } from '@ui/tui/store';
import { refreshAccessTokenIfNeeded } from '@lib/programs/authenticate';
import { isGrantRevoked } from '@shared/auth-session-state';
import { logToFile } from '@utils/debug';
import { parseRetryAfter } from './destinations/posthog';

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

robot comment

pnpm test:arch fails on this branch: wizard-run-sync.ts -> src/ui/tui/store.ts is a new lib->tui edge (the TaskItem type import on line 11).

pnpm test runs every project, so CI is red. snapshot() only needs { id?, source?, sourceStatus?, label, status }. A local type here removes the edge without grandfathering it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the potential issue: The import edge isn't the only red test. Four more fail at this head, so the local type alone won't turn CI green.

pnpm test at this head -> the standalone runner tests expect tasks without id and source, the legacy runner test's stream stub has no finishRun so the run exits 1, and the flag key test finds wizard-run-sync in the flag set with no declared constant -> CI stays red after the boundary fix

Suggested fix: Update the two standalone snapshot expectations, add finishRun to the legacy test's stream stub, and declare a WIZARD_RUN_SYNC_FLAG_KEY constant beside the other flag keys, then use it in the flag set and in selectRemoteSync.

Comment on lines +104 to 116
await Promise.allSettled(
[...shutdownFns].map((fn) => fn(error || code ? 'failed' : 'cancelled')),
);

// 2. Capture error in analytics. An 'error' ending with no Error object
// is captured as its code and message.
const status = options?.status ?? (error ? 'error' : 'cancelled');
const captured =
error ??
(status === 'error'
? new WizardError(message, undefined, code)
: undefined);
if (captured) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

robot comment

The hook outcome is derived from error || code while status five lines down is derived from options.status ?? error

  • Let explicit options.status win options.status ? (options.status === 'cancelled' ? 'cancelled' : 'failed') : (error || code ? 'failed' : 'cancelled')
  • add a { code, status: 'cancelled' } case to the new wizard-abort.test.ts block, and file the analytics status line separately

Comment on lines +389 to 410
this.destinations
.filter(
(d) => d.name !== 'posthog' || this.remoteSync === 'wizard-session',
)
.map((d) =>
// eslint-disable-next-line @typescript-eslint/no-empty-function
d.send(event, payload).catch(() => {}),
),
);
}

private selectRemoteSync(): void {
if (this.remoteSync || !this.store.session.credentials) return;
if (this.store.session.runPhase === RunPhase.Idle) return;
const flags = this.getFlags?.();
if (!flags) return;
this.remoteSync =
flags['wizard-run-sync'] === 'wizard-run'
? 'wizard-run'
: 'wizard-session';
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

robot comment

A push before the flag cache fills drops the session create

A push before getAllFlagsForWizard() has cached the flags drops the WizardSession create, and every later push is an update to a session that does not exist.

The order holds today only because run-agent-legacy.ts:170 fetches first. Either bail out before mutating created while the selection is pending, or default selectRemoteSync to wizard-session when flags are null. A test that pushes with null flags first would pin it.

Comment on lines +49 to +57
if (!subject) throw new Error('blank task name');
const rawStatus = item.sourceStatus ?? item.status;
const status = STATUS[rawStatus];
if (!Object.hasOwn(STATUS, rawStatus))
throw new Error('invalid task status');
const occurrence = (occurrences.get(subject) ?? 0) + 1;
occurrences.set(subject, occurrence);
const id = item.id
? JSON.stringify([item.source, item.id])

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

robot comment

A blank task name or unknown status stops run sync for the rest of the run

snapshot throws on a blank label or an unknown status, and capture then disables task sync for the rest of the run.

Pi's TaskCreate accepts empty content, so one empty call from the model ends run sync silently. The store is lenient for the TUI. Suggest a placeholder for blank names, unknown -> created, drop sourceStatus, and keep the throw for duplicate ids and the size cap.

Comment thread src/ui/tui/store.ts
Comment on lines +1156 to +1161
.filter(
(t) =>
todos.length > 0 &&
t.done &&
(t.source ? !sources.has(t.source) : !incomingLabels.has(t.label)),
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

robot comment

An empty snapshot wipes completed tasks from earlier agents

todos.length > 0 && makes an empty snapshot clear rows kept from other sources. Two emitters send an empty list without meaning "clear": the SDK path after deleting the last task, and the orchestrator's first renderQueue() when the seed produced nothing. In a composed run that erases the earlier program's completed rows.

Suggest dropping the length guard so rows from other sources survive empty input. If an explicit clear is ever needed, add a store method for it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the potential issue: The same filter drops failed rows too. It keeps earlier rows only when they're done, and done means completed. Now that the orchestrator reports failed rather than completed, a failed task from an earlier agent no longer survives the next agent's first sync.

the orchestrator runs the composed integration and one task fails -> the next agent sends its first task list under a new source -> the failed row isn't retained -> it vanishes from the task panel, and the next PUT deletes it from the WizardRun

Suggested fix: Keep any finished row from another source, failed or completed, rather than only done ones.

Comment thread src/lib/runners/run-wizard.ts Outdated
Comment on lines +242 to +244
unregisterShutdown = registerShutdown((outcome) =>
activeStream.shutdown(2000, outcome),
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

robot comment

This hook shuts the stream down while the phase is still Running, so the last legacy flush reports running and the Error phase set later by ui.outroError never goes out.

Set RunPhase.Error when the phase is Running, the same guard `onSignal uses, then call shutdown.

@gewenyu99 gewenyu99 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me @fercgomes thank you. I have a big moving stack, but I think this will fit well in the refactor. I will maybe add more tests in a separate PR to lock store <-> task stream changes more exactly.

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.

Comment thread src/lib/task-stream/wizard-run-sync.ts Outdated
if (!creds) return;
const response = await this.bounded(
async (requestSignal) => {
await refreshAccessTokenIfNeeded(

@gewenyu99 gewenyu99 Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the potential issue: Run sync refreshes the login's OAuth token from its background queue while the agent is still running. The agent keeps the token it started with for gateway re-mints and MCP calls, and the repo's own rule is that the agent can't swap tokens mid-run.

wizard-run variant, one-hour token, run still going after 10 minutes -> the next task PUT sees under 50 minutes left and rotates the grant -> the agent's next gateway re-mint or MCP call sends the replaced token -> 401 partway through the install

token endpoint answers slower than the 5 s request bound -> the CLI aborts after the server may already have rotated the refresh token -> the next refresh fails with invalid_grant and the login is marked revoked

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

? 🤔 I will confirm this myself my boy might be over stating this

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lemme stack a PR on top this is our slop

Comment thread src/env.ts
// yargs .env('POSTHOG_WIZARD') would claim them as CLI options and
// .strictOptions() would reject the run unless the wizard also declared them,
// which would tie every sandbox deploy to an npm release of this package.
| 'POSTHOG_WIZARD_RUN_ID'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the potential issue: POSTHOG_WIZARD_RUN_ID uses the prefix that the comment right above it says sandbox keys must avoid. yargs claims every POSTHOG_WIZARD_* variable as a CLI option, and strict parsing rejects any it doesn't know. This release declares a hidden --run-id to get past that, but the release the cloud worker installs by default, 2.74.1, doesn't.

the worker starts setting POSTHOG_WIZARD_RUN_ID -> it installs its default wizard 2.74.1, which has strict parsing and no run-id option -> yargs rejects the run with Unknown argument: runId -> every default cloud run dies before it starts

Suggested fix: The worker change hasn't shipped, so the rename is free now. Pick a name outside the prefix, like the neighbouring POSTHOG_TASK_RUN_ID, and drop the hidden option and the NEVER_FROM_ENV entry.

Comment thread src/ui/headless-ui.ts Outdated
}

setCredentials(credentials: Credentials): void {
this.store.setCredentials(credentials);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the potential issue: HeadlessUI.setCredentials now forwards to the store, and the store's setCredentials captures wizard: auth complete. Headless and --ci runs never sent that event before, since the logging UI's version was a no-op.

headless or --ci run authenticates -> HeadlessUI forwards the credentials to the store -> the store captures wizard: auth complete -> login funnels start counting non-interactive runs

Suggested fix: Forward through a store method that sets credentials without the capture, the way setAccessToken already does.

Comment thread src/shared/constants.ts
'health_issue:read',
'wizard_session:read',
'wizard_session:write',
'wizard_run:write',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the potential issue: Adding wizard_run:write to the base scope list means every login and refresh counts it as missing until the grant includes it. The orchestrator's failure message then tells the user to re-authorize for that permission, including on the default wizard-session variant, which never uses it.

a region's OAuth app doesn't allow wizard_run:write yet -> login succeeds with it missing -> an orchestrated run fails a required task -> the error says the run was authorized without wizard_run:write and asks the user to approve all permissions -> re-authorizing returns the same grant

a token from before this release refreshes -> refresh can't widen the grant, so wizard_run:write is marked missing -> the same misleading error on the next orchestrated failure

Suggested fix: Build that message from the scopes the orchestrator's tasks actually need, so an unused grant like wizard_run:write never shows up in it.

Run sync refreshed the login's OAuth token from its own queue while the
agent kept the token it booted with, so a mid-run rotation left the Pi
gateway re-mint and PostHog MCP on a replaced token.

`src/agent/oauth-session.ts` now holds the run's credentials the way
`gateway-session.ts` holds gateway auth: one process-wide holder with a
shared in-flight rotation. The host supplies the rotation, since the agent
never talks OAuth. Run sync, the Pi gateway re-mints and Pi MCP read the
current token from it, and the MCP token env follows each rotation. Dev
builds log the holder's lifecycle with token fingerprints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gewenyu99 and others added 2 commits September 24, 2026 19:20
The gateway cache was keyed by the OAuth access token, so every rotation
missed it and minted again, and mints are capped per week. The OAuth
session now tracks each login's token lineage, and the gateway keys its
cache by the lineage's first token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tsx runs leave NODE_ENV unset, so IS_DEV hid the lines there. The --local-* flags use the same !IS_PRODUCTION_BUILD gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gewenyu99
gewenyu99 added this pull request to stack #1359 September 25, 2026 00:05

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a fix for us to get a single place to manage oauth state stacked on your PR. I'll tentatively approve this one, that one covers the big token refresh issue here which is blocking

gewenyu99 and others added 7 commits September 24, 2026 20:18
Drop the token hashes and per-reader lines; CodeQL flags hashing a credential.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Type run-sync task items locally, so task-stream no longer imports the TUI store.
- Declare `WIZARD_RUN_SYNC_FLAG_KEY` beside the other flag keys.
- Expect task `id` and `source` in the standalone runner snapshots.
- Give the legacy runner's stream stub a `finishRun`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…zard into posthog/run-sync-no-refresh

# Conflicts:
#	src/lib/task-stream/wizard-run-sync.ts
The agent, run sync and the host all read it, so it belongs beside auth-session-state rather than behind the agent entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fercgomes
fercgomes marked this pull request as draft September 25, 2026 15:00
@fercgomes
fercgomes marked this pull request as ready for review September 25, 2026 15:04
@fercgomes
fercgomes merged commit c6b90b0 into main Sep 25, 2026
19 checks passed
@fercgomes
fercgomes deleted the codex/wizard-run-sync branch September 25, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants