Repository navigation
feat: synchronize Wizard CLI executions with WizardRun - #1323
Conversation
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
|
@fercgomes could you rebase / merge in main? it's mostly reorganizing. will review right after |
0b26cfb to
eb9d87f
Compare
| RunPhase, | ||
| type Credentials, | ||
| type WizardSession, | ||
| } from '@lib/wizard-session'; | ||
| import type { TaskItem } from '@ui/tui/store'; | ||
| import { refreshAccessTokenIfNeeded } from '@lib/programs/authenticate'; | ||
| import { isGrantRevoked } from '@shared/auth-session-state'; | ||
| import { logToFile } from '@utils/debug'; | ||
| import { parseRetryAfter } from './destinations/posthog'; |
There was a problem hiding this comment.
robot comment
pnpm test:arch fails on this branch: wizard-run-sync.ts -> src/ui/tui/store.ts is a new lib->tui edge (the TaskItem type import on line 11).
pnpm test runs every project, so CI is red. snapshot() only needs { id?, source?, sourceStatus?, label, status }. A local type here removes the edge without grandfathering it.
There was a problem hiding this comment.
Here's the potential issue: The import edge isn't the only red test. Four more fail at this head, so the local type alone won't turn CI green.
pnpm test at this head -> the standalone runner tests expect tasks without id and source, the legacy runner test's stream stub has no finishRun so the run exits 1, and the flag key test finds wizard-run-sync in the flag set with no declared constant -> CI stays red after the boundary fix
Suggested fix: Update the two standalone snapshot expectations, add finishRun to the legacy test's stream stub, and declare a WIZARD_RUN_SYNC_FLAG_KEY constant beside the other flag keys, then use it in the flag set and in selectRemoteSync.
| await Promise.allSettled( | ||
| [...shutdownFns].map((fn) => fn(error || code ? 'failed' : 'cancelled')), | ||
| ); | ||
|
|
||
| // 2. Capture error in analytics. An 'error' ending with no Error object | ||
| // is captured as its code and message. | ||
| const status = options?.status ?? (error ? 'error' : 'cancelled'); | ||
| const captured = | ||
| error ?? | ||
| (status === 'error' | ||
| ? new WizardError(message, undefined, code) | ||
| : undefined); | ||
| if (captured) { |
There was a problem hiding this comment.
robot comment
The hook outcome is derived from error || code while status five lines down is derived from options.status ?? error
- Let explicit
options.statuswinoptions.status ? (options.status === 'cancelled' ? 'cancelled' : 'failed') : (error || code ? 'failed' : 'cancelled') - add a
{ code, status: 'cancelled' }case to the newwizard-abort.test.ts block, and file the analytics status line separately
| this.destinations | ||
| .filter( | ||
| (d) => d.name !== 'posthog' || this.remoteSync === 'wizard-session', | ||
| ) | ||
| .map((d) => | ||
| // eslint-disable-next-line @typescript-eslint/no-empty-function | ||
| d.send(event, payload).catch(() => {}), | ||
| ), | ||
| ); | ||
| } | ||
|
|
||
| private selectRemoteSync(): void { | ||
| if (this.remoteSync || !this.store.session.credentials) return; | ||
| if (this.store.session.runPhase === RunPhase.Idle) return; | ||
| const flags = this.getFlags?.(); | ||
| if (!flags) return; | ||
| this.remoteSync = | ||
| flags['wizard-run-sync'] === 'wizard-run' | ||
| ? 'wizard-run' | ||
| : 'wizard-session'; | ||
| } | ||
| } |
There was a problem hiding this comment.
robot comment
A push before the flag cache fills drops the session create
A push before getAllFlagsForWizard() has cached the flags drops the WizardSession create, and every later push is an update to a session that does not exist.
The order holds today only because run-agent-legacy.ts:170 fetches first. Either bail out before mutating created while the selection is pending, or default selectRemoteSync to wizard-session when flags are null. A test that pushes with null flags first would pin it.
| if (!subject) throw new Error('blank task name'); | ||
| const rawStatus = item.sourceStatus ?? item.status; | ||
| const status = STATUS[rawStatus]; | ||
| if (!Object.hasOwn(STATUS, rawStatus)) | ||
| throw new Error('invalid task status'); | ||
| const occurrence = (occurrences.get(subject) ?? 0) + 1; | ||
| occurrences.set(subject, occurrence); | ||
| const id = item.id | ||
| ? JSON.stringify([item.source, item.id]) |
There was a problem hiding this comment.
robot comment
A blank task name or unknown status stops run sync for the rest of the run
snapshot throws on a blank label or an unknown status, and capture then disables task sync for the rest of the run.
Pi's TaskCreate accepts empty content, so one empty call from the model ends run sync silently. The store is lenient for the TUI. Suggest a placeholder for blank names, unknown -> created, drop sourceStatus, and keep the throw for duplicate ids and the size cap.
| .filter( | ||
| (t) => | ||
| todos.length > 0 && | ||
| t.done && | ||
| (t.source ? !sources.has(t.source) : !incomingLabels.has(t.label)), | ||
| ); |
There was a problem hiding this comment.
robot comment
An empty snapshot wipes completed tasks from earlier agents
todos.length > 0 && makes an empty snapshot clear rows kept from other sources. Two emitters send an empty list without meaning "clear": the SDK path after deleting the last task, and the orchestrator's first renderQueue() when the seed produced nothing. In a composed run that erases the earlier program's completed rows.
Suggest dropping the length guard so rows from other sources survive empty input. If an explicit clear is ever needed, add a store method for it.
There was a problem hiding this comment.
Here's the potential issue: The same filter drops failed rows too. It keeps earlier rows only when they're done, and done means completed. Now that the orchestrator reports failed rather than completed, a failed task from an earlier agent no longer survives the next agent's first sync.
the orchestrator runs the composed integration and one task fails -> the next agent sends its first task list under a new source -> the failed row isn't retained -> it vanishes from the task panel, and the next PUT deletes it from the WizardRun
Suggested fix: Keep any finished row from another source, failed or completed, rather than only done ones.
| unregisterShutdown = registerShutdown((outcome) => | ||
| activeStream.shutdown(2000, outcome), | ||
| ); |
There was a problem hiding this comment.
robot comment
This hook shuts the stream down while the phase is still Running, so the last legacy flush reports running and the Error phase set later by ui.outroError never goes out.
Set RunPhase.Error when the phase is Running, the same guard `onSignal uses, then call shutdown.
There was a problem hiding this comment.
This looks good to me @fercgomes thank you. I have a big moving stack, but I think this will fit well in the refactor. I will maybe add more tests in a separate PR to lock store <-> task stream changes more exactly.
gewenyu99
left a comment
There was a problem hiding this comment.
🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.
| if (!creds) return; | ||
| const response = await this.bounded( | ||
| async (requestSignal) => { | ||
| await refreshAccessTokenIfNeeded( |
There was a problem hiding this comment.
Here's the potential issue: Run sync refreshes the login's OAuth token from its background queue while the agent is still running. The agent keeps the token it started with for gateway re-mints and MCP calls, and the repo's own rule is that the agent can't swap tokens mid-run.
wizard-run variant, one-hour token, run still going after 10 minutes -> the next task PUT sees under 50 minutes left and rotates the grant -> the agent's next gateway re-mint or MCP call sends the replaced token -> 401 partway through the install
token endpoint answers slower than the 5 s request bound -> the CLI aborts after the server may already have rotated the refresh token -> the next refresh fails with invalid_grant and the login is marked revoked
There was a problem hiding this comment.
? 🤔 I will confirm this myself my boy might be over stating this
There was a problem hiding this comment.
Lemme stack a PR on top this is our slop
| // yargs .env('POSTHOG_WIZARD') would claim them as CLI options and | ||
| // .strictOptions() would reject the run unless the wizard also declared them, | ||
| // which would tie every sandbox deploy to an npm release of this package. | ||
| | 'POSTHOG_WIZARD_RUN_ID' |
There was a problem hiding this comment.
Here's the potential issue: POSTHOG_WIZARD_RUN_ID uses the prefix that the comment right above it says sandbox keys must avoid. yargs claims every POSTHOG_WIZARD_* variable as a CLI option, and strict parsing rejects any it doesn't know. This release declares a hidden --run-id to get past that, but the release the cloud worker installs by default, 2.74.1, doesn't.
the worker starts setting POSTHOG_WIZARD_RUN_ID -> it installs its default wizard 2.74.1, which has strict parsing and no run-id option -> yargs rejects the run with Unknown argument: runId -> every default cloud run dies before it starts
Suggested fix: The worker change hasn't shipped, so the rename is free now. Pick a name outside the prefix, like the neighbouring POSTHOG_TASK_RUN_ID, and drop the hidden option and the NEVER_FROM_ENV entry.
| } | ||
|
|
||
| setCredentials(credentials: Credentials): void { | ||
| this.store.setCredentials(credentials); |
There was a problem hiding this comment.
Here's the potential issue: HeadlessUI.setCredentials now forwards to the store, and the store's setCredentials captures wizard: auth complete. Headless and --ci runs never sent that event before, since the logging UI's version was a no-op.
headless or --ci run authenticates -> HeadlessUI forwards the credentials to the store -> the store captures wizard: auth complete -> login funnels start counting non-interactive runs
Suggested fix: Forward through a store method that sets credentials without the capture, the way setAccessToken already does.
| 'health_issue:read', | ||
| 'wizard_session:read', | ||
| 'wizard_session:write', | ||
| 'wizard_run:write', |
There was a problem hiding this comment.
Here's the potential issue: Adding wizard_run:write to the base scope list means every login and refresh counts it as missing until the grant includes it. The orchestrator's failure message then tells the user to re-authorize for that permission, including on the default wizard-session variant, which never uses it.
a region's OAuth app doesn't allow wizard_run:write yet -> login succeeds with it missing -> an orchestrated run fails a required task -> the error says the run was authorized without wizard_run:write and asks the user to approve all permissions -> re-authorizing returns the same grant
a token from before this release refreshes -> refresh can't widen the grant, so wizard_run:write is marked missing -> the same misleading error on the next orchestrated failure
Suggested fix: Build that message from the scopes the orchestrator's tasks actually need, so an unused grant like wizard_run:write never shows up in it.
Run sync refreshed the login's OAuth token from its own queue while the agent kept the token it booted with, so a mid-run rotation left the Pi gateway re-mint and PostHog MCP on a replaced token. `src/agent/oauth-session.ts` now holds the run's credentials the way `gateway-session.ts` holds gateway auth: one process-wide holder with a shared in-flight rotation. The host supplies the rotation, since the agent never talks OAuth. Run sync, the Pi gateway re-mints and Pi MCP read the current token from it, and the MCP token env follows each rotation. Dev builds log the holder's lifecycle with token fingerprints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gateway cache was keyed by the OAuth access token, so every rotation missed it and minted again, and mints are capped per week. The OAuth session now tracks each login's token lineage, and the gateway keys its cache by the lineage's first token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tsx runs leave NODE_ENV unset, so IS_DEV hid the lines there. The --local-* flags use the same !IS_PRODUCTION_BUILD gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gewenyu99
left a comment
There was a problem hiding this comment.
I have a fix for us to get a single place to manage oauth state stacked on your PR. I'll tentatively approve this one, that one covers the big token refresh issue here which is blocking
Drop the token hashes and per-reader lines; CodeQL flags hashing a credential. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Type run-sync task items locally, so task-stream no longer imports the TUI store. - Declare `WIZARD_RUN_SYNC_FLAG_KEY` beside the other flag keys. - Expect task `id` and `source` in the standalone runner snapshots. - Give the legacy runner's stream stub a `finishRun`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…zard into posthog/run-sync-no-refresh # Conflicts: # src/lib/task-stream/wizard-run-sync.ts
The agent, run sync and the host all read it, so it belongs beside auth-session-state rather than behind the agent entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o codex/wizard-run-sync
Wizard executions now publish task progress to WizardRun when the
wizard-run-syncflag selectswizard-run. A local execution creates one run for its selected program, sends ordered full task snapshots, and sends one terminal outcome after draining tasks. An assigned cloud execution publishes to its existing run and leaves finalization to the worker. Thewizard-sessionvariant and legacy headless launches retain WizardSession publishing.The CLI freezes task names across status, label, and order changes, retains completed and failed work from earlier agents, and rejects malformed snapshots without disabling later valid updates. The existing task stream owns delivery, and the shared OAuth session keeps run writes and agent gateway/MCP calls on the current login. The flag selection waits for the authenticated flag cache before sending a remote session create. Headless authentication preserves its previous analytics behavior.
The backend API shape and frontend remain unchanged. Cloud rollout still depends on the PostHog worker installing a Wizard version that accepts
POSTHOG_WIZARD_RUN_ID, or gating that environment variable by version: its current default, 2.74.1, rejects the new prefixed input. Both deployed Wizard OAuth apps also need awizard_run:writeceiling, and existing grants need reauthorization. Local creation POST retries remain disabled until deployed local idempotency is verified. Live backend integration was not run.Checks: focused WizardRun, store, UI, abort, and agent tests passed;
pnpm typecheck,pnpm test:arch, andpnpm lintpassed. An isolated production bundle built and contained no CI-only overrides. Localpnpm testis affected by concurrent writes todistand sandbox restrictions on local HTTP listeners/watchers; the PR's GitHub CI checks are the authoritative full-suite result.