Skip to content

fix(warehouse): ask once per source, and report what was actually created - #360

Merged
edwinyjlim merged 2 commits into
mainfrom
tom/warehouse-skill-credential-batching
Aug 26, 2026
Merged

edwinyjlim merged 2 commits into
mainfrom
tom/warehouse-skill-credential-batching

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

Two warehouse skills tell the wizard agent how to collect credentials. Both were out of step with the runtime, and the install-flow one gave advice the agent could not follow.

The wizard's wizard_ask batching guard counted every call in a run. After 3 calls in a row it refused to send the ask. A run routinely carries 5–8 in-cli sources, and Postgres alone needs 5 fields, so the third source tripped the guard.

integration-v2/warehouse had been written around that guard. Its tenet 2 said "don't make one call per source" and told the agent to pack several sources into one call. A call holds 8 questions. Five sources need 15–25 fields, so the instruction was impossible. Agents gave up and fell back to browser deep links.

The reporting rules then let a run with nothing created read as a success. Measured on project 2 since 2026-08-12, prod builds, internal users excluded: of 697 warehouse tasks that reported completed, only 247 produced a data warehouse source created event. That is a ~65% silent no-op rate. Runs whose credential ask was cancelled created a source 17% of the time; runs without a cancellation, 47%.

Changes

The wizard now counts the batching guard per subject instead of per run, so one call per source is never interrupted. Both skills now match that contract.

Credential batching (data-warehouse-source tenet 2 + step 4, integration-v2/warehouse tenet 2 + workflow):

  • Ask for one source's fields in one wizard_ask call, and set subject to the source kind.
  • Never put two different sources in one call.
  • Reuse a subject only for the same source, such as re-asking after a validation failure.
  • The one-time nudge is retryable. Send the call again; do not stop and do not fall back to the browser because of it.

More sources than one prompt can hold — the case the telemetry shows is common:

  • Work through the sources one at a time, and create each one before starting the next.
  • Split a kind that needs more than 8 fields into two calls with the same subject.
  • Never drop a source because the agent thinks it has asked too many questions.
  • A cancellation applies to that one source. Keep going, and stop asking only after two cancellations in a row.

Reporting, so a no-op run cannot read as a success:

  • The report opens with the count created: Created N of M detected sources in PostHog.
  • A deep link is a handoff, not a created source.
  • With no source created, say so in that first line and give the reason. Do not call the run complete, successful, or set up.
  • Each outcome maps onto a status complete_task already accepts. No new vocabulary: done only when at least one source was created; not needed when nothing was created and a retry cannot change that (the user declined, the hosts are unreachable, every source is deep-link); failed only for a tool error a retry could clear. The two [ABORT] cases are unchanged, and a test pins that no third one appeared.

One related fix: integration-v2/warehouse step 7 used to emit [ABORT] Source creation failed on the first failure, which ended the run and abandoned the remaining sources. It now records the error, gives that source its link, and continues. The abort is reserved for a run where every creation failed.

The runtime side lands in PostHog/wizard#1146. The two changes are one change: the runtime reads subject, and the skill sets it.

Test plan

npm test — 173 tests pass, 21 files. npm run build — both skill ZIPs emit cleanly.

New file scripts/lib/tests/data-warehouse-source-skill.test.js pins the cross-repo contract for both skills: the subject tag, the per-subject guard, the ban on packing two sources into one call, the retryable framing of the nudge, the cancel/timeout refund promise, keeping going after one cancellation, the created-count line, and the complete_task status mapping. It also asserts that no new [ABORT] case was invented.

…ated

The wizard runtime now counts its wizard_ask batching guard per `subject`
instead of per run, so one call per detected source is never interrupted.
Both warehouse skills now match that contract.

The install-flow step skill told the agent the opposite: batch up front and
pack several sources into one call. Five sources need 15-25 fields, so that
advice was impossible, and agents fell back to browser links instead. It now
asks for one tagged call per source, and it scopes a cancellation to that
source rather than to the whole round.

Telemetry showed 697 warehouse tasks reporting completed while only 247
created a source. Both skills now open the report with the count they
created, state that a deep link is a handoff rather than a connection, and
map each outcome onto a status complete_task already accepts: `done` only
when at least one source was created, `not needed` when a retry cannot help,
`failed` only for a tool error a retry could clear.

Adds a test pinning the cross-repo contract in both skills.
@Gilbert09
Gilbert09 requested review from a team as code owners August 26, 2026 08:19
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/groq
  • /wizard-ci ai-observability/manual-capture
Show more apps
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit

Results will be posted here when complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants