Repository navigation
fix(warehouse): ask once per source, and report what was actually created - #360
Merged
Merged
Conversation
…ated The wizard runtime now counts its wizard_ask batching guard per `subject` instead of per run, so one call per detected source is never interrupted. Both warehouse skills now match that contract. The install-flow step skill told the agent the opposite: batch up front and pack several sources into one call. Five sources need 15-25 fields, so that advice was impossible, and agents fell back to browser links instead. It now asks for one tagged call per source, and it scopes a cancellation to that source rather than to the whole round. Telemetry showed 697 warehouse tasks reporting completed while only 247 created a source. Both skills now open the report with the count they created, state that a deep link is a handoff rather than a connection, and map each outcome onto a status complete_task already accepts: `done` only when at least one source was created, `not needed` when a retry cannot help, `failed` only for a tool error a retry could clear. Adds a test pinning the cross-repo contract in both skills.
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
edwinyjlim
approved these changes
Aug 26, 2026
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two warehouse skills tell the wizard agent how to collect credentials. Both were out of step with the runtime, and the install-flow one gave advice the agent could not follow.
The wizard's
wizard_askbatching guard counted every call in a run. After 3 calls in a row it refused to send the ask. A run routinely carries 5–8 in-cli sources, and Postgres alone needs 5 fields, so the third source tripped the guard.integration-v2/warehousehad been written around that guard. Its tenet 2 said "don't make one call per source" and told the agent to pack several sources into one call. A call holds 8 questions. Five sources need 15–25 fields, so the instruction was impossible. Agents gave up and fell back to browser deep links.The reporting rules then let a run with nothing created read as a success. Measured on project 2 since 2026-08-12, prod builds, internal users excluded: of 697 warehouse tasks that reported completed, only 247 produced a
data warehouse source createdevent. That is a ~65% silent no-op rate. Runs whose credential ask was cancelled created a source 17% of the time; runs without a cancellation, 47%.Changes
The wizard now counts the batching guard per
subjectinstead of per run, so one call per source is never interrupted. Both skills now match that contract.Credential batching (
data-warehouse-sourcetenet 2 + step 4,integration-v2/warehousetenet 2 + workflow):wizard_askcall, and setsubjectto the source kind.More sources than one prompt can hold — the case the telemetry shows is common:
Reporting, so a no-op run cannot read as a success:
Created N of M detected sources in PostHog.complete_taskalready accepts. No new vocabulary:doneonly when at least one source was created;not neededwhen nothing was created and a retry cannot change that (the user declined, the hosts are unreachable, every source is deep-link);failedonly for a tool error a retry could clear. The two[ABORT]cases are unchanged, and a test pins that no third one appeared.One related fix:
integration-v2/warehousestep 7 used to emit[ABORT] Source creation failedon the first failure, which ended the run and abandoned the remaining sources. It now records the error, gives that source its link, and continues. The abort is reserved for a run where every creation failed.The runtime side lands in PostHog/wizard#1146. The two changes are one change: the runtime reads
subject, and the skill sets it.Test plan
npm test— 173 tests pass, 21 files.npm run build— both skill ZIPs emit cleanly.New file
scripts/lib/tests/data-warehouse-source-skill.test.jspins the cross-repo contract for both skills: the subject tag, the per-subject guard, the ban on packing two sources into one call, the retryable framing of the nudge, the cancel/timeout refund promise, keeping going after one cancellation, the created-count line, and thecomplete_taskstatus mapping. It also asserts that no new[ABORT]case was invented.