Repository navigation
fix(warehouse-e2e): stop the secret rule refusing every credential question - #1161
Merged
Merged
Conversation
…estion
The first real run of the warehouse e2e tier failed the ask contract on all
three legs: 2 questions on stripe-node, 1 on multi-source-next, 3 on
monorepo-env. Every one was a refusal, not a missing rule.
The data-warehouse-source-setup skill tells the agent to collect a warehouse
credential as ordinary, non-sensitive text, because
`external-data-sources-create` rejects a `{ secretRef }`. A `secret` askAnswers
rule answers only a *sensitive* text question. So the `stripe` and `password`
rules refused every credential the skill asked for, the run answered with the
`e2e` sentinel, and no source could ever be created. The LLM trace for the run
shows the agent quoting that skill line as its reason for leaving `sensitive`
off.
Drop `secret` from both rules. The values are fake, unroutable placeholders,
and the workbench's "no secret leakage" check is what keeps them out of the
report, the result payload, the MCP journal and the captured frames.
Also close the routing gaps the same trace exposed:
- `account[ _-]?id` ahead of `stripe`, so "Stripe Account ID (optional)" takes
an account id instead of the API key.
- `connection[ _-]?string` for the optional Postgres and Supabase field.
- `author` on the user rule, for the Hugging Face namespace field.
- a trailing credential net, so a token question the vendor rules miss gets a
placeholder instead of the sentinel.
Tests pin the precedence the order relies on, and replay the real question ids
the run asked: api_key, account_id, table_prefix, stripe_secret_key, and both
the bare and `pg-` prefixed Postgres fields.
The net needs `E2E_API_TOKEN` from the workbench runner. An unset var leaves
the rule inert, so this degrades to today's behaviour on an older runner.
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
edwinyjlim
approved these changes
Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first real run of the warehouse e2e tier failed the
ask contractcheck on all three legs:warehouse/stripe-nodewarehouse/multi-source-nextwarehouse/monorepo-envRun: wizard-workbench 33128463033.
Every one of those questions was refused, not unmatched.
The
data-warehouse-source-setupskill tells the agent to collect a warehouse credential as ordinary, non-sensitive text, becauseexternal-data-sources-createrejects a{ secretRef }. AsecretaskAnswersrule answers only a sensitive text question, and refuses every other shape. So thestripeandpasswordrules refused every credential the skill asked for, the run answered with thee2esentinel, and no source could ever be created.The LLM trace for the run shows the agent quoting that skill line as its reason:
The same traces give the real question ids, which no rule covered either:
api_key,account_id,table_prefixhost,port,database,user,password,schema; thenstripe_secret_key, …pg-host,pg-port,pg-database,pg-user,pg-password,pg-schemaReplaying the old rules over those batches reproduces 2, 3 and 1 refusals exactly.
Changes
secretfrom both rules. Every value here is a fake, unroutable placeholder. The workbench'sno secret leakagecheck is the rail that keeps them out of the report, the result payload, the MCP journal and the captured frames. Thesecretmechanism itself is untouched, so another program can still use it.account[ _-]?idahead ofstripe. The oldstriperule matched "Stripe Account ID (optional)" and would have handed it the API key.connection[ _-]?stringfor the optional Postgres and Supabase field, built from the sameE2E_PG_*vars.authoron the user rule, for the Hugging Face namespace field.Rules are first-match-wins, and every rule is tested against the question id before any rule is tested against the prompt.
askAnswerNotesin the JSON records the precedence the order relies on, including one deliberate trade: the net claims a vendor-lessapi_keyon the id pass, ahead of thestriperule that would claim the same question on the prompt pass. A placeholder token and a placeholder Stripe key are equally fake, and a question that reaches the sentinel breaks the create it feeds.The net reads
E2E_API_TOKEN, added by the companion workbench PR. An unset var leaves the rule inert, so this degrades to today's behaviour on an older runner.Verification
Re-ran the
warehouse/stripe-nodeleg on this branch:wizard-workbench 33161903184
(
wizard_ref: tom/warehouse-e2e-ask-rules).Was 2 refusals on the same leg. The agent asked the same three questions and every one was answered.
The leg still fails, on three checks that are not about the rules:
The agent's tool inventory for that run was
Read,Glob,Bash,Agent,Skill,TaskCreate,TaskUpdate,mcp__wizard-tools__check_env_keysandmcp__wizard-tools__wizard_ask. There is noPostHog MCP tool in it, so
external-data-sources-createis unreachable. That is the next blocker,and it sits upstream of both PRs.
Worth noting the run failed honestly this time: it aborted instead of reporting a source it never
created, so
claimedConnectedfound nothing to flag.Test plan
vitest run— 2419 tests pass. The warehouse profile suite grew from 5 tests to 8; new cases replay the real question ids from each leg and pin theaccount_id/stripeand net / vendor precedence.e2e-flow-snapshotsnapshot updated for the new rule list.prettier --checkandeslintclean on the touched files.tsc --noEmitreports 27 errors, all pre-existing onmainand none ine2e-harnessorwarehouse-source.Companion: PostHog/wizard-workbench#3627