Skip to content

fix(warehouse-e2e): stop the secret rule refusing every credential question - #1161

Merged
edwinyjlim merged 2 commits into
mainfrom
tom/warehouse-e2e-ask-rules
Aug 28, 2026
Merged

edwinyjlim merged 2 commits into
mainfrom
tom/warehouse-e2e-ask-rules

Conversation

@Gilbert09

@Gilbert09 Gilbert09 commented Aug 28, 2026 •

Copy link
Copy Markdown
Member

Problem

The first real run of the warehouse e2e tier failed the ask contract check on all three legs:

leg unanswered
warehouse/stripe-node 2
warehouse/multi-source-next 1
warehouse/monorepo-env 3

Run: wizard-workbench 33128463033.

Every one of those questions was refused, not unmatched.

The data-warehouse-source-setup skill tells the agent to collect a warehouse credential as ordinary, non-sensitive text, because external-data-sources-create rejects a { secretRef }. A secret askAnswers rule answers only a sensitive text question, and refuses every other shape. So the stripe and password rules refused every credential the skill asked for, the run answered with the e2e sentinel, and no source could ever be created.

The LLM trace for the run shows the agent quoting that skill line as its reason:

Per skill instructions: "For credentials you'll hand straight to those tools, collect them as normal (non-sensitive) text answers so you get the real value." So I should treat all these fields as plain text answers rather than marking them sensitive.

The same traces give the real question ids, which no rule covered either:

leg ids the agent asked
stripe-node api_key, account_id, table_prefix
monorepo-env host, port, database, user, password, schema; then stripe_secret_key, …
multi-source-next pg-host, pg-port, pg-database, pg-user, pg-password, pg-schema

Replaying the old rules over those batches reproduces 2, 3 and 1 refusals exactly.

Changes

  • Drop secret from both rules. Every value here is a fake, unroutable placeholder. The workbench's no secret leakage check is the rail that keeps them out of the report, the result payload, the MCP journal and the captured frames. The secret mechanism itself is untouched, so another program can still use it.
  • account[ _-]?id ahead of stripe. The old stripe rule matched "Stripe Account ID (optional)" and would have handed it the API key.
  • connection[ _-]?string for the optional Postgres and Supabase field, built from the same E2E_PG_* vars.
  • author on the user rule, for the Hugging Face namespace field.
  • A trailing credential net, so a token question the vendor rules miss takes a placeholder instead of the sentinel.

Rules are first-match-wins, and every rule is tested against the question id before any rule is tested against the prompt. askAnswerNotes in the JSON records the precedence the order relies on, including one deliberate trade: the net claims a vendor-less api_key on the id pass, ahead of the stripe rule that would claim the same question on the prompt pass. A placeholder token and a placeholder Stripe key are equally fake, and a question that reaches the sentinel breaks the create it feeds.

The net reads E2E_API_TOKEN, added by the companion workbench PR. An unset var leaves the rule inert, so this degrades to today's behaviour on an older runner.

Verification

Re-ran the warehouse/stripe-node leg on this branch:
wizard-workbench 33161903184
(wizard_ref: tom/warehouse-e2e-ask-rules).

E2E_CHECK PASS ask contract — 1 batch(es), max 3 questions, 0 unanswered

Was 2 refusals on the same leg. The agent asked the same three questions and every one was answered.

The leg still fails, on three checks that are not about the rules:

E2E_CHECK FAIL no silent no-op — Stripe: expected a create, the stub never saw one
E2E_CHECK FAIL report written — no report at /tmp/wizard-e2e-stripe-node/posthog-warehouse-report.md
E2E_CHECK FAIL abort — unexpected abort: "Source creation failed"

The agent's tool inventory for that run was Read, Glob, Bash, Agent, Skill, TaskCreate,
TaskUpdate, mcp__wizard-tools__check_env_keys and mcp__wizard-tools__wizard_ask. There is no
PostHog MCP tool in it, so external-data-sources-create is unreachable. That is the next blocker,
and it sits upstream of both PRs.

Worth noting the run failed honestly this time: it aborted instead of reporting a source it never
created, so claimedConnected found nothing to flag.

Test plan

  • vitest run — 2419 tests pass. The warehouse profile suite grew from 5 tests to 8; new cases replay the real question ids from each leg and pin the account_id / stripe and net / vendor precedence.
  • e2e-flow-snapshot snapshot updated for the new rule list.
  • prettier --check and eslint clean on the touched files. tsc --noEmit reports 27 errors, all pre-existing on main and none in e2e-harness or warehouse-source.

Companion: PostHog/wizard-workbench#3627

…estion

The first real run of the warehouse e2e tier failed the ask contract on all
three legs: 2 questions on stripe-node, 1 on multi-source-next, 3 on
monorepo-env. Every one was a refusal, not a missing rule.

The data-warehouse-source-setup skill tells the agent to collect a warehouse
credential as ordinary, non-sensitive text, because
`external-data-sources-create` rejects a `{ secretRef }`. A `secret` askAnswers
rule answers only a *sensitive* text question. So the `stripe` and `password`
rules refused every credential the skill asked for, the run answered with the
`e2e` sentinel, and no source could ever be created. The LLM trace for the run
shows the agent quoting that skill line as its reason for leaving `sensitive`
off.

Drop `secret` from both rules. The values are fake, unroutable placeholders,
and the workbench's "no secret leakage" check is what keeps them out of the
report, the result payload, the MCP journal and the captured frames.

Also close the routing gaps the same trace exposed:

- `account[ _-]?id` ahead of `stripe`, so "Stripe Account ID (optional)" takes
  an account id instead of the API key.
- `connection[ _-]?string` for the optional Postgres and Supabase field.
- `author` on the user rule, for the Hugging Face namespace field.
- a trailing credential net, so a token question the vendor rules miss gets a
  placeholder instead of the sentinel.

Tests pin the precedence the order relies on, and replay the real question ids
the run asked: api_key, account_id, table_prefix, stripe_secret_key, and both
the bare and `pg-` prefixed Postgres fields.

The net needs `E2E_API_TOKEN` from the workbench runner. An unset var leaves
the rule inert, so this degrades to today's behaviour on an older runner.
@Gilbert09
Gilbert09 requested review from a team as code owners August 28, 2026 10:02
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/groq
  • /wizard-ci ai-observability/manual-capture
Show more apps
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

@edwinyjlim
edwinyjlim merged commit f0e024d into main Aug 28, 2026
17 checks passed
@edwinyjlim
edwinyjlim deleted the tom/warehouse-e2e-ask-rules branch August 28, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants