Skip to content

feat(telemetry): stamp wizard_ask outcome events with the ask subject - #1162

Merged
edwinyjlim merged 1 commit into
mainfrom
posthog/wizard-ask-subject-telemetry
Aug 28, 2026
Merged

edwinyjlim merged 1 commit into
mainfrom
posthog/wizard-ask-subject-telemetry

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

The wizard_ask answered / wizard_ask cancelled events — the primary signal
for how users respond to the in-CLI credential prompt — carried the run
source but not the per-call subject (the thing the questions collect for,
e.g. a data-warehouse source kind). The seeded warehouse task asks one
wizard_ask call per detected source
, so a cancellation was visible at the
run level but could not be attributed to which source's credentials the user
actually declined.

That is the gap that makes the flow's dominant failure mode hard to act on: the
largest structured signal (users cancelling the credential prompt) cannot be
broken down by the per-source dimension the flow is analysed on, even though the
subject is already known at the call site and already stamped on the sibling
wizard_ask capped event.

Changes

  • Add an optional subject to WizardAskRequest and stamp it onto both
    wizard_ask answered and wizard_ask cancelled in the ask bridge.
  • Thread the subject from both harness facades (MCP wizard_ask and the pi
    native mirror), normalised through the same normaliseAskSubject the cap
    accounting uses, so the value joins cleanly to wizard_ask capped.

Purely additive — no behaviour change, and the two facades are updated
symmetrically so they can't drift.

Why

The warehouse-connect telemetry breaks outcomes down per provider, but the
cancellation event that drives most of the loss was provider-blind, so the two
couldn't be joined. This closes that gap in the same spirit as the earlier
reason-on-skip change.

Test plan

  • Extended wizard-ask-bridge unit tests to assert subject is stamped on both
    the answered and cancelled events.
  • pnpm build && pnpm test (ask-bridge + wizard-tools + pi harness suites) and
    pnpm lint pass (0 errors).

Created with PostHog Desktop

The `wizard_ask answered` / `wizard_ask cancelled` events carried the run
`source` but not the per-call `subject` (the thing the questions collect for,
e.g. a data-warehouse source kind). The warehouse task asks one `wizard_ask`
call per detected source, so without the subject a cancellation can be seen at
the run level but not attributed to which source's credentials the user
declined — the single largest structured signal in this flow cannot be broken
down by the very dimension the funnel is analysed on.

Thread the subject (already computed for the cap accounting, and already on the
`wizard_ask capped` event) through the ask bridge onto both outcome events,
normalised via the same `normaliseAskSubject` so the values join across the
three events. Purely additive: no behaviour change, both harness facades
(MCP + pi) updated symmetrically.

Generated-By: PostHog Desktop
Task-Id: c1dc9e8e-929c-49eb-988c-054298a47462
@Gilbert09
Gilbert09 requested a review from a team as a code owner August 28, 2026 11:55
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/groq
  • /wizard-ci ai-observability/manual-capture
Show more apps
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

@edwinyjlim
edwinyjlim merged commit d1cce49 into main Aug 28, 2026
20 checks passed
@edwinyjlim
edwinyjlim deleted the posthog/wizard-ask-subject-telemetry branch August 28, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants