Repository navigation
Description
Summary
The wizard ends a successful run with posthog-setup-report.md — a manifest of what landed. That's useful as a record, but it leaves a real gap between "wizard finished" and "PostHog is integrated and merged." Closing that gap requires a coding agent (or a human) to discover several issues independently — each one familiar to the wizard team, but currently not surfaced in the report.
I'd like to propose the wizard emit a second artifact alongside the manifest: a coding-agent handoff doc with verification steps, known SDK quirks, and project-specific touches the wizard intentionally didn't make. The wizard already has all this knowledge at run-time. Encoding it in a standard handoff turns "90% done" into "one prompt away from done" for any agent picking up the work.
Related: #133 (wizard next-gen — error correction loop), #208 (redesign CLI output), #288 (review error messages + document them).
Concrete gaps from a recent v2.12.0 install
Running npx @posthog/wizard@latest against a Next.js 15 / App Router app with a handful of Anthropic call sites. The integration was excellent at speed and breadth — events, identify, reverse proxy, LLM analytics, dashboards — but six issues post-install required manual fixing before the work could merge:
-
Build failure on first
next build. The wizard placed// eslint-disable-next-line @typescript-eslint/no-explicit-anyone line above the first PostHog field rather than directly above the} as any)closing line where theanyactually lives. ESLint flagged the unused disable directive AND the unsuppressedany, breaking CI. -
Streaming LLM analytics broken at runtime. The wizard's templates use
messages.stream(...)on the wrappedPostHogAnthropicclient. But@posthog/ai'sWrappedMessagesonly interceptscreate(), and its return value is a plainPromise<Stream>(not anAPIPromise). When Anthropic's SDK calls.withResponse()on the wrapped value internally, it throwsmessages.create(...).withResponse is not a function. An independent reviewer reproduced this against the wizard-as-shipped state. The fix ismessages.create({ stream: true, ... })instead — but no agent would know to do this without already having@posthog/aiSDK details in their context. -
An LLM call site was missed. The wizard reported instrumenting N Anthropic call sites; the codebase had N+1 (the missed one being a multi-loop agent in a separate module). The wizard's grep didn't find it, presumably because the import was structured slightly differently. The handoff doc could prompt: "Verify all
new Anthropic()constructors andfrom '@anthropic-ai/sdk'imports have been migrated." -
Identify-from-cookie path missed. The wizard wired client
posthog.identifyon the picker's submit handler. But the same picker auto-redirects returning users (with the auth cookie present) to the next page without hitting that handler. So returning visitors stayed on anonymous distinct ids. The handoff could prompt: "Verify identify runs on every page-load entry point, not just the explicit auth flow." -
Project-specific glue not touched (correctly). The wizard wisely didn't touch agent guides / project-specific bootstrap scripts /
.env.example/ source-map upload pipelines. But it knows it added.env.localand aninstrumentation-client.ts, so it could suggest touches: "AddNEXT_PUBLIC_POSTHOG_*to your.env.exampleand any onboarding script that copies env. If you maintain an agent guide (AGENTS.md/CLAUDE.md), document the integration there." -
Token-absent posture not stated. Fresh worktrees / e2e environments without the project token produced posthog-js init warnings and
posthog-noderetry storms. The handoff could note: "If you have CI/e2e environments without PostHog, consider noop-shimminggetPostHogClientfor safe absence."
Proposed shape
A second artifact, e.g. posthog-next-steps.md, written next to the manifest. Five short sections:
# PostHog setup: next steps
## Verify before merging
- [ ] Run `npm run build` — catches eslint/type issues from generated code.
- [ ] Run unit tests — wizard-rewritten routes may have outdated mocks.
- [ ] Smoke test one streaming and one non-streaming LLM call — confirm $ai_generation events arrive.
- [ ] Smoke test one identify path — confirm distinct id matches your user model.
- [ ] Verify there are no remaining `new Anthropic()` / direct `@anthropic-ai/sdk` imports outside type imports.
## Known quirks for this stack
- @posthog/ai's PostHogAnthropic only wraps `messages.create()`. Don't use `messages.stream()` — it bypasses the wrapper. Use `messages.create({ stream: true })` and iterate the lower-level events.
- (per-stack: rotate this list as the wrapper SDKs evolve)
## Project glue we did NOT touch
- `.env.example` — add `NEXT_PUBLIC_POSTHOG_PROJECT_TOKEN` and `NEXT_PUBLIC_POSTHOG_HOST` placeholders.
- Worktree / monorepo bootstrap scripts — add a step that copies `.env.local` if your team uses one.
- Agent guides (`AGENTS.md` / `CLAUDE.md`) — document the singletons and event-naming conventions.
- Source maps — wire `posthog-cli sourcemap` in CI for production stack traces.
## Token-absent behavior
By default the SDK warns if the token is unset. If you have environments without PostHog, consider noop-shimming the singletons.
## Hand this to your coding agent
Copy and run:
> "Read `posthog-setup-report.md` and `posthog-next-steps.md`. Verify each item in the checklist. Fix any gaps. Open a PR with the changes and a summary of what was verified."The "hand this to your coding agent" block at the bottom is the key piece — it makes the integration finishable in one prompt.
Why this fits the wizard's distribution model
The wizard is already designed to be invoked from inside a larger development task (per #133's framing — "agents could trigger an integration within a larger development task"). That positioning is exactly where a handoff doc shines: the calling agent gets a deterministic, local, copy-pasteable next-step instead of having to discover gaps by trial and error.
It's also cheap to build: most of the content is per-stack static text the wizard team writes once, plus a few dynamic facts (which call sites were touched, which packages were installed, which env vars were added).
Open questions
- Should the handoff doc be a separate file or a section in the existing report? Separate keeps each artifact's purpose clean (manifest vs. action items).
- Should the wizard emit a machine-readable JSON version too, so the calling agent can parse-and-act rather than re-reading prose? Probably yes long-term, but the markdown version is the higher-leverage starting point.
- Where does this overlap with Wizard: the next generation #133's next-gen wizard with self-correction? Complementary: even a perfect wizard will have project-specific touches it shouldn't make, and a handoff doc tells the calling agent what those are.
Happy to PR a first cut if the team thinks this direction is worth pursuing.
Activity
ethangui commentedon May 9, 2026
PR #448 implements this with three deliberate deviations from the original proposal worth flagging:
The "Hand this to your coding agent" section is NOT in the doc. The originally-proposed embed of a prompt that says "Read both files" inside one of the files those instructions reference is a circular reference — burns tokens on every agent re-read for nothing. The prompt is now surfaced in the wizard's terminal scrollback after
tui.unmount()instead, via a new workflow-agnosticsetPostExitMessage/getPostExitMessageaccessor pair. The doc body is purely the agent's reading material.The "Known SDK quirks" section currently registers no quirks. A real end-to-end wizard run surfaced that the wizard's LLM-analytics code generation is non-deterministic across two implementation strategies (the
@posthog/aiPostHogAnthropicwrapper vs OpenTelemetry auto-instrumentation) for the same JS framework + same package set. The originally-proposedmessages.stream(...)quirk only applies to the wrapper path, so registering it would have been wrong half the time. The handoff is now strategy-agnostic and points the reader at the agent-written setup report for strategy-specific details. The team is the right owner to register quirks when strategy-agnostic ones emerge.The PR also surfaces a wizard-architectural bug that's worth a separate fix from the team.
agent-runner.tsmutatessession.outroData = Xon a session reference captured at the start of the run; by the time the mutation happens, everysetKeycall during the agent run has replaced the atom's top-level session, and the mutation goes to a stranded object. This silently dropsreportFile,changes, and any other field ofoutroDataend-to-end (messagesurvives only because of an explicit fallback inink-ui.outro). My PR works around it viaframeworkContextdirect mutation — but the right fix is wizard-level (have agent-runner push outroData viastore.setOutroData(...)). PR body has more detail.ethangui commentedon Jun 15, 2026
Shipped across two PRs, both merged today:
posthog-setup-report.md, with the model picking the run-relevant items.Final shape differs from the original proposal in two ways worth recording: the handoff is a single exit line rather than an embedded doc section, and the checklist lives in context-mill (so it updates on the docs cadence, not the wizard release cadence) rather than in wizard code. Both were review outcomes — see the PR threads for the reasoning.
Closing as done.