Repository navigation
[CI] (0e81558) vercel-ai/nextjs-support-chat - #4261
Closed
wizard-ci-bot[bot] wants to merge 2 commits into
Closed
wizard-ci-bot[bot] wants to merge 2 commits into
wizard-ci-bot[bot] wants to merge 2 commits into
Conversation
Author
|
Now I have enough context to evaluate this PR. Let me produce the full evaluation report. PR Evaluation ReportSummaryThis PR instruments a Next.js App Router support chat (Vercel AI SDK) for PostHog AI observability. It creates
Confidence score: 4/5 👍
File changes
App sanity check
|
| Criteria | Result | Description |
|---|---|---|
| App builds and runs | Yes | No syntax errors; dependencies should resolve |
| Preserves existing env vars & configs | Yes | No pre-existing PostHog; tool logic and lib/orders.ts untouched |
| No syntax or type errors | Yes | TypeScript appears valid |
| Correct imports/exports | Yes | PostHogSpanProcessor from @posthog/ai/otel, registerTelemetry from ai v5, OpenTelemetry from @ai-sdk/otel all valid |
| Minimal, focused changes | Yes | Only AI observability instrumentation added |
| Pre-existing issues | None |
Issues
- No
.env.examplefile: New environment variablesPOSTHOG_API_KEYandPOSTHOG_HOSTare required but not documented in a.env.examplefile. Only the report file mentions them. [MEDIUM]
Other completed criteria
.gitignorecorrectly excludes.env— no credentials committed- Dependency versions (
@posthog/ai ^8.7.0,@ai-sdk/otel ^1.0.0,@opentelemetry/sdk-node ^0.208.0) are plausible for the APIs used forceFlush()called per-request infinallyblock to prevent span loss in serverless/edge runtimes
PostHog implementation ✅
| Criteria | Result | Description |
|---|---|---|
| PostHog SDKs installed | Yes | @posthog/ai ^8.7.0, @ai-sdk/otel ^1.0.0, @opentelemetry/sdk-node, @opentelemetry/resources |
| PostHog client initialized | Yes | PostHogSpanProcessor created once at module scope in instrumentation.ts, NodeSDK started once |
| capture() | Yes | Generations captured via OTel spans + PostHogSpanProcessor; experimental_telemetry: { isEnabled: true } enables span emission |
| identify() | Yes | posthog.distinct_id set to request userId via experimental_telemetry.metadata per call |
| Error tracking | No | No error tracking configured (not in scope for AI observability wizard) |
| Reverse proxy | N/A | Server-side only |
Issues
- Dead
enrichSpancode:registerTelemetryincludes anenrichSpancallback that readsruntimeContext?.distinctIdandruntimeContext?.sessionId, but nothing sets these values. The function will always returnundefinedfor both fields. The actual identity is correctly carried byexperimental_telemetry.metadataon thegenerateTextcall. This dead code is harmless but misleading. [MEDIUM]
Other completed criteria
- Instrumentation initialized exactly once at module scope, not inside the request handler
- `` correctly keyed (not
session_idor `ai_session_id`) - `` set to
threadIdfrom request body — stable per conversation, correct cardinality - Identity (
posthog.distinct_id, ``) passed per-request viaexperimental_telemetry.metadata, NOT on OTel Resource — correct for a multi-user server - No hand-authored spans around
lookupOrder— tool span comes from Vercel AI SDK'sai.toolCallautomatically - No pre-existing PostHog initialization to preserve
- API key and host read from environment variables, not hardcoded in source
PostHog insights and events ✅
| Filename | PostHog events | Description |
|---|---|---|
app/api/chat/route.ts |
(×2), (tool) |
Each POST produces one OTel trace: first generateText generation, lookupOrder tool span, final generateText generation. Session grouped by threadId, person by userId. |
Issues
None.
Other completed criteria
- Events form the expected tree:
session(threadId) → trace(POST) → generation + span(lookupOrder) + generation - Person attribution uses real
userIdfrom request body — not hardcoded, not random - No fabricated structure — no synthetic sessions or manual spans where none are needed
- No secrets or API keys placed in event properties
- Prompt and completion content captured by default (expected for AI observability)
Reviewed by wizard workbench PR evaluator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated wizard CI run
Source: wizard-pr
Trigger ID:
0e81558App:
vercel-ai/nextjs-support-chatApp directory:
apps/vercel-ai/nextjs-support-chatWorkbench branch:
wizard-ci-0e81558-vercel-ai-nextjs-support-chatWizard branch:
release-please--branches--main--components--wizardContext Mill branch:
mainPostHog (MCP) branch:
masterTimestamp: 2026-09-29T08:46:17.099Z
Duration: 191.6s
YARA Scanner