Skip to content

[CI] (0e81558) vercel-ai/nextjs-support-chat - #4261

Closed
wizard-ci-bot[bot] wants to merge 2 commits into
mainfrom
wizard-ci-0e81558-vercel-ai-nextjs-support-chat
Closed

wizard-ci-bot[bot] wants to merge 2 commits into
mainfrom
wizard-ci-0e81558-vercel-ai-nextjs-support-chat

Conversation

@wizard-ci-bot

@wizard-ci-bot wizard-ci-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown

Automated wizard CI run

Source: wizard-pr
Trigger ID: 0e81558
App: vercel-ai/nextjs-support-chat
App directory: apps/vercel-ai/nextjs-support-chat
Workbench branch: wizard-ci-0e81558-vercel-ai-nextjs-support-chat
Wizard branch: release-please--branches--main--components--wizard
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-09-29T08:46:17.099Z
Duration: 191.6s

YARA Scanner

✓ 39 tool calls scanned, 0 violations detected

No violations: ✓ 39 clean scans

@wizard-ci-bot

wizard-ci-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Author

Now I have enough context to evaluate this PR. Let me produce the full evaluation report.


PR Evaluation Report

Summary

This PR instruments a Next.js App Router support chat (Vercel AI SDK) for PostHog AI observability. It creates instrumentation.ts with OTel setup via PostHogSpanProcessor, adds experimental_telemetry metadata to the generateText call with per-request userId and threadId, and adds required dependencies. The approach is correct for a Vercel AI SDK app.

Files changed Lines added Lines removed
6 +133 -14

Confidence score: 4/5 👍

  • Dead enrichSpan code in registerTelemetry: The enrichSpan callback references runtimeContext?.distinctId and runtimeContext?.sessionId, but no code in the app populates these runtime context values. This code never produces values. The per-call experimental_telemetry.metadata is what actually carries identity — so the instrumentation works, but there's dead code that suggests a misunderstanding. [MEDIUM]
  • No .env.example: The new environment variables POSTHOG_API_KEY and POSTHOG_HOST are not documented in a .env.example file. They are mentioned in the report file, but that's not standard documentation. [MEDIUM]

File changes

Filename Score Description
instrumentation.ts 4/5 Creates PostHogSpanProcessor and OTel NodeSDK at module scope, exports processor for flushing. Contains dead enrichSpan code.
app/api/chat/route.ts 5/5 Adds experimental_telemetry with per-request posthog.distinct_id and ``, wraps in try/finally for flush. Tool loop and LLM request shape unchanged.
package.json 5/5 Adds @posthog/ai, @ai-sdk/otel, @opentelemetry/sdk-node, @opentelemetry/resources
.gitignore 5/5 Excludes .env from version control
.posthog-wizard-cache/.posthog-ai.json 5/5 Wizard cache file
posthog-ai-observability-report.md 4/5 Accurate self-report, mentions env vars but no .env.example created

App sanity check ⚠️

Criteria Result Description
App builds and runs Yes No syntax errors; dependencies should resolve
Preserves existing env vars & configs Yes No pre-existing PostHog; tool logic and lib/orders.ts untouched
No syntax or type errors Yes TypeScript appears valid
Correct imports/exports Yes PostHogSpanProcessor from @posthog/ai/otel, registerTelemetry from ai v5, OpenTelemetry from @ai-sdk/otel all valid
Minimal, focused changes Yes Only AI observability instrumentation added
Pre-existing issues None

Issues

  • No .env.example file: New environment variables POSTHOG_API_KEY and POSTHOG_HOST are required but not documented in a .env.example file. Only the report file mentions them. [MEDIUM]

Other completed criteria

  • .gitignore correctly excludes .env — no credentials committed
  • Dependency versions (@posthog/ai ^8.7.0, @ai-sdk/otel ^1.0.0, @opentelemetry/sdk-node ^0.208.0) are plausible for the APIs used
  • forceFlush() called per-request in finally block to prevent span loss in serverless/edge runtimes

PostHog implementation ✅

Criteria Result Description
PostHog SDKs installed Yes @posthog/ai ^8.7.0, @ai-sdk/otel ^1.0.0, @opentelemetry/sdk-node, @opentelemetry/resources
PostHog client initialized Yes PostHogSpanProcessor created once at module scope in instrumentation.ts, NodeSDK started once
capture() Yes Generations captured via OTel spans + PostHogSpanProcessor; experimental_telemetry: { isEnabled: true } enables span emission
identify() Yes posthog.distinct_id set to request userId via experimental_telemetry.metadata per call
Error tracking No No error tracking configured (not in scope for AI observability wizard)
Reverse proxy N/A Server-side only

Issues

  • Dead enrichSpan code: registerTelemetry includes an enrichSpan callback that reads runtimeContext?.distinctId and runtimeContext?.sessionId, but nothing sets these values. The function will always return undefined for both fields. The actual identity is correctly carried by experimental_telemetry.metadata on the generateText call. This dead code is harmless but misleading. [MEDIUM]

Other completed criteria

  • Instrumentation initialized exactly once at module scope, not inside the request handler
  • `` correctly keyed (not session_id or `ai_session_id`)
  • `` set to threadId from request body — stable per conversation, correct cardinality
  • Identity (posthog.distinct_id, ``) passed per-request via experimental_telemetry.metadata, NOT on OTel Resource — correct for a multi-user server
  • No hand-authored spans around lookupOrder — tool span comes from Vercel AI SDK's ai.toolCall automatically
  • No pre-existing PostHog initialization to preserve
  • API key and host read from environment variables, not hardcoded in source

PostHog insights and events ✅

Filename PostHog events Description
app/api/chat/route.ts (×2), (tool) Each POST produces one OTel trace: first generateText generation, lookupOrder tool span, final generateText generation. Session grouped by threadId, person by userId.

Issues

None.

Other completed criteria

  • Events form the expected tree: session(threadId) → trace(POST) → generation + span(lookupOrder) + generation
  • Person attribution uses real userId from request body — not hardcoded, not random
  • No fabricated structure — no synthetic sessions or manual spans where none are needed
  • No secrets or API keys placed in event properties
  • Prompt and completion content captured by default (expected for AI observability)

Reviewed by wizard workbench PR evaluator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants