Skip to content

[CI] (624abe2) vercel-ai/nextjs-support-chat - #4475

Closed
wizard-ci-bot[bot] wants to merge 2 commits into
mainfrom
wizard-ci-624abe2-vercel-ai-nextjs-support-chat
Closed

wizard-ci-bot[bot] wants to merge 2 commits into
mainfrom
wizard-ci-624abe2-vercel-ai-nextjs-support-chat

Conversation

@wizard-ci-bot

@wizard-ci-bot wizard-ci-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

Automated wizard CI run

Source: wizard-pr
Trigger ID: 624abe2
App: vercel-ai/nextjs-support-chat
App directory: apps/vercel-ai/nextjs-support-chat
Workbench branch: wizard-ci-624abe2-vercel-ai-nextjs-support-chat
Wizard branch: release-please--branches--main--components--wizard
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-10-08T09:34:00.364Z
Duration: 301.5s

YARA Scanner

✓ 53 tool calls scanned, 0 violations detected

No violations: ✓ 53 clean scans

@wizard-ci-bot

wizard-ci-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown
Author

Now I have everything I need to evaluate this PR. Let me produce the full evaluation.


PR Evaluation Report

Summary

This PR instruments a Next.js App Router support chat (Vercel AI SDK) for PostHog AI observability. It creates an instrumentation.ts file that initializes an OTel NodeSDK with PostHogSpanProcessor, adds experimental_telemetry to the generateText call with per-request `` (threadId) and posthog.distinct_id (userId), and flushes spans after each request. The approach precisely matches the README's expected outcome.

Files changed Lines added Lines removed
7 +104 -0

Confidence score: 5/5 🧙

  • Unused @ai-sdk/otel dependency: The package is declared in package.json but never imported in any source file — dead dependency. [LOW]

File changes

Filename Score Description
instrumentation.ts 5/5 Creates OTel NodeSDK with PostHogSpanProcessor at module scope; env var validation; exports span processor for per-request flush
app/api/chat/route.ts 5/5 Adds experimental_telemetry with per-request session/identity metadata and post-call flush; tool loop and message construction untouched
package.json 4/5 Correct dependencies added (@posthog/ai, @opentelemetry/sdk-node, @opentelemetry/resources), but @ai-sdk/otel is declared and never used
.env.example 5/5 Documents POSTHOG_PROJECT_TOKEN and POSTHOG_HOST without values
.gitignore 5/5 Ignores .env to prevent credential leaks

App sanity check ✅

Criteria Result Description
App builds and runs Yes Report confirms npm run typecheck and npm run build both pass
Preserves existing env vars & configs Yes No pre-existing PostHog client; OpenAI credential untouched; lib/orders.ts unmodified
No syntax or type errors Yes Build and typecheck pass
Correct imports/exports Yes PostHogSpanProcessor from @posthog/ai/otel, NodeSDK from @opentelemetry/sdk-node, resourceFromAttributes from @opentelemetry/resources all resolve correctly
Minimal, focused changes Yes Only AI observability instrumentation added; tool loop and LLM request shape completely preserved
Pre-existing issues None Clean base app

Issues

  • Dead @ai-sdk/otel dependency: Declared in package.json but never imported in any source file. Harmless but unnecessary bloat. Remove from dependencies. [LOW]

Other completed criteria

  • Environment variables read from process.env, not hardcoded
  • .env.example documents all new variables
  • lib/orders.ts (tool logic) completely untouched
  • No restructuring of the tool-dispatch loop or LLM request shape

PostHog implementation ✅

Criteria Result Description
PostHog SDKs installed Yes @posthog/ai ^8.7.0, @opentelemetry/sdk-node, @opentelemetry/resources
PostHog client initialized Yes PostHogSpanProcessor initialized once at module scope in instrumentation.ts, fed into NodeSDK
capture() Yes Vercel AI SDK telemetry (experimental_telemetry: { isEnabled: true }) emits OTel spans processed by PostHogSpanProcessor → `` events
identify() N/A Server-only; identity passed per-request via posthog.distinct_id in telemetry metadata
Error tracking No Not added (not in scope for AI observability wizard)
Reverse proxy N/A Server-only app

Issues

No critical or medium issues.

Other completed criteria

  • OTel SDK initialized once at module scope, not inside request handler
  • PostHogSpanProcessor correctly configured with projectToken and host from env
  • forceFlush() called per-request after generateText to ensure spans are exported before response
  • No provider instrumentor (Vercel AI SDK emits OTel natively with experimental_telemetry)
  • Per-request identity (posthog.distinct_id: userId) in telemetry metadata, NOT on OTel Resource — correct for a multi-user server
  • Per-request session (: threadId) in telemetry metadata, NOT on OTel Resource
  • `` uses the correct literal key (not session_id or `ai_session_id`)
  • Session cardinality is correct: threadId from the request body, shared across requests with the same thread
  • No hand-authored spans around lookupOrder — Vercel AI SDK auto-traces tool calls
  • No pre-existing PostHog to disrupt

PostHog insights and events ✅

Filename PostHog events Description
app/api/chat/route.ts (×2), (lookupOrder tool) Each POST produces one trace containing two generations (tool_use + final) and one tool span, attributed to userId with threadId as session

Issues

No critical or medium issues.

Other completed criteria

  • Events form the expected tree: session(threadId) → trace(POST) → generation + span(lookupOrder) + generation
  • Person attribution uses userId from the request body — a real user identifier, not a placeholder
  • No fabricated structure: session maps to threadId which is a genuine conversation identifier in the app
  • No credentials or secrets in event properties
  • LLM input/output captured by default via Vercel AI SDK telemetry (expected behavior)

Reviewed by wizard workbench PR evaluator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants