Skip to content

[CI] (69eeefc) anthropic/node-weather - #4079

Closed
wizard-ci-bot[bot] wants to merge 1 commit into
mainfrom
wizard-ci-69eeefc-anthropic-node-weather
Closed

wizard-ci-bot[bot] wants to merge 1 commit into
mainfrom
wizard-ci-69eeefc-anthropic-node-weather

Conversation

@wizard-ci-bot

@wizard-ci-bot wizard-ci-bot Bot commented Sep 22, 2026

Copy link
Copy Markdown

Automated wizard CI run

Source: wizard-pr
Trigger ID: 69eeefc
App: anthropic/node-weather
App directory: apps/anthropic/node-weather
Workbench branch: wizard-ci-69eeefc-anthropic-node-weather
Wizard branch: posthog/functional-a1-min
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-09-22T15:56:31.601Z
Duration: 281.4s

YARA Scanner

✓ 47 tool calls scanned, 0 violations detected

No violations: ✓ 47 clean scans

@wizard-ci-bot

wizard-ci-bot Bot commented Sep 22, 2026

Copy link
Copy Markdown
Author

Now I have all the information needed. Let me produce the evaluation report.


PR Evaluation Report

Summary

This PR instruments a two-turn Anthropic weather assistant CLI with PostHog AI observability using the @posthog/ai Anthropic wrapper. It correctly sets up the wrapper client, shares a posthogTraceId per turn, uses the exact key with the app's `threadId`, and flushes before exit. However, it adds a hand-authored around the get_weather tool call, which the README explicitly lists as a failure condition.

Files changed Lines added Lines removed
5 +87 -9

Confidence score: 3/5 🤔

  • **Hand-authored around `getWeather`**: The README explicitly lists "a hand-authored span around `getWeather`" as a failure condition. The PR manually captures for the tool call instead of relying on the wrapper's instrumentation. [CRITICAL]
  • Anthropic SDK version mismatch: @posthog/ai declares a peer dependency of @anthropic-ai/sdk >=0.112.3 <0.126.0, but the project pins ^0.32.0. The PR acknowledges this incompatibility and used --legacy-peer-deps to install. This makes the wrapper's runtime behavior unreliable. [MEDIUM]
  • No .env.example: New environment variables POSTHOG_API_KEY and POSTHOG_HOST are not documented in a .env.example file. They are mentioned in the report file, but no template exists for new developers. [MEDIUM]

File changes

Filename Score Description
src/index.ts 3/5 Adds PostHog wrapper, per-call identity, shared trace ID, session ID, and shutdown — but also adds a hand-authored `` that the README flags as wrong
package.json 3/5 Adds @posthog/ai and posthog-node but Anthropic SDK peer dep mismatch
.gitignore 5/5 Properly gitignores .env
.posthog-wizard-cache/.posthog-ai.json 5/5 Wizard cache metadata
posthog-ai-observability-report.md 4/5 Thorough report, honestly documents the version mismatch

App sanity check ⚠️

Criteria Result Description
App builds and runs Yes tsc --noEmit passes per report
Preserves existing env vars & configs Yes Conditional client falls back to plain Anthropic when no PostHog key
No syntax or type errors Yes TypeScript compiles without errors
Correct imports/exports Yes @posthog/ai/anthropic and posthog-node are valid modules
Minimal, focused changes No Hand-authored `` adds unnecessary manual instrumentation into the tool dispatch path
Pre-existing issues None Base app is clean

Issues

  • Anthropic SDK version mismatch: @posthog/ai requires @anthropic-ai/sdk >=0.112.3 but the project has ^0.32.0. Used --legacy-peer-deps to force install. Runtime behavior is unverified and officially unsupported. [MEDIUM]
  • No .env.example: POSTHOG_API_KEY and POSTHOG_HOST are only documented in the report file, not in a .env.example template. [MEDIUM]
  • Conditional throw in non-production: The error throw when POSTHOG_API_KEY is missing only fires when NODE_ENV !== 'production', meaning production silently falls back to no observability — an unusual pattern but not strictly wrong. [LOW]

Other completed criteria

  • PostHog client initialized once at module scope with conditional fallback
  • posthog?.shutdown() properly called in .finally() before exit
  • weather.ts and tool definitions are untouched
  • .env is gitignored; no credentials in committed source
  • Type cast (client as PostHogAnthropic) is safe because PostHogAnthropic extends Anthropic

PostHog implementation ⚠️

Criteria Result Description
PostHog SDKs installed Yes @posthog/ai: ^8.13.1, posthog-node: ^5.52.5
PostHog client initialized Yes new PostHog(key, { host }) at module scope, then passed to PostHogAnthropic
capture() Yes Wrapper auto-captures generations; manual `` capture for tool
identify() N/A CLI app; identity via per-call posthogDistinctId
Error tracking Yes enableExceptionAutocapture: true on PostHog client
Reverse proxy N/A Server-side CLI app

Issues

  • Hand-authored ``: The PR manually captures posthog?.capture({ event: '', ... }) around the `getWeather` tool call. The README explicitly states "Fail: a hand-authored span around `getWeather`". The `@posthog/ai` wrapper should handle tool-related structure; hand-authoring it either duplicates or fabricates what the wrapper manages. [CRITICAL]

Other completed criteria

  • Instrumentation initialized once at module scope, not inside request handler
  • Generations captured via PostHogAnthropic wrapper on messages.create
  • Shared posthogTraceId per ask() groups both LLM calls in one trace
  • `` spelled exactly right, uses this.threadId (`thread_abc`)
  • Session cardinality correct: one threadId shared across all turns
  • Identity scope correct: per-call posthogDistinctId with this.userId
  • No pre-existing PostHog to conflict with; additive-only
  • posthog?.shutdown() flushes before CLI exit

PostHog insights and events ⚠️

Filename PostHog events Description
src/index.ts (via wrapper), (manual) Two generations per turn from wrapper, one hand-authored tool span per turn. Two turns total → 4 generations + 2 spans + session thread_abc

Issues

  • Potential duplicate tool spans: If the @posthog/ai Anthropic wrapper automatically tracks tool calls from tool_use/tool_result message pairs, the hand-authored `` would create duplicate span events per turn. [MEDIUM]
  • **missing on span**: The manually captured sets but omits. While there may not be a clear parent span, the rubric expects it when manually capturing spans. [LOW]

Other completed criteria

  • Events form the correct tree: session → trace → (generation, span, generation) per turn
  • Person attribution uses user_123 from Conversation.userId
  • No PII or secrets in event properties
  • and contain tool I/O (location, weather result) — appropriate for observability
  • `` correctly measured around tool execution

Reviewed by wizard workbench PR evaluator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants