Skip to content

[CI] (3f39e0e) openai/node-weather - #4144

Closed
wizard-ci-bot[bot] wants to merge 2 commits into
mainfrom
wizard-ci-3f39e0e-openai-node-weather
Closed

wizard-ci-bot[bot] wants to merge 2 commits into
mainfrom
wizard-ci-3f39e0e-openai-node-weather

Conversation

@wizard-ci-bot

@wizard-ci-bot wizard-ci-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

Automated wizard CI run

Source: wizard-pr
Trigger ID: 3f39e0e
App: openai/node-weather
App directory: apps/openai/node-weather
Workbench branch: wizard-ci-3f39e0e-openai-node-weather
Wizard branch: release-please--branches--main--components--wizard
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-09-23T21:39:17.081Z
Duration: 307.4s

YARA Scanner

✓ 54 tool calls scanned, 0 violations detected

No violations: ✓ 54 clean scans

@wizard-ci-bot

wizard-ci-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown
Author

Good — weather.ts has no diff, confirming it was untouched. Now I have everything needed to evaluate.

Let me trace through the key rubric items carefully:

README expected outcome:

  • One session, one trace: generation → span(getWeather) → generation, all on user_123
  • @posthog/ai OpenAI wrapper with per-call posthogDistinctId/posthogProperties, shared posthogTraceId
  • `` spelled exactly; flushed before exit
  • weather.ts and tool loop untouched
  • Fail conditions: hand-authored span around getWeather; a trace per model call; no flush

Now, the README says "Fail: a hand-authored span around getWeather". The PR does add a hand-authored `` capture around getWeather (lines 67-79). However, the README's trace structure also shows `getWeather` as a span. Let me re-read this more carefully...

The README says the expected tree includes getWeather ← span. But it also says "Fail: a hand-authored span around getWeather". This means the span should come from the wrapper/instrumentor automatically, NOT be hand-authored via posthog.capture({ event: '', ... }).

Wait — looking at this more carefully: the @posthog/ai wrapper captures generations automatically, but it does NOT automatically capture tool/span events. The tool span has to come from somewhere. The app registers a tool with OpenAI, and when the model calls it, the @posthog/ai wrapper might emit a span for the tool call. Let me think about this...

Actually, looking at the @posthog/ai wrapper: it wraps chat.completions.create to capture events. It does NOT automatically create events for tool calls — those are between the two LLM calls and happen in user code. So the README expects a span but says hand-authoring one is a fail condition. This seems contradictory...

But re-reading: "Fail: a hand-authored span around getWeather" — the key phrase is "around getWeather". The @posthog/ai wrapper, when it sees tool_calls in the response, should emit tool spans automatically as part of the generation tracking. But actually, I need to check: does @posthog/ai capture tool spans? Looking at the PostHog docs provided, the wrapper path captures generations. Tool spans are mentioned as coming from "the app's own tool registration; the instrumentor emits it."

Given the evaluation rubric says: "Spans should come from the app's registered tools, or from explicit `` capture for a genuinely non-LLM step." And the ph_no_hand_authored_spans criterion says: "The PR does not wrap ordinary helper functions in hand-authored spans to fabricate structure."

The getWeather IS a tool call that the model requests — it's a genuinely non-LLM step that IS registered as a tool. The capture here documents the tool execution with proper trace linking. This seems like a legitimate use case for explicit capture. But the README explicitly says "Fail: a hand-authored span around getWeather".

I'll follow the README which is more specific. The README says hand-authored span around getWeather is a failure condition. This is a critical issue.

Wait, let me re-read the system prompt: "Read the app's README.md. These test apps document an 'Expected wizard outcome' list — the app-specific rubric — plus the session/trace/span structure the app is shaped to produce. Where that list is more specific than the items below, prefer it."

So the README says hand-authored span = fail. But... the tree structure in the README also shows a span for getWeather. How should the span be produced then? Through the @posthog/ai wrapper automatically? Let me think...

Actually, looking at how @posthog/ai works with OpenAI: the wrapper captures generations. When a tool call happens, the wrapper would need to know about the tool execution to emit a span. But the tool execution happens in user code between two create() calls. The wrapper can't automatically capture that.

So the expected structure shows a span, but the README says hand-authoring it is a fail. This is the trap — the README expects the instrumentor to handle it, but the @posthog/ai wrapper doesn't auto-emit tool spans. The wizard should NOT add hand-authored spans. The span in the tree is what would appear if the instrumentor handled it (which it doesn't for this path). The correct behavior is: just use the wrapper for generations, don't hand-author spans.

So: ph_no_hand_authored_spans = NO (the PR adds a hand-authored ``).

Now let me finalize the evaluation:


PR Evaluation Report

Summary

This PR instruments an OpenAI Node.js weather assistant CLI with PostHog AI observability using the @posthog/ai OpenAI wrapper. It replaces the direct OpenAI import with @posthog/ai/openai, initializes a PostHog client from environment variables, adds per-call posthogDistinctId/posthogTraceId/posthogProperties, and flushes before exit. It also adds a hand-authored `` capture around the getWeather tool call.

Files changed Lines added Lines removed
5 +105 -8

Confidence score: 4/5 👍

  • **Hand-authored around `getWeather`**: The README explicitly lists "a hand-authored span around `getWeather`" as a fail condition. The PR manually captures a event with posthog.capture() wrapping the getWeather call (lines 67-79 of src/index.ts). This fabricates structure that should come from the instrumentor, not user code. [CRITICAL]
  • missing: The hand-authored span sets but omits, so it won't parent correctly to the first generation in the tree. [MEDIUM]

File changes

Filename Score Description
src/index.ts 3/5 Swaps OpenAI client for @posthog/ai wrapper, adds per-call identity/tracing, but includes a hand-authored `` the README forbids
package.json 5/5 Adds @posthog/ai and posthog-node dependencies correctly
.gitignore 5/5 Adds .env to gitignore
.posthog-wizard-cache/.posthog-ai.json 5/5 Wizard cache metadata
posthog-ai-observability-report.md 4/5 Clear report, but describes the hand-authored span as intentional

App sanity check ⚠️

Criteria Result Description
App builds and runs Yes TypeScript compiles; dependencies resolve
Preserves existing env vars & configs Yes No pre-existing PostHog client to conflict with
No syntax or type errors Yes TypeScript-safe casts and correct API usage
Correct imports/exports Yes @posthog/ai/openai and posthog-node imports are valid
Minimal, focused changes No Hand-authored `` capture adds ~15 lines of manual event construction that shouldn't be there
Pre-existing issues None Base app is clean

Issues

  • Hand-authored `` adds unnecessary code: The README explicitly says hand-authoring a span around getWeather is a fail condition. The 13 lines of `posthog.capture({ event: '', ... })` code should be removed entirely. The `@posthog/ai` wrapper handles generation capture; the tool execution between calls does not need manual span instrumentation. [CRITICAL]

Other completed criteria

  • .env is gitignored and credentials read from environment variables
  • weather.ts is completely untouched (confirmed via git diff)
  • Tool dispatch loop structure preserved — same messages array construction, same tool schema, same control flow
  • Dependencies @posthog/ai and posthog-node added with reasonable version ranges
  • posthog.shutdown() correctly called in finally block before CLI exit

PostHog implementation ⚠️

Criteria Result Description
PostHog SDKs installed Yes @posthog/ai ^7.21.0 and posthog-node ^5.53.0
PostHog client initialized Yes new PostHog() at module scope with env vars
capture() Yes Wrapper auto-captures generations; manual `` also present
identify() N/A Server-side CLI; posthogDistinctId used per-call instead
Error tracking Yes enableExceptionAutocapture: true set on PostHog client
Reverse proxy N/A Server-side CLI app

Issues

  • Hand-authored `` violates README spec: The README lists "Fail: a hand-authored span around getWeather". The PR wraps `getWeather` in a manual `posthog.capture({ event: '' })` call. This is exactly the failure case the app was designed to test. [CRITICAL]
  • missing: Even if the span were appropriate, it lacks `` to link it to the preceding generation, breaking proper tree rendering in PostHog's trace view. [MEDIUM]

Other completed criteria

  • Instrumentation initialized once at module scope, not per-request
  • Both chat.completions.create calls share the same posthogTraceId — one trace per ask() turn ✓
  • `` is spelled with the correct $ prefix and exact key name ✓
  • AI_SESSION_ID is a single randomUUID() generated once per CLI process — correct cardinality ✓
  • Identity scope is correct: CLI process = one user, so module-level USER_ID passed per-call is fine ✓
  • posthog.shutdown() in finally block ensures flush before exit ✓
  • POSTHOG_API_KEY and POSTHOG_HOST read from environment, not hardcoded ✓
  • Existing app had no prior PostHog integration, so additive-only is satisfied ✓

PostHog insights and events ⚠️

Filename PostHog events Description
src/index.ts (×2, auto-captured by wrapper), (×1, hand-authored) Two generations from the wrapper with shared trace ID and session ID; one manual span for getWeather tool execution

Issues

  • Tree structure partially correct but includes fabricated span: The two events correctly share one trace and one session, which is good. But the is hand-authored (the README's explicit fail case) and lacks ``, so it would float as an orphan within the trace rather than nesting under the first generation. [CRITICAL]

Other completed criteria

  • Person attribution uses USER_ID = 'user_123' from the app's existing constant — a real identifier, not random ✓
  • No credentials or secrets placed in event properties ✓
  • Session ID uses correct `` property key ✓
  • LLM prompts/completions are captured (privacyMode: false) as expected ✓

Reviewed by wizard workbench PR evaluator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants