Skip to content

[CI] (3f39e0e) manual-capture/node-http-chat - #4141

Closed
wizard-ci-bot[bot] wants to merge 1 commit into
mainfrom
wizard-ci-3f39e0e-manual-capture-node-http-chat
Closed

wizard-ci-bot[bot] wants to merge 1 commit into
mainfrom
wizard-ci-3f39e0e-manual-capture-node-http-chat

Conversation

@wizard-ci-bot

@wizard-ci-bot wizard-ci-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

Automated wizard CI run

Source: wizard-pr
Trigger ID: 3f39e0e
App: manual-capture/node-http-chat
App directory: apps/manual-capture/node-http-chat
Workbench branch: wizard-ci-3f39e0e-manual-capture-node-http-chat
Wizard branch: release-please--branches--main--components--wizard
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-09-23T21:38:10.188Z
Duration: 240.9s

YARA Scanner

✓ 31 tool calls scanned, 0 violations detected

No violations: ✓ 31 clean scans

@wizard-ci-bot

wizard-ci-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown
Author

Now I have all the context I need. Let me evaluate this PR thoroughly.


PR Evaluation Report

Summary

This PR adds manual and capture to a Node.js HTTP chat app that calls an OpenAI-compatible endpoint via plain fetch. The instrumentation correctly reuses the existing PostHog client from src/posthog.ts, adds proper and scoping, and captures tool executions as `` events with parent linkage. The changes are clean, minimal, and well-structured.

Files changed Lines added Lines removed
5 +123 -4

Confidence score: 5/5 🧙

  • No critical or medium issues found. The implementation precisely matches the README's expected structure and uses all correct property keys.

File changes

Filename Score Description
src/index.ts 5/5 Adds manual capture in `complete()`, for tool execution, AiContext type threading trace/session IDs, and toAiSessionId helper. Existing product-analytics code untouched.
.env.example 5/5 Documents POSTHOG_PROJECT_API_KEY and POSTHOG_HOST — matches what src/posthog.ts reads.
.gitignore 5/5 Ignores .env to prevent credential leaks.
.posthog-wizard-cache/.posthog-ai.json 5/5 Wizard cache metadata, correctly identifies provider and package.
posthog-ai-observability-report.md 5/5 Thorough report describing what was done and how to verify.

App sanity check ✅

Criteria Result Description
App builds and runs Yes tsc --noEmit passes; no new dependencies needed since posthog-node and node:crypto are already available.
Preserves existing env vars & configs Yes Existing PostHog client in src/posthog.ts is reused. identify(), chat_message_sent capture, and shutdown() are all untouched.
No syntax or type errors Yes TypeScript compiles cleanly. AiContext type is well-formed.
Correct imports/exports Yes randomUUID imported from node:crypto (built-in). All other imports unchanged.
Minimal, focused changes Yes Only adds AI observability instrumentation. Tool loop, message construction, and control flow are preserved.
Pre-existing issues None —

Other completed criteria

  • No credentials hardcoded; .env.example has empty placeholders
  • posthog-node ^4.2.0 already declared, no new dependency needed
  • Existing posthog.shutdown() preserved in both normal and error paths — flushes before exit

PostHog implementation ✅

Criteria Result Description
PostHog SDKs installed Yes posthog-node ^4.2.0 already in package.json, reused.
PostHog client initialized Yes Existing singleton in src/posthog.ts reused — no second client created.
capture() Yes Manual and events captured with correct property keys.
identify() N/A Pre-existing identify() call preserved, not part of this wizard's scope.
Error tracking N/A Not part of the AI observability wizard scope.
Reverse proxy N/A Server-only Node.js app.

Other completed criteria

  • Instrumentation setup is not inside a per-request handler but integrated into the complete() function which is the correct placement for manual capture
  • uses exactly the property key on every event (both generations and spans)
  • Session ID derived from Thread.threadId via toAiSessionId() — stable per thread, shared across traces
  • generated once per `ask()` call and shared across all `complete()` calls and events within a turn
  • correctly set on events, pointing to the trace ID
  • No hand-authored spans around arbitrary helper functions — only the actual lookup_order tool dispatch is captured as a span
  • Existing identify() and chat_message_sent captures are completely untouched

PostHog insights and events ✅

Filename PostHog events Description
src/index.ts `` Captured for every complete() call with model, provider, input/output messages, token counts, latency, HTTP status, tools
src/index.ts `` Captured for each lookup_order tool execution with trace linkage and ``
src/index.ts chat_message_sent (pre-existing) Unchanged product-analytics capture per turn

Expected event tree per run

Session: thread-thread_abc
├─ Trace 1 (ask "Where is my order?")
│  ├─  (complete with tool_calls)
│  ├─  (lookup_order)
│  └─  (final answer)
└─ Trace 2 (ask "Can I get a refund?")
   └─  (single completion)

This matches the README's expected structure exactly.

Other completed criteria

  • Events carry user_123 as distinctId — real user identifier from the app
  • events correctly set and ``
  • No fabricated structure — session, traces, and spans all correspond to real app concepts (thread, question/turn, tool call)
  • No credentials or secrets in event properties; prompts and completions are expected AI observability content

Reviewed by wizard workbench PR evaluator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants