Skip to content

[CI] (624abe2) manual-capture/node-http-chat - #4458

Closed
wizard-ci-bot[bot] wants to merge 1 commit into
mainfrom
wizard-ci-624abe2-manual-capture-node-http-chat
Closed

wizard-ci-bot[bot] wants to merge 1 commit into
mainfrom
wizard-ci-624abe2-manual-capture-node-http-chat

Conversation

@wizard-ci-bot

@wizard-ci-bot wizard-ci-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

Automated wizard CI run

Source: wizard-pr
Trigger ID: 624abe2
App: manual-capture/node-http-chat
App directory: apps/manual-capture/node-http-chat
Workbench branch: wizard-ci-624abe2-manual-capture-node-http-chat
Wizard branch: release-please--branches--main--components--wizard
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-10-08T09:12:46.818Z
Duration: 169.4s

YARA Scanner

✓ 29 tool calls scanned, 0 violations detected

No violations: ✓ 29 clean scans

@wizard-ci-bot

wizard-ci-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown
Author

PR Evaluation Report

Summary

This PR adds manual AI observability instrumentation to a plain-fetch Node.js chat app that uses Ollama. It captures events for each LLM call and events for tool executions, linking them into a correct session → trace → generation/span tree using the existing PostHog client.

Files changed Lines added Lines removed
4 +110 -4

Confidence score: 5/5 🧙

No critical or medium issues found. The instrumentation is clean, correct, and minimal.


File changes

Filename Score Description
src/index.ts 5/5 Adds capture inside `complete()`, capture for tool dispatch, and wires trace/session/parent IDs through an AiContext parameter. Existing code (identify, chat_message_sent, shutdown) is untouched.
.gitignore 5/5 Adds .env to prevent credential leaks.
.posthog-wizard-cache/.posthog-ai.json 5/5 Wizard metadata file.
posthog-ai-observability-report.md 5/5 Accurate self-report describing the captured tree and privacy notes.

App sanity check ✅

Criteria Result Description
App builds and runs Yes Report confirms tsc --noEmit succeeds; no syntax errors visible
Preserves existing env vars & configs Yes Existing POSTHOG_PROJECT_API_KEY and POSTHOG_HOST from src/posthog.ts are reused; no new env vars needed
No syntax or type errors Yes Clean TypeScript: new types (AiContext, extended Completion) are well-formed
Correct imports/exports Yes randomUUID from node:crypto is correct; posthog already imported
Minimal, focused changes Yes Only AI observability instrumentation added; no scope creep
Pre-existing issues None —

Other completed criteria

  • Existing PostHog client from src/posthog.ts reused — no second instance created
  • Existing identify(), chat_message_sent capture, and posthog.shutdown() are fully preserved
  • No credentials hardcoded; .env added to .gitignore
  • complete() function signature extended but fetch call body and control flow are unchanged

PostHog implementation ✅

Criteria Result Description
PostHog SDKs installed Yes posthog-node already installed; no new dependencies needed
PostHog client initialized Yes Reuses the existing singleton from src/posthog.ts
capture() Yes Manual and captures with correct event names and property keys
identify() N/A Pre-existing identify() call preserved; not in scope for AI observability wizard
Error tracking N/A Not in scope for AI observability wizard
Reverse proxy N/A Server-only Node.js app

Other completed criteria

  • Instrumentation is inline within the existing functions (not initialized per-request)
  • uses the exact key (not a variant)
  • Session ID is this.threadId (thread_abc), shared across all traces in the conversation
  • Trace ID (randomUUID()) created once per ask() call and shared across all events in that turn
  • Parent chain: first generation's parent is the trace root; tool span's parent is the first generation; final generation's parent is the last tool span
  • posthog.shutdown() preserved in both success and error paths

PostHog insights and events ✅

Filename PostHog events Description
src/index.ts `` Captured per complete() call with model, provider, input messages, output choices, token counts, latency, and tools
src/index.ts `` Captured per tool execution (lookup_order) with input/output state, latency, and parent linkage
src/index.ts chat_message_sent (pre-existing) Existing product-analytics event preserved unchanged

Other completed criteria

  • Expected tree structure verified by tracing the code:
    • Turn 1: generation (parent=traceId) → span/lookup_order (parent=gen1.spanId) → generation (parent=span.spanId) — all sharing one traceId
    • Turn 2: generation (parent=traceId2) — standalone generation with its own traceId
    • Both turns share : 'thread_abc' and distinctId: 'user_123'
  • No fabricated structure — spans correspond to actual tool dispatch
  • Privacy: prompts and completions captured as / (expected and correct)
  • No secrets in event properties

Reviewed by wizard workbench PR evaluator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants