Repository navigation
[CI] (9c26590) openai/python-docs-rag - #4484
Draft
wizard-ci-bot[bot] wants to merge 1 commit into
Draft
wizard-ci-bot[bot] wants to merge 1 commit into
wizard-ci-bot[bot] wants to merge 1 commit into
Conversation
Author
|
Now I have everything I need to evaluate this PR. PR Evaluation ReportSummaryThis PR instruments an existing OpenAI-based RAG docs Q&A CLI script with PostHog AI observability using the
Confidence score: 5/5 🧙
File changes
App sanity check ✅
Issues(none)
|
| Criteria | Result | Description |
|---|---|---|
| PostHog SDKs installed | Yes | posthog added to pyproject.toml dependencies |
| PostHog client initialized | Yes | Posthog() with os.environ["POSTHOG_API_KEY"] and os.environ["POSTHOG_HOST"], enable_exception_autocapture=True |
| capture() | Yes | Generations captured via wrapper — posthog.ai.openai.OpenAI intercepts embeddings.create and responses.create |
| identify() | N/A | Server-only CLI script; distinct_id passed per-call via posthog_distinct_id |
| Error tracking | Yes | enable_exception_autocapture=True configured |
| Reverse proxy | N/A | Server-only script |
Issues
(none)
Other completed criteria
- Instrumentation initialized once at module scope, not inside request handlers
posthog.ai.openai.OpenAIwrapper correctly replaces directopenai.OpenAI— generations will be captured as/- Shared
posthog_trace_idinanswer()groups the embedding and generation into one trace - `` used with correct key spelling, shared across all calls via module-level
AI_SESSION_ID - Session id generated once per process (
f"rag-{uuid.uuid4()}"at module scope), correct cardinality for a CLI posthog_distinct_id=USER_IDon every call provides person attribution using the app's ownuser_123- No hand-authored spans —
retrieve()is correctly left uninstrumented (it's not a tool) - No pre-existing PostHog code to preserve — additive-only criterion satisfied trivially
- Identity scope correct: CLI is one-user-per-process, and distinct_id is per-call on the wrapper path
PostHog insights and events ✅
| Filename | PostHog events | Description |
|---|---|---|
main.py |
(×6), (×2) |
4 corpus embeddings at startup (own traces), 2 answer turns each with 1 query embedding + 1 generation (shared trace). All grouped under one ``. |
Issues
(none)
Other completed criteria
- Events form the tree structure from the README: 1 session → 2 answer traces (embedding + generation each) + startup embedding traces
- Person attribution via
USER_ID = "user_123"— a real app identifier, not a random UUID - No spans captured, which is correct — no tools registered
- No fabricated structure — session exists because the app has a conversation concept (the Q&A run)
- No secrets in event properties; prompts/completions are expected AI observability content
Reviewed by wizard workbench PR evaluator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated wizard CI run
Source: scheduled
Trigger ID:
9c26590App:
openai/python-docs-ragApp directory:
apps/openai/python-docs-ragWorkbench branch:
wizard-ci-9c26590-openai-python-docs-ragWizard branch:
mainContext Mill branch:
mainPostHog (MCP) branch:
masterTimestamp: 2026-10-14T16:22:31.090Z
Duration: 129.5s
YARA Scanner