Repository navigation
Closed
Description
Issue 9: Telemetry + experiment instrumentation
Epic: Task-queue orchestrator runner · Depends on: #624 (emits during
execution), finalize after #626 and #627
Why
The dark launch is judged primarily on responsiveness: a short time to first
visible progress, and steady incremental progress. Per-task latency and model are
the headline metrics, and tokens are a secondary cost to watch. We need clean
per-task and per-run telemetry, segmentable from the linear baseline by VARIANT.
Scope / deliverable
- Variant tagging.
VARIANT: 'orchestrator'flows into existing events via
headers (buildAgentEnv,agent-interface.ts:371) once Issue 1: Shared bootstrap extraction + variant gating #621 lands. Also
analytics.setTag('variant', 'orchestrator')sosessionProperties
(analytics.ts:16) and thesetup wizard finishedshutdown event (:166)
carry it. The existing per-taskagent startedandagent completedevents
become the A/B spine. - Orchestrator events via
analytics.wizardCapture:orchestrator seeded { task_count, types }orchestrator task started|completed|failed { type, model, attempts, duration_ms }orchestrator task enqueued { type, enqueued_by, depth, dynamic }orchestrator guard tripped { guard, type }, emitted from Issue 4: Orchestrator MCP tools (inwizard-tools) #623's guardsorchestrator run finished { tasks_total, tasks_done, tasks_failed, total_duration_ms }. Aresumedflag is added by Issue 10: Resume across runs/crashes (deferred, low priority) #629.
- Responsiveness metrics, the headline. Capture:
- time to first task started, from launch to the first task reaching
in_progress, the key "no long silent gap" signal, - per-task
duration_ms, and the gap between consecutive task starts, so no
single step dominates wall-clock, - per-task
model, to confirm cheap models carry the cheap work.
Keep each task'sagent completedandduration_ms(agent-interface.ts:862).
- time to first task started, from launch to the first task reaching
- Per-task token capture, secondary. Surface token usage per task. Summing
across tasks versus the linear baseline's singleagent completedis a cost to
watch, not the pass/fail metric. - Run-end remark once. Wire the Stop-hook
requestRemarkflag from Issue 5: Executor framework + fresh per-task agent #624 so the
remark (WIZARD_REMARK_EVENT_NAME,:859) fires once at run end, not per task.
Key files
src/lib/programs/orchestrator/orchestrator-runner.ts(emit run and task events)src/lib/programs/orchestrator/queue-tools.ts(emit enqueue and guard events)src/lib/agent/agent-interface.ts(remark gating, per-task token capture)src/utils/analytics.ts(setTag('variant', ...))
Acceptance criteria
- A run emits
orchestrator seeded, then per-taskstartedandcompleted,
thenorchestrator run finished, all carryingVARIANT=orchestrator. - Baseline and orchestrator runs are distinguishable in PostHog by
VARIANT. - Time to first task started, per-task
duration_ms, gap between starts, and
per-taskmodelare captured, the responsiveness headline. - Per-task token usage is captured as a secondary cost to watch.
- The remark fires once per run.
Metadata
Metadata
Assignees
Labels
No labels
Activity
feat(orchestrator): run telemetry — the responsiveness A/B spine
gewenyu99 commentedon Jul 11, 2026
Shipped via #619 and hardened by #714/#725/#726/#727/#831 — orchestrator telemetry events + run-metrics.ts (time_to_first_task/completion, per-task started/completed/failed/skipped).