Skip to content

Report anonymous turn_finished and device_paired events - #5318

Open
brsbl wants to merge 5 commits into
bb/telemetry-automation-split-view-thr_gqshddt6w2from
bb/turn-and-device-telemetry-thr_gqshddt6w2
Open

brsbl wants to merge 5 commits into
bb/telemetry-automation-split-view-thr_gqshddt6w2from
bb/turn-and-device-telemetry-thr_gqshddt6w2

Conversation

@brsbl

@brsbl brsbl commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Human comments

What was wrong

bb couldn't see whether an agent turn finished or broke, so a new install whose first run failed looked the same as one whose user simply left. It also couldn't see whether people added a phone or another machine. No usage event covered either one.

What changed

turn_finished (server-side, apps/server/src/services/system/turn-telemetry.ts)

  • When it fires: whenever a thread's turn ends. It's reported from applyLoggedThreadLifecycleEvent, beside the existing turn.failed announcement, so every applied run.succeeded, run.failed, or stop.settled reports exactly once. That includes provisioning failures, host disconnects, daemon restarts, and server-settled stops. A fork seed settling from starting is not a turn and isn't counted. The report is deferred past the transaction.
  • outcome: completed, failed, or stopped.
  • provider: a provider id declared in a bundled plugin's manifest (experimental_providers). Any custom ACP agent or third-party provider is reported as other, so private agent names never leave the server. Core holds no provider literals.
  • error_category: a fixed enum. It comes from the latest system error since the turn started (process_exited, start_failed, host_lost), otherwise from the provider's own error classification via the same lookup turn.failed uses. It never contains error text.
  • first_turn: claimed once through an app-settings marker (claimFirstFinishedTurn). Installs that already had more than one top-level thread before the marker existed never claim it.

device_paired (added to the client telemetry allow-list)

  • Server: reported when a new machine enrolls with a join code (internal/hosts.ts). The server's own machine and re-enrollments of an existing host don't count.
  • Push Notifications plugin: reports mobile_ios or mobile_android when a new device registers.
  • Properties: only the device kind and first_of_kind. No labels, push tokens, hostnames, or ids.
  • Browser sign-ins through bb connect are not covered. That pairing happens in the hosted service and the local server never sees it.

Both events

  • They respect the usage-data opt-out.
  • The privacy page, docs/configuration.md, and the api_to_audit.md entry describe them. The Push Notifications overview is already at its 4,000-character marketplace limit, so it isn't changed.
  • Plugin SDK 0.6.41. Push Notifications now requires bbPluginSdk >=0.6.41.

Stack: this PR sits on top of #5316. device_paired from the plugin needs #5222's telemetry route, and the docs extend the same paragraph.

How you verified

Privacy page telemetry paragraph (Chrome for Testing, local apps/web, 2×):

Before (#5316 head) After (this PR)
Privacy telemetry paragraph before Privacy telemetry paragraph after
  • Server tests:
    • a completed root turn reports once, as the first turn;
    • a provider process exit reports failed / process_exited and hides a custom provider id as other;
    • the first-turn marker is claimed once, and never on an install that already had several threads;
    • the category and provider mapping.
  • Host enrollment test: the server's own loopback enrollment reports nothing, and two join-code machines report first and not-first.
  • Lifecycle tests: a failure from any path reports failed, a settled stop reports stopped, and a fork seed settling reports nothing.
  • Push Notifications test: a new phone reports mobile_ios, first; a second one reports not-first; re-registering the same token reports nothing.
  • Slop Cop review at 65fbe55 found two P1s, both fixed in ab9e72c (the CI provider-literal ratchet also required 6da510a):
    • a fresh install reported its own machine as paired;
    • server-settled turn ends were missed.
  • P2 follow-ups left out of this PR:
    • apply the same bundled-provider rule to thread_created and user_message_sent, which still send raw provider ids;
    • make machine server-only in the schema;
    • base first_turn on prior turn history for upgraded single-thread installs;
    • report managed (provider-created) machines and count only persistent hosts;
    • compute the push first-of-kind count inside the subscription store's mutation queue, since token rotation counts as a new phone;
    • read only errorInfo instead of building the full turn.failed payload.
  • CI runs the suites.

BB-Thread: Retention measurement

🤖 Generated with Claude Code

AGENT GENERATED

turn_finished is captured on the server when a root agent turn ends: on
turn completion, on a provider process exit, and when a turn's start
command fails. Each reports the outcome, a built-in provider id or
"other", a coarse error category mapped from the provider's error
classification (never error text), and whether this was the install's
first finished turn. A one-time app-settings marker claims the first turn,
and installs that already had several top-level threads never claim it.

device_paired is added to the client telemetry allow-list. The server
captures it when a new machine enrolls (not its own machine, not a
re-enrollment), and the Push Notifications plugin reports it when a new
iOS or Android device registers. Each carries only the device kind and
whether it is the first of that kind.

Both respect the usage-data opt-out. Privacy copy, configuration docs,
the API audit entry, and the Push Notifications overview describe them.
Plugin SDK 0.6.41.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@brsbl
brsbl added this pull request to stack #5232 October 10, 2026 03:34
brsbl and others added 4 commits October 9, 2026 20:47
…ines

Every turn end now reports from applyLoggedThreadLifecycleEvent beside
the turn.failed announcement, deferred past the transaction. That covers
server-settled stops, host disconnects, daemon restarts, and provisioning
failures that never pass through event effects, and a fork seed settling
from starting is not counted. The failure category comes from the latest
system error since the turn started, else the provider's classification.
The first-turn marker is claimed once per process instead of on every turn.

device_paired for machines now fires only for join-code enrollments, so a
fresh install no longer reports its own machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The overview was already at the 4,000-character marketplace limit, so the
usage-data note lives only in the privacy page and configuration docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Core no longer lists provider ids. turn_finished reports a provider id
only when a bundled plugin's manifest declares it, so custom ACP agents
and third-party providers still report as other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
enrollHost returned only the host id, so the route could not tell a
join-code enrollment from the server's own loopback one and never
reported machines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant