Skip to content

ref(span-buffer): Optimize fan out to only target possible candidates#118152

Merged
cmanallen merged 3 commits into
masterfrom
cmanallen/span-buffer-optimize-zrem-fan-out
Jun 22, 2026
Merged

ref(span-buffer): Optimize fan out to only target possible candidates#118152
cmanallen merged 3 commits into
masterfrom
cmanallen/span-buffer-optimize-zrem-fan-out

Conversation

@cmanallen

@cmanallen cmanallen commented Jun 22, 2026

Copy link
Copy Markdown
Member

The only item added to the queue is a segment_key.

zadd_items = queue_adds.setdefault(queue_key, {})
zadd_items[result.segment_key] = new_deadline

A span can be in the queue if that span was once a segment root. A span that always knew its parent never becomes a segment_key so zrem-ing it does nothing. Given our max (and near average) batch size is 2000 we're zrem-ing <2000 keys that have never existed in the queue. In add-buffer.lua we know who the former segment roots were. From #117804: "Presence of child_ic implies that this span is a root span". Let's return those merged root spans and remove them from the queue.

Updating the queue takes 17ms (9us per span) on average. Removing 2000 no-ops will probably have a large impact. In local profiling it was about a microsecond per span (2ms per batch). In production this number could vary.

@cmanallen cmanallen requested review from a team as code owners June 22, 2026 15:20
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jun 22, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ebf072f. Configure here.

Comment thread src/sentry/spans/buffer_store.py
@cmanallen cmanallen merged commit 478d020 into master Jun 22, 2026
68 checks passed
@cmanallen cmanallen deleted the cmanallen/span-buffer-optimize-zrem-fan-out branch June 22, 2026 16:05
sehr-m pushed a commit that referenced this pull request Jun 23, 2026
…#118152)

The only item added to the queue is a segment_key. 

```python
zadd_items = queue_adds.setdefault(queue_key, {})
zadd_items[result.segment_key] = new_deadline
```

A span can be in the queue if that span was once a segment root. A span
that always knew its parent never becomes a segment_key so zrem-ing it
does nothing. Given our max (and near average) batch size is 2000 we're
zrem-ing <2000 keys that have never existed in the queue. In
`add-buffer.lua` we know who the former segment roots were. From
#117804: "Presence of child_ic implies that this span is a root span".
Let's return those merged root spans and remove them from the queue.

Updating the queue takes 17ms (9us per span) on average. Removing 2000
no-ops will probably have a large impact. In local profiling it was
about a microsecond per span (2ms per batch). In production this number
could vary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants