Skip to content

fix(metrics): Skip oversized buckets when batching - #6629

Open
jjbayer wants to merge 3 commits into
masterfrom
fix/oversized-metric-buckets
Open

jjbayer wants to merge 3 commits into
masterfrom
fix/oversized-metric-buckets

Conversation

@jjbayer

@jjbayer jjbayer commented Nov 3, 2026

Copy link
Copy Markdown
Member

Continue batching after buckets that cannot fit into an empty batch, and stop retrying those buckets in global metrics forwarding.

This was flagged by warden here: #6622 (comment)

Base automatically changed from ref/bucket-processor to master November 3, 2026 13:27
Continue batching after buckets that cannot fit into an empty batch, and stop retrying those buckets in global metrics forwarding.

Co-authored-by: Codex <noreply@openai.com>
@jjbayer
jjbayer force-pushed the fix/oversized-metric-buckets branch from 4ba3599 to 6692437 Compare November 3, 2026 13:36
@jjbayer
jjbayer marked this pull request as ready for review November 3, 2026 14:37
@jjbayer
jjbayer requested a review from a team as a code owner November 3, 2026 14:37
Comment thread relay-metrics/src/view.rs

/// Iterator which slices the source view into segments with an approximate size of `size_in_bytes`.
///
/// Buckets that cannot fit into an empty segment, even after splitting, are skipped.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So practically this can only happen if the batch size is really small?

@jjbayer jjbayer Nov 4, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was thinking instead of skipping whether we should just include the minimum possible size of the bucket. That silent data loss kinda irks me.

An easier alternative is also to internally clamp the minimum size to a threshold which never runs into that issue.

}

#[tokio::test]
async fn test_encode_metrics_global_drops_oversized_buckets() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's one hell of a test.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah sorry, should have screened it before opening up the PR for review. Will remove.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants