Skip to content

ref(server): Split processor into envelope and bucket processors - #6622

Merged
jjbayer merged 5 commits into
masterfrom
ref/bucket-processor
Nov 3, 2026
Merged

jjbayer merged 5 commits into
masterfrom
ref/bucket-processor

Conversation

@jjbayer

@jjbayer jjbayer commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Split Envelope and Bucket processor to break cycles in the shutdown sequence. The two services still use the same async pool and the same redis rate limiter, so the split only separates their input queues.

The code is split across three new submodules of processor: envelope, bucket, and common.

This PR also introduces a reusable AsyncPoolService helper for services that delegate their work to an AsyncPool.

Old Flow

flowchart LR
    EnvelopeProcessor --MergeBucket-->  Aggregator
    Aggregator --FlushBucket--> EnvelopeProcessor
Loading

New Flow

flowchart LR
    EnvelopeProcessor --MergeBucket-->  Aggregator
    Aggregator --FlushBucket--> BucketProcessor
Loading

ref: RELAY-47

jjbayer and others added 3 commits November 3, 2026 09:49
Co-authored-by: Codex <noreply@openai.com>
@linear-code

linear-code Bot commented Nov 3, 2026

Copy link
Copy Markdown

RELAY-47


use super::*;

async fn create_test_processor(config: Config) -> InnerProcessor {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test helper was only used in this module, so I moved it from testutils so it can use the private InnerProcessor.

}

#[cfg(feature = "processing")]
async fn rate_limit_buckets(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bulk of the diff is functions being moved to either bucket or common.

let (envelope_processor, envelope_processor_rx) = match current_config.relay_mode() {
RelayMode::Proxy => channel(ProxyProcessorService::name()),
RelayMode::Managed => channel(EnvelopeProcessorService::name()),
RelayMode::Managed => channel("envelope_processor"),

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name of this service queue would have changed anyway (from relay_server::services::processor::envelopeprocessorservice to something like relay_threading::service::asyncpoolservice::..), so I decided to keep it simple and just name it envelope_processor.

Comment on lines +82 to +83
AsyncPoolService::new(
Arc::new(InnerProcessor::new(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New: The generic AsyncPoolService wraps a private message handler and the async pool itself.

classDiagram

    class Service <<trait>>
    class MessageHandler<<trait>> 
    
    Service <|--  AsyncPoolService
    AsyncPoolService *--  AsyncPool
    AsyncPoolService *--  MessageHandler
    MessageHandler <|-- EnvelopeProcessor
    MessageHandler <|-- BucketProcessor
Loading

Comment on lines +390 to +393
if partition.is_empty() {
return;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encode_metrics_global insert retry can spin forever on oversized buckets

When http.global_metrics is on, a counter/gauge (or any bucket whose key alone exceeds max_flush_bytes) makes Partition::insert return the same bucket while send_global_partition no-ops on an empty partition, so the while-retry never progresses and pins a bucket-processor pool worker. Drop or force-split the oversized bucket on no-progress, matching BucketsView::by_size.

Evidence
  • encode_metrics_global (line 428+) retries via while let Some(bucket) = remaining.take() calling partition.insert then send_global_partition when insert returns a remainder (lines 455–468); the comment already notes an endless loop if no request is produced.
  • BucketView::split returns MoveToNextBatch for non-splittable counters/gauges larger than the budget, and for any bucket whose base size (name+tags) alone exceeds the budget (relay-metrics/src/view.rs), so Partition::insert adds nothing and returns Some(same_bucket).
  • send_global_partition (lines 390–393) returns immediately when the partition is empty and does not consume the remainder, so remaining is set back to the same bucket and the loop never terminates.
  • Sibling path BucketsView::by_size stops on no progress (if start == self.current { return None; }); tag count is uncapped in validate_bucket (only per-key/value lengths), so a few-MB counter with many tags is within the 200 MiB envelope limit and exceeds default max_flush_bytes (5 MB).
  • Reachable only when config.http_global_metrics() is true (default false); hung tasks run on AsyncPoolService and permanently occupy pool workers.

Identified by Warden · wrdn-dos-review · PNJ-DZJ

@jjbayer jjbayer Nov 3, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems legit, but this is a bug independent of this PR. Will fix in a follow-up.

@jjbayer
jjbayer marked this pull request as ready for review November 3, 2026 12:41
@jjbayer
jjbayer requested a review from a team as a code owner November 3, 2026 12:41

@Dav1dde Dav1dde left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trusting the refactor/moving of functions is correct.

I am not sure about whether relay-threading should expose a service, like this, but might as well try it out and see how it works.

@jjbayer

jjbayer commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

I am not sure about whether relay-threading should expose a service

It feels weird, but it makes sense dependency-wise: relay-threading already depends on relay-system, so it can expose utilities that rely on both crates. But I'm happy to move the utility to relay-server instead, which would also be a good fit.

@jjbayer
jjbayer added this pull request to the merge queue Nov 3, 2026
Merged via the queue into master with commit 233adec Nov 3, 2026
36 checks passed
@jjbayer
jjbayer deleted the ref/bucket-processor branch November 3, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants