Repository navigation
ref(server): Split processor into envelope and bucket processors - #6622
Conversation
|
|
||
| use super::*; | ||
|
|
||
| async fn create_test_processor(config: Config) -> InnerProcessor { |
There was a problem hiding this comment.
This test helper was only used in this module, so I moved it from testutils so it can use the private InnerProcessor.
| } | ||
|
|
||
| #[cfg(feature = "processing")] | ||
| async fn rate_limit_buckets( |
There was a problem hiding this comment.
The bulk of the diff is functions being moved to either bucket or common.
| let (envelope_processor, envelope_processor_rx) = match current_config.relay_mode() { | ||
| RelayMode::Proxy => channel(ProxyProcessorService::name()), | ||
| RelayMode::Managed => channel(EnvelopeProcessorService::name()), | ||
| RelayMode::Managed => channel("envelope_processor"), |
There was a problem hiding this comment.
The name of this service queue would have changed anyway (from relay_server::services::processor::envelopeprocessorservice to something like relay_threading::service::asyncpoolservice::..), so I decided to keep it simple and just name it envelope_processor.
| AsyncPoolService::new( | ||
| Arc::new(InnerProcessor::new( |
There was a problem hiding this comment.
New: The generic AsyncPoolService wraps a private message handler and the async pool itself.
classDiagram
class Service <<trait>>
class MessageHandler<<trait>>
Service <|-- AsyncPoolService
AsyncPoolService *-- AsyncPool
AsyncPoolService *-- MessageHandler
MessageHandler <|-- EnvelopeProcessor
MessageHandler <|-- BucketProcessor
| if partition.is_empty() { | ||
| return; | ||
| } | ||
|
|
There was a problem hiding this comment.
encode_metrics_global insert retry can spin forever on oversized buckets
When http.global_metrics is on, a counter/gauge (or any bucket whose key alone exceeds max_flush_bytes) makes Partition::insert return the same bucket while send_global_partition no-ops on an empty partition, so the while-retry never progresses and pins a bucket-processor pool worker. Drop or force-split the oversized bucket on no-progress, matching BucketsView::by_size.
Evidence
encode_metrics_global(line 428+) retries viawhile let Some(bucket) = remaining.take()callingpartition.insertthensend_global_partitionwhen insert returns a remainder (lines 455–468); the comment already notes an endless loop if no request is produced.BucketView::splitreturnsMoveToNextBatchfor non-splittable counters/gauges larger than the budget, and for any bucket whose base size (name+tags) alone exceeds the budget (relay-metrics/src/view.rs), soPartition::insertadds nothing and returnsSome(same_bucket).send_global_partition(lines 390–393) returns immediately when the partition is empty and does not consume the remainder, soremainingis set back to the same bucket and the loop never terminates.- Sibling path
BucketsView::by_sizestops on no progress (if start == self.current { return None; }); tag count is uncapped invalidate_bucket(only per-key/value lengths), so a few-MB counter with many tags is within the 200 MiB envelope limit and exceeds defaultmax_flush_bytes(5 MB). - Reachable only when
config.http_global_metrics()is true (default false); hung tasks run onAsyncPoolServiceand permanently occupy pool workers.
Identified by Warden · wrdn-dos-review · PNJ-DZJ
There was a problem hiding this comment.
Seems legit, but this is a bug independent of this PR. Will fix in a follow-up.
Dav1dde
left a comment
There was a problem hiding this comment.
Trusting the refactor/moving of functions is correct.
I am not sure about whether relay-threading should expose a service, like this, but might as well try it out and see how it works.
It feels weird, but it makes sense dependency-wise: |
Split Envelope and Bucket processor to break cycles in the shutdown sequence. The two services still use the same async pool and the same redis rate limiter, so the split only separates their input queues.
The code is split across three new submodules of
processor:envelope,bucket, andcommon.This PR also introduces a reusable
AsyncPoolServicehelper for services that delegate their work to anAsyncPool.Old Flow
flowchart LR EnvelopeProcessor --MergeBucket--> Aggregator Aggregator --FlushBucket--> EnvelopeProcessorNew Flow
flowchart LR EnvelopeProcessor --MergeBucket--> Aggregator Aggregator --FlushBucket--> BucketProcessorref: RELAY-47