From 564084afbbdf0c25cb61d2585c0240ef4a769400 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Wed, 8 Jul 2026 22:27:59 -0700 Subject: [PATCH 1/4] Add proposals for client-identity rate limiting (write and query paths) Cortex's ingestion and query rate limits are enforced at the tenant level only. When multiple independent services or teams share a tenant, one noisy or misbehaving client can exhaust the whole tenant's budget for everyone else, and there is no way to distinguish who, within a tenant, is responsible. Adds two companion design proposals: - client-identity-rate-limiting.md: write-path (distributor) enforcement, keyed on a new trusted X-User-ID header, following the same trust model Cortex already uses for X-Scope-OrgID. - client-identity-query-rate-limiting.md: the read-path (query-frontend) follow-on, reusing the same header and identity approach. Both are opt-in, additive to existing limits, and covered as experimental features on the proposed rollout plan. Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 242 ++++++++++++++ .../client-identity-rate-limiting.md | 315 ++++++++++++++++++ 2 files changed, 557 insertions(+) create mode 100644 docs/proposals/client-identity-query-rate-limiting.md create mode 100644 docs/proposals/client-identity-rate-limiting.md diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md new file mode 100644 index 0000000000..249f35375a --- /dev/null +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -0,0 +1,242 @@ +--- +title: "Client-Identity Query Rate Limiting" +linkTitle: "Client-Identity Query Rate Limiting" +weight: 1 +slug: client-identity-query-rate-limiting +--- + +- Author: Krunal Jain +- Date: July 2026 +- Status: Proposed + +## Background + +This is a follow-on to +[Client-Identity Rate Limiting](./client-identity-rate-limiting.md) (write path), applying the same +idea to queries. + +Cortex has no per-request-rate limiter on the read path today. What exists is a different kind of +control: a per-tenant limit on how many requests can be *queued at once*, and a family of per-query +cost limits that bound how *expensive* a single query can be. Neither bounds *how many queries per +second* a tenant's traffic issues, and, same as the write path, neither can distinguish one query +source from another within a shared tenant. + +The query-frontend already reads and logs a couple of per-request identifying headers today, purely +for observability (Grafana's dashboard and panel identifiers, included in slow-query log lines). +These headers are not used for enforcement today, only logging, and this proposal does not use them +for identity either; it uses a new, generic header instead (see Proposed Design). + +A single noisy dashboard, a runaway alert rule evaluating too frequently, or one team's ad hoc +Explore usage can all generate a query load that consumes a shared tenant's query-processing +capacity (queriers, store-gateways) at the expense of every other user/dashboard under that same +`X-Scope-OrgID`, mirroring exactly the ingestion-side problem the write-path proposal addresses, +just on reads. + +## Problem + +Provide a query-rate-limiting dimension *below* the tenant, at the query-frontend, so that one noisy +query source cannot degrade query performance for every other user sharing a tenant. + +Requirements (mirroring the write-path proposal): + +- **Opt-in and backward compatible.** Deployments that don't set the identity header, or don't + enable this feature, see no behavior change. +- **Per-tenant configurable**, following the same default-plus-per-tenant-override pattern used + throughout Cortex. +- **Presence-gated, not mandatory.** A request without the identity header is not rejected; it is + simply not subject to the additional per-identity check, and counts only toward whatever + tenant-level behavior already exists today (which, notably, currently has no *rate* limit to fall + back to on the read path, see Interaction with Existing Limits). +- **Same generic, gateway-set identity header as the write-path proposal (`X-User-ID`), not a + per-client-application-specific one.** Consistent with the write-path proposal's principle: + Cortex trusts one opaque identity value, set by whatever sits in front of it, the same way it + already trusts `X-Scope-OrgID`. It does not special-case any particular client application. + +## Out of Scope + +- The write (ingestion) path: covered by the companion proposal, + [Client-Identity Rate Limiting](./client-identity-rate-limiting.md). The two share a design + philosophy (trusted, gateway-forwarded identity header, opt-in, additive to existing limits) and, + as of this revision, the *same* identity header. They are kept as separate proposals because they + have different enforcement points (distributor vs. query-frontend), not because of any difference + in identity source. +- Per-query cost limiting: unaffected and orthogonal; this proposal bounds *frequency*, those bound + *cost per query*. +- Any new authentication mechanism, and any client-application-specific identity header. Exactly + like the write-path proposal, this reuses a header trusted the same way `X-Scope-OrgID` already + is; it does not add token validation, a new identity protocol, or special-cased handling for any + particular upstream client. +- Enforcement in the querier itself (as opposed to the query-frontend/scheduler). The query-frontend + is the natural analog to the distributor here: it is the first internal component every query + passes through, exactly as the distributor is for writes. + +## Proposed Design + +### Identity extraction + +The same generic, trusted header as the write-path proposal, `X-User-ID`, read once when a query +request enters the query-frontend, alongside the existing Grafana dashboard/panel header reads. +Using the exact same header and identity-extraction approach as the write-path proposal means a +deployment that sets `X-User-ID` once, at its gateway, for all Cortex-bound traffic gets consistent +per-client identity across both ingestion and queries with no additional configuration: one +identity mechanism, reused at both enforcement points, rather than two different ones. + +### Configuration + +New limits, following the same pattern as the write-path proposal: + +```yaml +# Default (flags), applied to all tenants without an override +frontend: + client_identity_query_rate_limit: 0 # 0 = disabled (default; no behavior change) + client_identity_query_burst_size: 0 + # Same bounded-tracking cap as the write-path proposal, sized independently since the + # frontend and distributor are separate processes with separate memory budgets. + client_identity_query_tracked_clients_limit: 10000 + +# Per-tenant override via runtime config +overrides: + tenant-123: + client_identity_query_rate_limit: 50 + client_identity_query_burst_size: 100 +``` + +Units are queries per second (matching how Cortex's existing outstanding-requests limit is already +a count, not a byte/sample rate; this is a request-frequency limit, deliberately independent of +query cost). + +### Enforcement + +Same two-gate structure as the write-path proposal: + +1. The tenant has a non-zero `client_identity_query_rate_limit` configured. +2. The request carries a non-empty `X-User-ID` value. + +``` +Query request arrives at query-frontend + │ + v +┌───────────────────────────┐ no (either gate) ┌───────────────┐ +│ X-User-ID present AND │──────────────────────>│ Continue │ +│ tenant limit configured? │ │ (queue/forward) │ +└───────────────────────────┘ └───────────────┘ + │ yes + v +┌────────────────────────────┐ fail ┌──────────────────┐ +│ Per-(tenant, client) rate │────────────────>│ 429 Too Many │ +│ limit check (new) │ │ Requests │ +└────────────────────────────┘ └──────────────────┘ + │ pass + v +Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) +``` + +A new rate limiter instance, reusing the same local rate-limiting mechanism as the write-path +proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses +the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection +under this feature is observable the same way existing query-frontend rejections already are. + +The rate-limiter strategy backing it mirrors the write-path proposal's approach exactly: it splits +the combined tenant/client key and looks up the per-tenant configured limit, the same per-tenant +limits interface pattern already used elsewhere for query-side limits. + +### Where this runs relative to the frontend/scheduler split + +Cortex's query-frontend can run with or without a separate query-scheduler. The query-frontend's +request handler runs in the query-frontend process in both configurations: it is the entrypoint +before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather +than inside the scheduler's own queueing logic, covers both deployment topologies with one check and +avoids needing per-scheduler-replica state coordination for something that is already a +per-frontend-replica local rate limit, the same "local" semantics the write-path proposal's default +strategy already has. + +### Bounding memory: capped tracking, not an unbounded map + +Same concern as the write-path proposal, and the same fix: the number of distinct tenant+client +keys here is bounded only by how many distinct `X-User-ID` values are presented to the +query-frontend, so tracking is capped with least-recently-used eviction rather than kept in a plain +unbounded map, exactly as described in the write-path proposal's equivalent section. The two caps +are configured independently (see Configuration above) since the query-frontend and distributor are +separate processes with independent memory budgets, but the mechanism and rationale are identical. + +### Local enforcement precision, and interaction with the results cache + +The query-frontend, like the distributor, typically runs multiple replicas behind a load balancer, +so the same local-enforcement precision caveat raised in the write-path proposal applies here too: +one client's query traffic is a smaller, lumpier stream than a tenant's aggregate traffic, so +which replica a given request lands on matters more than it does for tenant-wide limits. The same +resolution applies: ship local-only first, treat a global variant as a near-term follow-up based on +operational experience, not indefinitely deferred work. + +A second, read-path-specific question: the query-frontend serves some requests entirely from its +results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check +before knowing whether a request will be a cache hit means a client issuing frequent but cheap, +repeated, cacheable queries (for example, an auto-refreshing dashboard hitting the same query +window) could be throttled even though those requests would have cost negligible querier/store +gateway capacity. Given this feature's stated goal is protecting shared query-processing capacity, +not query volume for its own sake, this is worth resolving before this feature is considered +complete: either check cache-eligibility first and only rate-limit the cache-miss path, or accept +the imprecision in a first version and document it clearly as a known limitation. This proposal +does not resolve that choice; it is called out explicitly as an open question below rather than +silently deferred. + +### Metrics + +- Reuses the query-frontend's existing rejected-query counter with a new discard reason for this + limit; no new metric family needed, consistent with how other query-frontend rejections are + already tracked. +- Per-client rejection visibility is obtained from query-frontend logs rather than a dedicated + metric label, the same way dashboard/panel identifiers already are, and for the same reason as + the write-path proposal: adding a client label to an existing metric that already carries several + labels would multiply its cardinality by however many distinct clients are tracked. + +### Interaction with existing limits + +- **Orthogonal to per-query cost limits**: those bound how expensive one query is; this bounds how + many queries per second one client issues. +- **Orthogonal to the existing per-tenant outstanding-requests limit**: that bounds queue depth (how + many requests can be waiting at once); this bounds arrival rate. A client could stay under the + queue depth limit while still issuing queries fast enough to starve other queue occupants of + querier time; this closes that gap. +- **Consistent with the write-path proposal's philosophy**: purely additive, off by default, + identical two-gate opt-in structure, same underlying rate-limiting mechanism. + +## Rollout Plan + +Same phased approach as the write-path proposal: introduced as an **experimental** feature, local +enforcement only, disabled by default (`client_identity_query_rate_limit: 0`). A global variant and +graduation out of experimental status follow once operational experience from Phase 1 shows how +much the local-only imprecision, and the cache-interaction question raised above, matter in +practice. + +## Alternatives Considered + +- **Enforce in the scheduler instead of the frontend.** Would require either scheduler-side rate + limiting to be per-scheduler-replica (fine, but then it needs its own identical mechanism since + the scheduler is a separate component from the frontend) or shared/global state across scheduler + replicas, a much larger change mirroring the local-vs-global split the write-path proposal already + has to consider. Enforcing once in the frontend, before a request is ever queued or forwarded, is + simpler and sufficient for a first version; a global variant could be considered later exactly as + it was for the write path. +- **A single combined proposal covering both ingestion and query paths.** Considered and explicitly + rejected; see the write-path proposal's cross-reference to this one. Different enforcement + points and independent review/merge risk argue for two focused proposals over one broad one, even + though both now share the same identity mechanism. + +## Open Questions + +- Should this feature ship together with the write-path proposal, or fully independently, given they + touch different components and teams (distributor vs. query-frontend) might review them + separately? Leaning towards shipping independently, sharing only the small identity-extraction + logic, given the review benefits of keeping them as separate changes discussed above. +- Should the per-client query limit be expressible as a percentage of the tenant's overall query + capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports + fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow + refinement rather than blocking the initial absolute-number implementation. +- Should the per-client check run before or after results-cache eligibility is known (see "Local + enforcement precision, and interaction with the results cache" above)? Checking after would avoid + throttling cheap, cache-served requests, at the cost of being a slightly more invasive change to + the request handling order; checking before (as currently proposed) is simpler but risks + throttling based on request *count* rather than actual resource consumption. This should be + resolved before the feature is considered ready to implement, not left as a known limitation by + default. diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md new file mode 100644 index 0000000000..0885dfbf95 --- /dev/null +++ b/docs/proposals/client-identity-rate-limiting.md @@ -0,0 +1,315 @@ +--- +title: "Client-Identity Rate Limiting" +linkTitle: "Client-Identity Rate Limiting" +weight: 1 +slug: client-identity-rate-limiting +--- + +- Author: Krunal Jain +- Date: July 2026 +- Status: Proposed + +## Background + +Cortex enforces ingestion rate limits at the tenant level only. Every request is identified solely +by the `X-Scope-OrgID` header value, and the distributor's rate limiter applies a single shared +budget, local or global, across everything sent under that tenant. + +In practice, a single tenant is very often not a single writer. Multiple independent services, +teams, or clusters frequently share one `X-Scope-OrgID` (splitting tenants further has real +operational cost: more ring shards, more per-tenant limits to tune, more override entries to +maintain). When that happens, today's rate limiter cannot distinguish a well-behaved writer from a +noisy or misbehaving one sharing the same tenant. A single client scaling up unexpectedly (bad +config, retry storm, new deployment) can exhaust the *entire tenant's* ingestion budget and start +throttling every other legitimate writer sharing that org ID, even though none of the throttled +traffic was the cause. + +Cortex's trust model already has precedent for exactly this shape of problem at the tenant level: +`X-Scope-OrgID` itself is a plain, unauthenticated-by-Cortex header. Cortex trusts it entirely +because it assumes something in front of it (a reverse proxy, an auth gateway such as the one +described in the [Authentication Gateway](./auth-gateway.md) proposal) authenticates the caller and +sets the header correctly, and that untrusted clients can never reach Cortex directly. There is no +equivalent mechanism today for identifying *who, within a tenant,* sent a given write. + +## Problem + +Provide a second, optional rate-limiting dimension *below* the tenant: throttle by client identity, +where identity is an opaque string supplied by the same trusted gateway/proxy layer that already +sets `X-Scope-OrgID`, so that one noisy client cannot exhaust a shared tenant's ingestion budget for +everyone else. + +Requirements: + +- **Opt-in and backward compatible.** Deployments that don't set the identity header, or don't + enable this feature, see no behavior change; the tenant-level limiter continues to work exactly + as it does today. +- **Per-tenant configurable**, following the same default-plus-per-tenant-override pattern as every + other Cortex limit. +- **Consistent trust model with the rest of Cortex.** The identity header is trusted the same way + `X-Scope-OrgID` already is: because it arrives from a trusted network path, not because Cortex + independently verifies it. This proposal does not attempt to add a stronger guarantee than + Cortex's existing multi-tenancy model provides; it reuses the same assumption rather than + introducing a new one. +- **Presence-gated, not mandatory.** A request without the identity header is not rejected or + treated as suspicious; it simply isn't subject to the additional per-identity check, and its + usage counts only toward the tenant's existing aggregate budget, unchanged from today. + +## Out of Scope + +- Any form of authentication Cortex does not already support (JWT validation, OAuth, SSO, mTLS + client-cert identity). An earlier draft of this proposal considered deriving identity from a + verified TLS client certificate; that was dropped because it only works when Cortex itself + terminates client TLS, which excludes the common case of a service mesh (Istio, Linkerd) or + gateway terminating mTLS in front of Cortex, a real gap given Cortex's own case studies document + exactly this kind of mesh-fronted deployment. Reusing the existing `X-Scope-OrgID` trust model + avoids that gap entirely. +- Cross-tenant limiting or identity: this is purely a sub-division of the existing per-tenant + ingestion rate limit. +- Read-path (query) rate limiting by client identity is covered by a **separate, companion + proposal**, [Client-Identity Query Rate Limiting](./client-identity-query-rate-limiting.md), kept + independent of this one because it has a different enforcement point (query-frontend, not + distributor); it reuses the same `X-User-ID` header and identity-extraction approach introduced + here. Filed alongside this proposal rather than folded into it, so each can be reviewed and + merged on its own timeline. +- Defining *how* an operator's gateway derives the identity value (API key ID, service account, + mesh workload identity, job name, etc.). Exactly like Cortex never defines what a "tenant" is + organizationally, this proposal treats the identity as an opaque string the gateway is trusted to + set consistently, which is out of scope for Cortex itself to prescribe. + +## Proposed Design + +### Identity extraction + +Add a new trusted header, `X-User-ID`, read once when a write request enters Cortex. This mirrors +the existing `X-Scope-OrgID` header handling: like `X-Scope-OrgID`, `X-User-ID` is trusted as-is, +because Cortex assumes it is set by a trusted gateway/proxy in front of it, not supplied directly by +an untrusted caller. If the header is absent, the request is unaffected by anything in this +proposal. + +Cortex's HTTP library dependency already defines an `X-Scope-UserID` header, distinct from the +`X-User-ID` proposed here, but it is unused anywhere in Cortex today and its original intent isn't +documented. Reusing it was considered and rejected: adopting an existing-but-dormant header with no +clear record of its intended semantics risks silently changing behavior for any deployment that +happens to already send that header for an unrelated reason, whereas a new, clearly-scoped header +name carries no such risk. The similarity in naming is coincidental and worth calling out explicitly +so it doesn't read as an oversight during review. + +The extracted identity is threaded through the request the same way source IPs are today (already +carried from the HTTP entrypoint through to the distributor for logging purposes), so it survives +the handoff into the write path without changing any internal request/response shapes. + +### Configuration + +New limits, following the same default-plus-per-tenant-override pattern as every other Cortex +limit: + +```yaml +# Default (flags), applied to all tenants without an override +distributor: + client_identity_ingestion_rate_limit: 0 # 0 = disabled (default; no behavior change) + client_identity_ingestion_burst_size: 0 + # Global cap (not per-tenant) on how many distinct tracked tenant+client entries the + # per-client rate limiter keeps in memory at once; oldest-used entries are evicted once + # this is reached. See "Bounding memory" below. + client_identity_ingestion_tracked_clients_limit: 10000 + +# Per-tenant override via runtime config +overrides: + tenant-123: + client_identity_ingestion_rate_limit: 5000 + client_identity_ingestion_burst_size: 10000 +``` + +`client_identity_ingestion_rate_limit: 0` is the default and means "disabled", the same convention +Cortex already uses elsewhere for a limit that should only take effect once explicitly configured. +This guarantees zero behavior change for any tenant that doesn't explicitly opt in. + +### Enforcement + +Enforcement is gated on **two independent conditions**, both of which must hold: + +1. The tenant has a non-zero `client_identity_ingestion_rate_limit` configured. +2. The request carries a non-empty `X-User-ID` value. + +If either is false, the request is subject only to the existing tenant-level ingestion rate check, +exactly as today: a missing header is not an error, and a tenant that hasn't opted in never pays +any cost for this feature. + +``` +Push request + │ + v +┌─────────────────────┐ fail ┌──────────────────┐ +│ Tenant IngestionRate │────────────────>│ 429 Too Many │ +│ check (existing) │ │ Requests │ +└─────────────────────┘ └──────────────────┘ + │ pass + v +┌───────────────────────────┐ no (either gate) ┌───────────────┐ +│ X-User-ID present AND │──────────────────────>│ Continue to │ +│ tenant limit configured? │ │ ingestion │ +└───────────────────────────┘ └───────────────┘ + │ yes + v +┌────────────────────────────┐ fail ┌──────────────────┐ +│ Per-(tenant, client) rate │────────────────>│ 429 Too Many │ +│ limit check (new) │ │ Requests │ +└────────────────────────────┘ └──────────────────┘ + │ pass + v +Continue to ingestion +``` + +A second rate limiter instance, reusing Cortex's existing local rate-limiting mechanism unmodified, +is added to the distributor, keyed by tenant and client identity together rather than by tenant +alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing +tenant-level rate limit; this only prevents one client from consuming the entire budget. + +The per-tenant limit and burst are looked up the same way every other tenant limit is: the +enforcement point resolves the tenant from the combined key and reads the configured value from +that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other +Cortex limit; only the token bucket enforcing it is split per client. + +### Local enforcement is less precise here than at the tenant level + +Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces +its share of the limit independently) and a "global" mode (the limit is divided evenly across +healthy distributors, trading a little coordination overhead for much tighter enforcement). +"Local" is imprecise by design: which distributor a given request lands on is effectively random, +so a client's traffic isn't evenly spread across replicas in the short term, and a burst can land +disproportionately on one distributor. + +That imprecision matters less at the tenant level, where traffic volume is usually large enough +relative to distributor count for the law of large numbers to smooth it out. At the per-client +granularity this proposal adds, that assumption is weaker: one client's traffic is a smaller, +lumpier stream, so local-only enforcement is more likely to either erroneously throttle a +well-behaved client (bad luck concentrates its requests on an already-busy distributor) or let a +misbehaving client through for longer than intended (its requests happen to spread across +distributors, each individually staying under its local share). This is a real precision gap in a +local-only first version, not merely a style choice inherited from the tenant-level limiter. + +Given that, this proposal's initial version still ships local-only, matching how Cortex introduces +most new limits, but a global variant (dividing the per-client limit across healthy distributors, +mirroring the existing global tenant-level strategy) should be treated as a near-term follow-up +once operational experience shows how much the imprecision matters in practice, not as indefinitely +deferred future work. + +### Bounding memory: capped tracking, not an unbounded map + +Unlike the existing tenant-level limiter, whose per-tenant map stays small because the number of +tenants is small and operator-controlled, the number of distinct tenant+client keys here is +bounded only by how many distinct `X-User-ID` values a gateway ever sends. Left as a plain +unbounded map, this becomes both a slow memory leak under normal churn (clients renamed, rotated, +or retired over time never get cleaned up) and, combined with the header trust boundary above, a +denial-of-service vector: a caller able to set arbitrary `X-User-ID` values could grow that map +without limit simply by rotating identities. + +Cortex already solves an analogous problem elsewhere: the tenant-federation regex resolver bounds +a similar per-key cache with a fixed-size, least-recently-used eviction cache rather than a plain +map, sized by an operator-configurable limit. This proposal adopts the same approach: the per-client +rate limiter tracking is capped by a new limit (default sized generously enough for typical +deployments, e.g. on the order of 10,000 tracked clients) using least-recently-used eviction once +the cap is reached. This bounds worst-case memory regardless of how many distinct identities are +ever presented, at the cost of possible churn (an idle client's tracked state being evicted to make +room for an active one) under sustained high-cardinality abuse, a fairness tradeoff, not a resource +leak, and one already covered by the existing guidance to keep the identity header behind a trusted +gateway. + +### Header trust boundary + +Because `X-User-ID` is trusted the same way `X-Scope-OrgID` is, deployments that expose Cortex's +HTTP endpoints directly to untrusted clients (rather than through a gateway/proxy that strips and +re-sets both headers) would let a malicious caller set an arbitrary identity, for example to evade +throttling by rotating identity values, or to frame another client. This is not a new risk specific +to this feature: it is the exact risk profile `X-Scope-OrgID` already carries today, and Cortex's +existing guidance (run behind a reverse proxy/gateway that authenticates callers and controls these +headers) applies unchanged. This should be stated explicitly in the docs for this feature, the same +way multi-tenancy setup docs already caution about `X-Scope-OrgID` exposure. The bounded tracking +above ensures that even under this threat model, the failure mode is bounded churn, not unbounded +memory growth. + +### Metrics + +- The existing discarded-samples metric gains a new discard reason for this rate limit, following + the same reason-labeling convention already used for the tenant-level rate limiter, so it's + observable per tenant the same way tenant-level rate limiting already is. +- Per-client rejection visibility (which specific client tripped the limit) is available from + request logs rather than a dedicated per-client metric label. Adding a client label directly to + the discarded-samples metric would multiply its existing cardinality (already keyed by discard + reason and tenant) by the number of distinct tracked clients; logs are the better fit for a + dimension that's expected to have many more distinct values than tenants do. This mirrors the + same choice made in the companion query-path proposal, for consistency between the two. + +A cardinality caveat on tracking itself, separate from the metrics question above: client identity +values are gateway-controlled, but a deployment that sets a distinct identity per end user (rather +than per service/team) could still produce many distinct tracked clients. The bounded-tracking cap +described above already limits the worst case; this is called out here because it's the same +underlying cardinality concern Cortex's existing per-labelset limits feature already warns about, +just showing up in a different place (limiter memory) rather than metrics. + +### Interaction with existing limits + +This is purely additive: + +- **The tenant-level ingestion rate limit is unaffected** and remains the hard ceiling for the + tenant as a whole. +- **Per-labelset limits** partition by data content (label matchers); this proposal partitions by + data origin (who sent it). They are orthogonal and can be used together. +- If a tenant has no client-identity limit configured (the default), or the request has no + `X-User-ID` header, the new check is a no-op and behavior is identical to today. + +## Rollout Plan + +Introduced as an **experimental** feature, consistent with how Cortex introduces most new limits +(disabled by default, documented as experimental, graduated to stable after operational +experience). Rollout in two phases: + +**Phase 1**: Ship local-only enforcement (see "Local enforcement is less precise" above) behind the +`client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` (disabled) so existing +deployments see no change. Validate with a small number of opted-in tenants. + +**Phase 2**: Based on operational feedback on how much the local-only imprecision matters in +practice, add the global variant (dividing the per-client limit across healthy distributors) and +consider graduating the feature out of experimental status. + +## Alternatives Considered + +- **mTLS client certificate identity.** Cryptographically stronger than a trusted header, and + Cortex's server already supports client-cert authentication. Rejected as the primary mechanism + because it only works when Cortex itself terminates client TLS: a common deployment pattern is a + service mesh or gateway terminating mTLS *before* Cortex, in which case the certificate Cortex + sees reflects the mesh sidecar, not the original caller. It would also reopen an unresolved + Common-Name-vs-Subject-Alternative-Name identity question. The header-based approach sidesteps + both problems and matches Cortex's existing `X-Scope-OrgID` trust model rather than introducing a + second, different one. Could still be revisited later as an alternate identity *source* feeding + the same enforcement path, for deployments that do terminate TLS at Cortex. +- **JWT claim-based identity.** Would require Cortex to parse and validate bearer tokens itself, + which is a meaningfully larger surface (token validation, key rotation, clock skew) than reading a + header the same way `X-Scope-OrgID` already is. Left as potential future work if operators need + cryptographic identity guarantees stronger than the trusted-header model provides. +- **Basic Auth username.** Only applicable if a deployment terminates HTTP Basic Auth at Cortex + itself, which is uncommon for the write path in practice (most Cortex deployments put + authentication at a gateway/proxy in front of Cortex, per the Authentication Gateway proposal), + the same gateway that would set `X-User-ID` under this proposal, making a separate Basic Auth + path redundant. + +## Open Questions + +- Should the client-identity limiter default to enabled with a very high limit (encouraging + visibility/metrics even for tenants that don't want enforcement), or fully opt-in (0 = off, as + described above)? This proposal defaults to fully opt-in to guarantee zero behavior change on + upgrade, consistent with how most new Cortex limits are introduced as disabled-by-default + experimental features. +- Should the header name be configurable, the way Cortex already allows configuring the header used + for source-IP logging, rather than a hardcoded `X-User-ID`? Leaning toward configurable, to avoid + collisions with header names deployments may already use for a similar purpose. +- Should this eventually extend to the query path (read-side per-client throttling, reusing this + same `X-User-ID` header)? **Yes: see the companion proposal, + [Client-Identity Query Rate Limiting](./client-identity-query-rate-limiting.md),** filed alongside + this one and deliberately kept separate rather than expanding this proposal's scope. +- What's the right default for the tracked-clients cap, and should evictions under sustained + pressure be observable (e.g. a counter for evictions caused by hitting the cap, distinct from + normal idle-entry turnover)? A visible eviction-rate metric would let operators tell "the cap is + properly sized" apart from "we're actively being hit with identity churn," which matters for + diagnosing the tradeoff described above. From ebf105ec12d6caaa63ee664eb7eea6f1265092c9 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Thu, 9 Jul 2026 20:57:38 -0700 Subject: [PATCH 2/4] proposals: use global counters for per-client rate limiting Both proposals now specify global enforcement (dividing the per-client limit across healthy replicas via the existing ring/distributor-count mechanism) rather than local-only with global as a follow-up. - Write path: rename section 'Local enforcement is less precise' -> 'Global enforcement'; update enforcement description and rollout plan - Query path: rename section 'Local enforcement precision...' -> 'Global enforcement and interaction with the results cache'; update frontend/scheduler-split section, rate-limiter description, rollout plan, and open-questions cross-reference Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 35 ++++++-------- .../client-identity-rate-limiting.md | 46 ++++++++----------- 2 files changed, 35 insertions(+), 46 deletions(-) diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md index 249f35375a..c1f21e071b 100644 --- a/docs/proposals/client-identity-query-rate-limiting.md +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -131,7 +131,7 @@ Query request arrives at query-frontend Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) ``` -A new rate limiter instance, reusing the same local rate-limiting mechanism as the write-path +A new rate limiter instance, reusing the same global rate-limiting mechanism as the write-path proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection under this feature is observable the same way existing query-frontend rejections already are. @@ -145,10 +145,10 @@ limits interface pattern already used elsewhere for query-side limits. Cortex's query-frontend can run with or without a separate query-scheduler. The query-frontend's request handler runs in the query-frontend process in both configurations: it is the entrypoint before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather -than inside the scheduler's own queueing logic, covers both deployment topologies with one check and -avoids needing per-scheduler-replica state coordination for something that is already a -per-frontend-replica local rate limit, the same "local" semantics the write-path proposal's default -strategy already has. +than inside the scheduler's own queueing logic, covers both deployment topologies with one check. +Using global counters (shared across all query-frontend replicas via the same mechanism as the +write-path proposal) means a client cannot route around the limit by landing on a less-loaded +frontend replica. ### Bounding memory: capped tracking, not an unbounded map @@ -159,16 +159,13 @@ unbounded map, exactly as described in the write-path proposal's equivalent sect are configured independently (see Configuration above) since the query-frontend and distributor are separate processes with independent memory budgets, but the mechanism and rationale are identical. -### Local enforcement precision, and interaction with the results cache +### Global enforcement and interaction with the results cache -The query-frontend, like the distributor, typically runs multiple replicas behind a load balancer, -so the same local-enforcement precision caveat raised in the write-path proposal applies here too: -one client's query traffic is a smaller, lumpier stream than a tenant's aggregate traffic, so -which replica a given request lands on matters more than it does for tenant-wide limits. The same -resolution applies: ship local-only first, treat a global variant as a near-term follow-up based on -operational experience, not indefinitely deferred work. +This proposal uses global counters, consistent with the write-path proposal. The same enforcement +precision that global counters provide on the write path applies here: a client cannot route around +the limit by spreading requests across frontend replicas. -A second, read-path-specific question: the query-frontend serves some requests entirely from its +A read-path-specific question remains: the query-frontend serves some requests entirely from its results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check before knowing whether a request will be a cache hit means a client issuing frequent but cheap, repeated, cacheable queries (for example, an auto-refreshing dashboard hitting the same query @@ -203,11 +200,10 @@ silently deferred. ## Rollout Plan -Same phased approach as the write-path proposal: introduced as an **experimental** feature, local -enforcement only, disabled by default (`client_identity_query_rate_limit: 0`). A global variant and -graduation out of experimental status follow once operational experience from Phase 1 shows how -much the local-only imprecision, and the cache-interaction question raised above, matter in -practice. +Same phased approach as the write-path proposal: introduced as an **experimental** feature with +global enforcement, disabled by default (`client_identity_query_rate_limit: 0`). Graduation out of +experimental status follows once operational experience from Phase 1, including resolution of the +cache-interaction question above, confirms the design is sound. ## Alternatives Considered @@ -233,8 +229,7 @@ practice. capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow refinement rather than blocking the initial absolute-number implementation. -- Should the per-client check run before or after results-cache eligibility is known (see "Local - enforcement precision, and interaction with the results cache" above)? Checking after would avoid +- Should the per-client check run before or after results-cache eligibility is known (see "Global enforcement and interaction with the results cache" above)? Checking after would avoid throttling cheap, cache-served requests, at the cost of being a slightly more invasive change to the request handling order; checking before (as currently proposed) is simpler but risks throttling based on request *count* rather than actual resource consumption. This should be diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md index 0885dfbf95..9fdf71c74a 100644 --- a/docs/proposals/client-identity-rate-limiting.md +++ b/docs/proposals/client-identity-rate-limiting.md @@ -160,9 +160,8 @@ Push request Continue to ingestion ``` -A second rate limiter instance, reusing Cortex's existing local rate-limiting mechanism unmodified, -is added to the distributor, keyed by tenant and client identity together rather than by tenant -alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +A second rate limiter instance, reusing Cortex's existing global rate-limiting mechanism, is added +to the distributor, keyed by tenant and client identity together rather than by tenant alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing tenant-level rate limit; this only prevents one client from consuming the entire budget. @@ -171,29 +170,25 @@ enforcement point resolves the tenant from the combined key and reads the config that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other Cortex limit; only the token bucket enforcing it is split per client. -### Local enforcement is less precise here than at the tenant level +### Global enforcement Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces its share of the limit independently) and a "global" mode (the limit is divided evenly across healthy distributors, trading a little coordination overhead for much tighter enforcement). -"Local" is imprecise by design: which distributor a given request lands on is effectively random, -so a client's traffic isn't evenly spread across replicas in the short term, and a burst can land -disproportionately on one distributor. - -That imprecision matters less at the tenant level, where traffic volume is usually large enough -relative to distributor count for the law of large numbers to smooth it out. At the per-client -granularity this proposal adds, that assumption is weaker: one client's traffic is a smaller, -lumpier stream, so local-only enforcement is more likely to either erroneously throttle a -well-behaved client (bad luck concentrates its requests on an already-busy distributor) or let a -misbehaving client through for longer than intended (its requests happen to spread across -distributors, each individually staying under its local share). This is a real precision gap in a -local-only first version, not merely a style choice inherited from the tenant-level limiter. - -Given that, this proposal's initial version still ships local-only, matching how Cortex introduces -most new limits, but a global variant (dividing the per-client limit across healthy distributors, -mirroring the existing global tenant-level strategy) should be treated as a near-term follow-up -once operational experience shows how much the imprecision matters in practice, not as indefinitely -deferred future work. + +At the per-client granularity this proposal adds, local enforcement is more likely to either +erroneously throttle a well-behaved client (bad luck concentrates its requests on an already-busy +distributor) or let a misbehaving client through for longer than intended (its requests happen to +spread across distributors, each individually staying under its local share). One client's traffic +is a smaller, lumpier stream than a tenant's aggregate traffic, so the law-of-large-numbers +smoothing that makes local enforcement tolerable at the tenant level is weaker here. + +This proposal therefore ships with **global enforcement by default**, mirroring Cortex's existing +`ingestion_rate_strategy: global` option for tenant-level limits. The per-client limit is divided +evenly across healthy distributor replicas using the same ring membership and distributor count +already tracked for global tenant-level enforcement. This ensures a client cannot route around the +limit by having its requests spread across replicas, and avoids the erroneous-throttle / bypass +risks that a local-only strategy would introduce at this granularity. ### Bounding memory: capped tracking, not an unbounded map @@ -265,13 +260,12 @@ Introduced as an **experimental** feature, consistent with how Cortex introduces (disabled by default, documented as experimental, graduated to stable after operational experience). Rollout in two phases: -**Phase 1**: Ship local-only enforcement (see "Local enforcement is less precise" above) behind the +**Phase 1**: Ship with global enforcement (see "Global enforcement" above) behind the `client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` (disabled) so existing deployments see no change. Validate with a small number of opted-in tenants. -**Phase 2**: Based on operational feedback on how much the local-only imprecision matters in -practice, add the global variant (dividing the per-client limit across healthy distributors) and -consider graduating the feature out of experimental status. +**Phase 2**: Based on operational feedback, consider graduating the feature out of experimental +status. ## Alternatives Considered From 19f20b7b757f79a2a03858914537b92c23703237 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Thu, 9 Jul 2026 21:01:55 -0700 Subject: [PATCH 3/4] proposals: clarify global counters use gossip, not divide-by-N The previous commit said 'global enforcement' but implied Cortex's existing divide-by-replica-count approximation. This updates both proposals to describe the intended gossip-based mechanism: - Each replica tracks a local per-client counter - Counters are gossiped to peers via the existing memberlist transport - Enforcement is against the cluster-wide sum, not a per-replica share This is more accurate than divide-by-N: it handles uneven load distribution and replica count changes without reconfiguration. The gossip convergence lag (order of memberlist gossip interval) is noted as an acceptable property of this best-effort limit. Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 25 +++++++----- .../client-identity-rate-limiting.md | 39 ++++++++++++------- 2 files changed, 41 insertions(+), 23 deletions(-) diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md index c1f21e071b..a4a4122a24 100644 --- a/docs/proposals/client-identity-query-rate-limiting.md +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -131,8 +131,8 @@ Query request arrives at query-frontend Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) ``` -A new rate limiter instance, reusing the same global rate-limiting mechanism as the write-path -proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses +A new rate limiter instance, using the same gossip-based global counter mechanism as the +write-path proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection under this feature is observable the same way existing query-frontend rejections already are. @@ -146,9 +146,12 @@ Cortex's query-frontend can run with or without a separate query-scheduler. The request handler runs in the query-frontend process in both configurations: it is the entrypoint before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather than inside the scheduler's own queueing logic, covers both deployment topologies with one check. -Using global counters (shared across all query-frontend replicas via the same mechanism as the -write-path proposal) means a client cannot route around the limit by landing on a less-loaded -frontend replica. + +Each query-frontend replica maintains a local per-client counter, gossips it to peers via the +existing memberlist transport (the same mechanism as the write-path proposal), and enforces against +the cluster-wide sum. A client cannot route around the limit by landing on a less-loaded frontend +replica, and enforcement stays accurate as replicas scale up or down without any reconfiguration of +per-replica shares. ### Bounding memory: capped tracking, not an unbounded map @@ -159,11 +162,13 @@ unbounded map, exactly as described in the write-path proposal's equivalent sect are configured independently (see Configuration above) since the query-frontend and distributor are separate processes with independent memory budgets, but the mechanism and rationale are identical. -### Global enforcement and interaction with the results cache +### Global enforcement via gossip and interaction with the results cache -This proposal uses global counters, consistent with the write-path proposal. The same enforcement -precision that global counters provide on the write path applies here: a client cannot route around -the limit by spreading requests across frontend replicas. +This proposal uses the same gossip-based global counters as the write-path proposal: each +query-frontend replica maintains a local per-client counter, gossips it to peers via memberlist, +and enforces against the cluster-wide sum. A client cannot route around the limit by spreading +requests across frontend replicas, and the gossip convergence lag (on the order of the memberlist +gossip interval) is acceptable for a best-effort rate limit, the same as on the write path. A read-path-specific question remains: the query-frontend serves some requests entirely from its results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check @@ -229,7 +234,7 @@ cache-interaction question above, confirms the design is sound. capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow refinement rather than blocking the initial absolute-number implementation. -- Should the per-client check run before or after results-cache eligibility is known (see "Global enforcement and interaction with the results cache" above)? Checking after would avoid +- Should the per-client check run before or after results-cache eligibility is known (see "Global enforcement via gossip and interaction with the results cache" above)? Checking after would avoid throttling cheap, cache-served requests, at the cost of being a slightly more invasive change to the request handling order; checking before (as currently proposed) is simpler but risks throttling based on request *count* rather than actual resource consumption. This should be diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md index 9fdf71c74a..1629a92c1d 100644 --- a/docs/proposals/client-identity-rate-limiting.md +++ b/docs/proposals/client-identity-rate-limiting.md @@ -160,8 +160,10 @@ Push request Continue to ingestion ``` -A second rate limiter instance, reusing Cortex's existing global rate-limiting mechanism, is added -to the distributor, keyed by tenant and client identity together rather than by tenant alone. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +A second rate limiter instance is added to the distributor, keyed by tenant and client identity +together rather than by tenant alone. It tracks a local per-client counter, gossips it to peers +via memberlist, and enforces against the cluster-wide sum — see "Global enforcement via gossip" +above. This check runs as an *additional* step alongside, not instead of, the existing tenant-level limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing tenant-level rate limit; this only prevents one client from consuming the entire budget. @@ -170,25 +172,36 @@ enforcement point resolves the tenant from the combined key and reads the config that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other Cortex limit; only the token bucket enforcing it is split per client. -### Global enforcement +### Global enforcement via gossip Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces its share of the limit independently) and a "global" mode (the limit is divided evenly across -healthy distributors, trading a little coordination overhead for much tighter enforcement). +healthy distributors by replica count). Neither is quite right for per-client enforcement. At the per-client granularity this proposal adds, local enforcement is more likely to either erroneously throttle a well-behaved client (bad luck concentrates its requests on an already-busy distributor) or let a misbehaving client through for longer than intended (its requests happen to spread across distributors, each individually staying under its local share). One client's traffic -is a smaller, lumpier stream than a tenant's aggregate traffic, so the law-of-large-numbers -smoothing that makes local enforcement tolerable at the tenant level is weaker here. - -This proposal therefore ships with **global enforcement by default**, mirroring Cortex's existing -`ingestion_rate_strategy: global` option for tenant-level limits. The per-client limit is divided -evenly across healthy distributor replicas using the same ring membership and distributor count -already tracked for global tenant-level enforcement. This ensures a client cannot route around the -limit by having its requests spread across replicas, and avoids the erroneous-throttle / bypass -risks that a local-only strategy would introduce at this granularity. +is a smaller, lumpier stream than a tenant's aggregate, so the law-of-large-numbers smoothing that +makes local enforcement tolerable at the tenant level is weaker here. + +The existing "global" divide-by-N mode improves on this but still assumes even load distribution: +when a client's traffic lands disproportionately on one replica, that replica's share of the budget +is exhausted faster than others and enforcement becomes uneven again. + +This proposal therefore uses **gossip-based global counters**. Each distributor maintains a local +per-client counter, gossips it to peers via the existing memberlist transport (the same mechanism +already used for the distributor ring), and enforces against the **cluster-wide sum** of all +replicas' local counts. This means: + +- A client's rate is enforced against its true cluster-wide arrival rate, not an approximation + derived from assuming even distribution across replicas. +- Enforcement remains accurate as replicas are added, removed, or experience uneven load — no + reconfiguration of per-replica shares is needed. +- The gossip convergence window introduces a small lag (on the order of the memberlist gossip + interval), during which a burst can transiently exceed the limit before all replicas see the + updated sum. This is an inherent property of eventual consistency and is acceptable for a + best-effort rate limit of this kind. ### Bounding memory: capped tracking, not an unbounded map From 5780a0fc06588c7adaf862cfbe911c553a4e4818 Mon Sep 17 00:00:00 2001 From: Krunal Jain Date: Fri, 10 Jul 2026 13:36:04 -0700 Subject: [PATCH 4/4] proposals: address review feedback on global/per-tenant language and enforcement design - Replace 'global enforcement via gossip' with ring-based distributed enforcement: shard (tenant, client) onto the existing ring so one replica owns the authoritative counter per key; works on any ring backend (Consul/Etcd/memberlist), not memberlist-only. - Rename 'global' counter language throughout both proposals to 'distributed per-tenant' to avoid confusion with cross-tenant scope. - Move tracked_clients_limit from a process-wide cap to a per-tenant override, consistent with every other Cortex limit. - Clarify LRU eviction semantics under rate limiting: eviction resets a client's counter (allowing a brief burst), which is the intentional tradeoff over refusing new clients when the cap is full. Signed-off-by: Krunal Jain --- .../client-identity-query-rate-limiting.md | 63 +++++--- .../client-identity-rate-limiting.md | 140 +++++++++++------- 2 files changed, 133 insertions(+), 70 deletions(-) diff --git a/docs/proposals/client-identity-query-rate-limiting.md b/docs/proposals/client-identity-query-rate-limiting.md index a4a4122a24..34e51ec640 100644 --- a/docs/proposals/client-identity-query-rate-limiting.md +++ b/docs/proposals/client-identity-query-rate-limiting.md @@ -90,15 +90,16 @@ New limits, following the same pattern as the write-path proposal: frontend: client_identity_query_rate_limit: 0 # 0 = disabled (default; no behavior change) client_identity_query_burst_size: 0 - # Same bounded-tracking cap as the write-path proposal, sized independently since the - # frontend and distributor are separate processes with separate memory budgets. - client_identity_query_tracked_clients_limit: 10000 # Per-tenant override via runtime config overrides: tenant-123: client_identity_query_rate_limit: 50 client_identity_query_burst_size: 100 + # Per-tenant cap on how many distinct tracked client entries the per-client rate limiter + # keeps in memory for this tenant; oldest-used entries are evicted once this is reached. + # See "Bounding memory" in the write-path proposal for eviction semantics. + client_identity_query_tracked_clients_limit: 1000 ``` Units are queries per second (matching how Cortex's existing outstanding-requests limit is already @@ -131,7 +132,7 @@ Query request arrives at query-frontend Continue (queued locally, or forwarded to the scheduler, depending on deployment topology) ``` -A new rate limiter instance, using the same gossip-based global counter mechanism as the +A new rate limiter instance, using the same ring-based distributed enforcement as the write-path proposal, is added at the query-frontend, keyed by tenant and client identity together. This reuses the query-frontend's existing rejected-request counter and discard-reason pattern, so a rejection under this feature is observable the same way existing query-frontend rejections already are. @@ -147,28 +148,29 @@ request handler runs in the query-frontend process in both configurations: it is before a request is either queued locally or forwarded to the scheduler, so enforcing here, rather than inside the scheduler's own queueing logic, covers both deployment topologies with one check. -Each query-frontend replica maintains a local per-client counter, gossips it to peers via the -existing memberlist transport (the same mechanism as the write-path proposal), and enforces against -the cluster-wide sum. A client cannot route around the limit by landing on a less-loaded frontend -replica, and enforcement stays accurate as replicas scale up or down without any reconfiguration of -per-replica shares. +Each query-frontend replica uses the same ring-based distributed enforcement as the write-path +proposal: the `(tenant, client)` key is hashed onto the existing ring, and the owning replica +maintains the authoritative per-tenant, per-client counter. A client cannot route around the limit +by landing on a less-loaded frontend replica, and enforcement stays accurate as replicas scale up +or down without any reconfiguration of per-replica shares. ### Bounding memory: capped tracking, not an unbounded map Same concern as the write-path proposal, and the same fix: the number of distinct tenant+client keys here is bounded only by how many distinct `X-User-ID` values are presented to the -query-frontend, so tracking is capped with least-recently-used eviction rather than kept in a plain -unbounded map, exactly as described in the write-path proposal's equivalent section. The two caps -are configured independently (see Configuration above) since the query-frontend and distributor are +query-frontend, so tracking is capped per tenant with least-recently-used eviction rather than kept +in a plain unbounded map, exactly as described in the write-path proposal's equivalent section +(including its discussion of the enforcement consequence of eviction). The two caps are configured +independently per tenant (see Configuration above) since the query-frontend and distributor are separate processes with independent memory budgets, but the mechanism and rationale are identical. -### Global enforcement via gossip and interaction with the results cache +### Distributed enforcement and interaction with the results cache -This proposal uses the same gossip-based global counters as the write-path proposal: each -query-frontend replica maintains a local per-client counter, gossips it to peers via memberlist, -and enforces against the cluster-wide sum. A client cannot route around the limit by spreading -requests across frontend replicas, and the gossip convergence lag (on the order of the memberlist -gossip interval) is acceptable for a best-effort rate limit, the same as on the write path. +This proposal uses the same ring-based distributed enforcement as the write-path proposal: each +`(tenant, client)` key is sharded onto the existing ring, and the owning query-frontend replica +maintains the authoritative per-tenant, per-client counter. A client cannot route around the +per-tenant limit by spreading requests across frontend replicas, and enforcement stays accurate as +replicas scale without any per-replica reconfiguration. A read-path-specific question remains: the query-frontend serves some requests entirely from its results cache, without ever reaching a querier or store-gateway. Enforcing the per-client check @@ -203,6 +205,26 @@ silently deferred. - **Consistent with the write-path proposal's philosophy**: purely additive, off by default, identical two-gate opt-in structure, same underlying rate-limiting mechanism. +## Value of the Identity Header Beyond Rate Limiting + +The same `X-User-ID` header established on the write path (see the companion proposal) carries +equivalent value on the read path once it is available: + +- **Query resource consumption per client.** Query execution time, querier CPU, and store-gateway + reads can all be attributed per client in logs, enabling identification of expensive query + sources — runaway alert rules, auto-refreshing dashboards, ad hoc exploration — without manual + log correlation. +- **Per-client slow query attribution.** The query-frontend already logs slow queries; with + `X-User-ID` present, slow query log lines carry the client identity directly, making it + straightforward to identify which client or dashboard is responsible without cross-referencing + external systems. +- **Future per-client query cost quotas.** Once client identity is a reliable, trusted dimension on + the read path, per-client cost controls — for example, a cap on total querier time or + store-gateway bytes scanned per client per interval — become implementable without any additional + identity plumbing. + +These are downstream opportunities enabled by this proposal's identity header, not in scope here. + ## Rollout Plan Same phased approach as the write-path proposal: introduced as an **experimental** feature with @@ -230,6 +252,11 @@ cache-interaction question above, confirms the design is sound. touch different components and teams (distributor vs. query-frontend) might review them separately? Leaning towards shipping independently, sharing only the small identity-extraction logic, given the review benefits of keeping them as separate changes discussed above. +- What's the right default for the per-tenant tracked-clients cap? The cap is configurable per + tenant via `client_identity_query_tracked_clients_limit` in the overrides; evictions caused by + hitting the cap are observable via a dedicated eviction counter, letting operators distinguish + "cap is properly sized" from "we're actively being hit with identity churn" without needing to + inspect memory profiles. - Should the per-client query limit be expressible as a percentage of the tenant's overall query capacity, similar to how Cortex's existing max-queriers-per-tenant limit supports fractional/percentage values, rather than an absolute queries/second number? Left as a fast-follow diff --git a/docs/proposals/client-identity-rate-limiting.md b/docs/proposals/client-identity-rate-limiting.md index 1629a92c1d..bb045f2c6f 100644 --- a/docs/proposals/client-identity-rate-limiting.md +++ b/docs/proposals/client-identity-rate-limiting.md @@ -108,16 +108,16 @@ limit: distributor: client_identity_ingestion_rate_limit: 0 # 0 = disabled (default; no behavior change) client_identity_ingestion_burst_size: 0 - # Global cap (not per-tenant) on how many distinct tracked tenant+client entries the - # per-client rate limiter keeps in memory at once; oldest-used entries are evicted once - # this is reached. See "Bounding memory" below. - client_identity_ingestion_tracked_clients_limit: 10000 # Per-tenant override via runtime config overrides: tenant-123: client_identity_ingestion_rate_limit: 5000 client_identity_ingestion_burst_size: 10000 + # Per-tenant cap on how many distinct tracked client entries the per-client rate limiter + # keeps in memory for this tenant; oldest-used entries are evicted once this is reached. + # See "Bounding memory" below. + client_identity_ingestion_tracked_clients_limit: 1000 ``` `client_identity_ingestion_rate_limit: 0` is the default and means "disabled", the same convention @@ -161,9 +161,9 @@ Continue to ingestion ``` A second rate limiter instance is added to the distributor, keyed by tenant and client identity -together rather than by tenant alone. It tracks a local per-client counter, gossips it to peers -via memberlist, and enforces against the cluster-wide sum — see "Global enforcement via gossip" -above. This check runs as an *additional* step alongside, not instead of, the existing tenant-level +together rather than by tenant alone. It enforces against the authoritative per-tenant, per-client +counter via ring-based sharding — see "Distributed enforcement via the distributor ring" +below. This check runs as an *additional* step alongside, not instead of, the existing tenant-level limiter: a request must pass both. A tenant's aggregate throughput is still capped by the existing tenant-level rate limit; this only prevents one client from consuming the entire budget. @@ -172,36 +172,34 @@ enforcement point resolves the tenant from the combined key and reads the config that tenant's overrides. The *limit value* is still a single per-tenant number, same as every other Cortex limit; only the token bucket enforcing it is split per client. -### Global enforcement via gossip - -Cortex's existing tenant-level rate limiter supports both a "local" mode (each distributor enforces -its share of the limit independently) and a "global" mode (the limit is divided evenly across -healthy distributors by replica count). Neither is quite right for per-client enforcement. - -At the per-client granularity this proposal adds, local enforcement is more likely to either -erroneously throttle a well-behaved client (bad luck concentrates its requests on an already-busy -distributor) or let a misbehaving client through for longer than intended (its requests happen to -spread across distributors, each individually staying under its local share). One client's traffic -is a smaller, lumpier stream than a tenant's aggregate, so the law-of-large-numbers smoothing that -makes local enforcement tolerable at the tenant level is weaker here. - -The existing "global" divide-by-N mode improves on this but still assumes even load distribution: -when a client's traffic lands disproportionately on one replica, that replica's share of the budget -is exhausted faster than others and enforcement becomes uneven again. - -This proposal therefore uses **gossip-based global counters**. Each distributor maintains a local -per-client counter, gossips it to peers via the existing memberlist transport (the same mechanism -already used for the distributor ring), and enforces against the **cluster-wide sum** of all -replicas' local counts. This means: - -- A client's rate is enforced against its true cluster-wide arrival rate, not an approximation - derived from assuming even distribution across replicas. -- Enforcement remains accurate as replicas are added, removed, or experience uneven load — no - reconfiguration of per-replica shares is needed. -- The gossip convergence window introduces a small lag (on the order of the memberlist gossip - interval), during which a burst can transiently exceed the limit before all replicas see the - updated sum. This is an inherent property of eventual consistency and is acceptable for a - best-effort rate limit of this kind. +### Distributed enforcement via the distributor ring + +Per-client limits are enforced using the existing distributor ring rather than gossip-based counter +aggregation. Each `(tenant, client)` key is sharded onto the ring: the distributor instance that +owns the shard for that key maintains the authoritative rate-limit counter for it. + +When a distributor receives a write request carrying an `X-User-ID` value, it hashes +`(tenant_id, client_id)` to determine the owning ring member, then: + +- If it owns that shard, it increments and enforces the counter locally. +- If another replica owns the shard, it forwards the rate-limit check to that replica via an + internal RPC before accepting or rejecting the request. + +This approach: + +- **Works on any ring backend** (Consul, Etcd, memberlist) — not memberlist-only — because the + ring is used only for key ownership, not as a gossip transport. +- **Eliminates convergence lag**: the owning replica holds the single authoritative counter; there + is no eventual-consistency window during which a burst can transiently exceed the configured + per-tenant limit before all replicas converge. +- **Stays accurate under uneven load**: a client's requests landing on different distributors always + count against the same shard owner, regardless of which replica physically received each request. + +The tradeoff is a potential per-request remote call to the shard owner when the receiving +distributor does not own the relevant shard. For deployments with random load balancing in front of +distributors, a majority of checks will require this hop. Whether the added latency is acceptable +depends on the intra-cluster RPC latency for a given deployment; this is a known cost and worth +calling out explicitly in the implementation phase. ### Bounding memory: capped tracking, not an unbounded map @@ -216,13 +214,25 @@ without limit simply by rotating identities. Cortex already solves an analogous problem elsewhere: the tenant-federation regex resolver bounds a similar per-key cache with a fixed-size, least-recently-used eviction cache rather than a plain map, sized by an operator-configurable limit. This proposal adopts the same approach: the per-client -rate limiter tracking is capped by a new limit (default sized generously enough for typical -deployments, e.g. on the order of 10,000 tracked clients) using least-recently-used eviction once -the cap is reached. This bounds worst-case memory regardless of how many distinct identities are -ever presented, at the cost of possible churn (an idle client's tracked state being evicted to make -room for an active one) under sustained high-cardinality abuse, a fairness tradeoff, not a resource -leak, and one already covered by the existing guidance to keep the identity header behind a trusted -gateway. +tracking for each tenant is capped by a new per-tenant limit (configurable via +`client_identity_ingestion_tracked_clients_limit` in the tenant's overrides, with a sensible default +on the order of 1,000 tracked clients per tenant) using least-recently-used eviction once the cap +is reached. + +Unlike the federation regex resolver, where eviction only affects latency (a cache miss means a +slower regex rebuild), eviction here has an enforcement consequence: an evicted client's counter is +discarded, so if it reappears it starts from zero and can issue a fresh burst before its token +bucket fills and enforcement kicks in again. This is a deliberate tradeoff — the alternative +(refusing new clients once the cap is full) risks denying service to legitimate new clients when a +tenant has many active writers. The LRU approach bounds memory and preserves fairness under normal +cardinality, at the cost of a brief enforcement gap when a previously-evicted client resumes. Under +sustained high-cardinality abuse (an attacker rotating identities to flood the LRU), the failure +mode is bounded churn (evictions) rather than unbounded memory growth, which is acceptable given +the existing guidance to keep `X-User-ID` behind a trusted gateway. + +Operators can tune the per-tenant cap upward for tenants with many legitimate concurrent writers, +or observe the eviction counter (a dedicated counter incremented in the LRU eviction callback) to +distinguish "cap is properly sized" from "we are actively seeing identity churn." ### Header trust boundary @@ -267,15 +277,41 @@ This is purely additive: - If a tenant has no client-identity limit configured (the default), or the request has no `X-User-ID` header, the new check is a no-op and behavior is identical to today. +## Value of the Identity Header Beyond Rate Limiting + +Establishing `X-User-ID` as a first-class header in the write path opens up a broader set of +per-client observability and control capabilities that are not in scope for this proposal but +become straightforward extensions once the identity is available: + +- **Resource consumption attribution.** Samples ingested, bytes written, and active series can all + be broken down per client in logs and metrics, enabling capacity planning and chargeback at the + client level rather than only at the tenant level. +- **Out-of-order sample detection per client.** The distributor already detects and discards + out-of-order samples; with `X-User-ID` present, those rejections can be attributed to the + specific client sending them, making it straightforward to identify a misconfigured or misbehaving + writer without manual log correlation across requests. +- **Per-client cardinality tracking.** Today cardinality can only be attributed to a tenant as a + whole. With client identity available on the write path, high-cardinality contributors can be + identified by source, not just by label patterns. +- **Audit and debugging.** A client identity on every write request makes it possible to answer + questions like "which client started sending this metric series?" or "which client's deployment + caused this ingestion spike?" directly from logs, without cross-referencing external systems. +- **Future per-client quota enforcement.** Once client identity is a reliable, trusted dimension, + per-client quotas (not just rate limits) — for example, a cap on active series or total samples + per day per client — become implementable without any additional identity plumbing. + +None of these require changes to this proposal; they are downstream opportunities that justify +establishing a clean, trusted identity header now rather than bolting it on later. + ## Rollout Plan Introduced as an **experimental** feature, consistent with how Cortex introduces most new limits (disabled by default, documented as experimental, graduated to stable after operational experience). Rollout in two phases: -**Phase 1**: Ship with global enforcement (see "Global enforcement" above) behind the -`client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` (disabled) so existing -deployments see no change. Validate with a small number of opted-in tenants. +**Phase 1**: Ship with distributed enforcement (see "Distributed enforcement via the distributor ring" +above) behind the `client_identity_ingestion_rate_limit` per-tenant override, defaulting to `0` +(disabled) so existing deployments see no change. Validate with a small number of opted-in tenants. **Phase 2**: Based on operational feedback, consider graduating the feature out of experimental status. @@ -315,8 +351,8 @@ status. same `X-User-ID` header)? **Yes: see the companion proposal, [Client-Identity Query Rate Limiting](./client-identity-query-rate-limiting.md),** filed alongside this one and deliberately kept separate rather than expanding this proposal's scope. -- What's the right default for the tracked-clients cap, and should evictions under sustained - pressure be observable (e.g. a counter for evictions caused by hitting the cap, distinct from - normal idle-entry turnover)? A visible eviction-rate metric would let operators tell "the cap is - properly sized" apart from "we're actively being hit with identity churn," which matters for - diagnosing the tradeoff described above. +- What's the right default for the per-tenant tracked-clients cap? The cap is configurable per + tenant via `client_identity_ingestion_tracked_clients_limit` in the overrides; evictions caused + by hitting the cap are observable via a dedicated eviction counter, letting operators distinguish + "cap is properly sized" from "we're actively being hit with identity churn" without needing to + inspect memory profiles.