Skip to content

aggregator-head-lag: drop 'unless changes==0' from series queries #1221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 1 commit into from
Jul 16, 2026
Merged

aggregator-head-lag: drop 'unless changes==0' from series queries #1221

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions benchmarks/aggregator-head-lag.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,14 +161,14 @@ providers:
mean: avg_over_time(head_lag_seconds{aggregator="mobula"}[24h]) * 1000
success: clamp_max(count_over_time(head_lag_seconds{aggregator="mobula"}[24h]) / 5760, 1)
sample_size: sum(count_over_time(head_lag_seconds{aggregator="mobula"}[24h]))
# `unless ... == 0` drops the sample when the gauge saw no fresh
# event in the last 15 min. Without this, the head_lag gauge stays
# frozen at its last value during a full WS outage and the chart
# draws a flat line where the reader would expect a gap. The
# per-series match means a partial outage on one chain leaves the
# other chains' points untouched (aggregate line stays continuous
# when at least one chain is still receiving events).
series: avg_over_time(head_lag_seconds{aggregator="mobula"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="mobula"}[15m]) == 0
# No `unless changes == 0` guard on the historical series: it
# false-drops points on backfilled data (Prom-compacted to ~1
# sample per hour, so `changes[15m]` returns 0 even though the
# gauge was healthy at the time). The 30D view was losing the
# first ~8 days of data to this. Live-feed silence is still
# surfaced by the `live_activity` scalar below (badge + banner
# + downtime pill), which runs against fresh 15s-cadence data.
series: avg_over_time(head_lag_seconds{aggregator="mobula"}[1h]) * 1000
# Total gauge-value changes on this aggregator across chains and
# regions in the last 15min. Non-zero = fresh events arrived; 0 =
# every subscription has been silent for 15+ minutes (the gauge is
Expand All @@ -178,13 +178,13 @@ providers:
regions:
- region: us-east
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="mobula", region="us-east"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="mobula", region="us-east"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="mobula", region="us-east"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="mobula", region="us-east"}[1h]) * 1000
- region: eu-west
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="mobula", region="eu-west"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="mobula", region="eu-west"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="mobula", region="eu-west"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="mobula", region="eu-west"}[1h]) * 1000
- region: ap-southeast
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="mobula", region="sgp"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="mobula", region="sgp"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="mobula", region="sgp"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="mobula", region="sgp"}[1h]) * 1000

- slug: codex
name: Codex
Expand All @@ -197,18 +197,18 @@ providers:
mean: avg_over_time(head_lag_seconds{aggregator="codex"}[24h]) * 1000
success: clamp_max(count_over_time(head_lag_seconds{aggregator="codex"}[24h]) / 5760, 1)
sample_size: sum(count_over_time(head_lag_seconds{aggregator="codex"}[24h]))
series: avg_over_time(head_lag_seconds{aggregator="codex"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="codex"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="codex"}[1h]) * 1000
live_activity: sum(changes(head_lag_seconds{aggregator="codex"}[15m]))
regions:
- region: us-east
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="codex", region="us-east"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="codex", region="us-east"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="codex", region="us-east"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="codex", region="us-east"}[1h]) * 1000
- region: eu-west
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="codex", region="eu-west"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="codex", region="eu-west"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="codex", region="eu-west"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="codex", region="eu-west"}[1h]) * 1000
- region: ap-southeast
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="codex", region="sgp"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="codex", region="sgp"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="codex", region="sgp"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="codex", region="sgp"}[1h]) * 1000

- slug: geckoterminal
name: GeckoTerminal
Expand All @@ -221,16 +221,16 @@ providers:
mean: avg_over_time(head_lag_seconds{aggregator="geckoterminal"}[24h]) * 1000
success: clamp_max(count_over_time(head_lag_seconds{aggregator="geckoterminal"}[24h]) / 5760, 1)
sample_size: sum(count_over_time(head_lag_seconds{aggregator="geckoterminal"}[24h]))
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="geckoterminal"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal"}[1h]) * 1000
live_activity: sum(changes(head_lag_seconds{aggregator="geckoterminal"}[15m]))
regions:
- region: us-east
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="geckoterminal", region="us-east"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal", region="us-east"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="geckoterminal", region="us-east"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal", region="us-east"}[1h]) * 1000
- region: eu-west
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="geckoterminal", region="eu-west"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal", region="eu-west"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="geckoterminal", region="eu-west"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal", region="eu-west"}[1h]) * 1000
- region: ap-southeast
p50: quantile_over_time(0.50, head_lag_seconds{aggregator="geckoterminal", region="sgp"}[24h]) * 1000
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal", region="sgp"}[1h]) * 1000 unless changes(head_lag_seconds{aggregator="geckoterminal", region="sgp"}[15m]) == 0
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal", region="sgp"}[1h]) * 1000

Loading