Skip to content

alert: silence non-Mobula head_lag stale alerts #922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Flotapponnier merged 1 commit into from
Jul 6, 2026
Merged

alert: silence non-Mobula head_lag stale alerts #922

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions harnesses/aggregator-head-lag/prometheus/alert_rules.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,24 +191,27 @@ groups:
summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Head lag metrics missing'
description: "No head_lag_seconds metrics received for 5 minutes. Check if monitors are running."

# Per-aggregator staleness: fires when one provider stops pushing data
# (e.g. Codex WS disconnect) while others keep running — global absent()
# would NOT catch this.
- alert: AggregatorHeadLagStale
expr: (time() - timestamp(head_lag_seconds)) > 300
# Per-aggregator staleness — Mobula only. Historically the rule fired
# on every aggregator (gecko, codex, ...) but the operator only cares
# about Mobula health here; noise from third-party WS instability on
# low-volume BSC pools was spamming the channel every few hours
# (2026-07-06 feedback). Non-Mobula aggregator health is tracked via
# log tailing, not alerts.
- alert: MobulaHeadLagStale
expr: (time() - timestamp(head_lag_seconds{aggregator="mobula"})) > 300
for: 1m
labels:
severity: warning
alert_type: head_lag_stale
app: aggregator_latency_monitor
annotations:
summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}{{ $labels.aggregator }} head_lag stale on {{ $labels.chain }} ({{ $labels.region }})'
summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula head_lag stale on {{ $labels.chain }} ({{ $labels.region }})'
description: |
**{{ $labels.aggregator }}** hasn't pushed a head_lag sample for **{{ $labels.chain }} / {{ $labels.region }}** in over 5 minutes.
**Mobula** hasn't pushed a head_lag sample for **{{ $labels.chain }} / {{ $labels.region }}** in over 5 minutes.

Likely cause: WebSocket disconnected, JWT expired, proxy IP banned, or auth cookie rotated.

Check the monitor logs for `[HEAD-LAG][{{ $labels.aggregator | toUpper }}]` errors.
Check the monitor logs for `[HEAD-LAG][MOBULA]` errors.

# Service availability
- alert: CodexServiceDown
Expand Down
Loading