Skip to content

fix(001): true cross-chain avg with avg by (aggregator) - #1543

Merged
Flotapponnier merged 1 commit into
devfrom
fix/agg-head-lag-true-avg
Jul 26, 2026
Merged

fix(001): true cross-chain avg with avg by (aggregator)#1543
Flotapponnier merged 1 commit into
devfrom
fix/agg-head-lag-true-avg

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Problem

Previous fix removed All chains to hide the bug, but the real root cause was never fixed: prom.scalar takes result[0] from multi-series Prometheus vectors. Series come back alphabetically, so chain="base" always wins and every provider's global p50 was secretly just their Base latency.

Codex (0.8s on Base) appeared as global #1 despite Mobula winning on BNB, Solana, and Robinhood.

Fix

Wrap every p50/p90/p99/mean/success scalar query with avg by (aggregator) (...):

# Before — returns result[0] = Base data by alphabet
quantile_over_time(0.50, head_lag_seconds{aggregator="mobula"}[24h]) * 1000

# After — collapses to a single honest cross-chain mean
avg by (aggregator) (quantile_over_time(0.50, head_lag_seconds{aggregator="mobula"}[24h])) * 1000

When a chain filter is injected (e.g. Base tab selected), avg by (aggregator) collapses the 3 region series instead → still one clean scalar per provider.

After

  • All chains tab restored with honest data
  • Mobula ~858ms (avg Base 1477 + BNB 1000 + Solana 86 + Robinhood 868) < Codex ~1021ms → correct global leader
  • Per-chain tabs unchanged: Codex still correctly shows as drop journal framing #1 on Base
  • AT A GLANCE, homepage, search all now show Mobula as global leader

prom.scalar takes result[0] from multi-series vectors. Wrapping every
scalar query with avg by (aggregator)(...) collapses the multi-chain
(or multi-region) results into one correct mean before prom.scalar
reads. With a chain filter injected it collapses regions instead.

Result: Mobula ~858ms avg < Codex ~1021ms avg cross-chain, reflecting
3/4 chain wins. All chains tab restored with honest data.
@Flotapponnier
Flotapponnier merged commit 851b221 into dev Jul 26, 2026
1 check failed
Flotapponnier added a commit that referenced this pull request Jul 26, 2026
prom.scalar takes result[0] from multi-series vectors. Wrapping every
scalar query with avg by (aggregator)(...) collapses the multi-chain
(or multi-region) results into one correct mean before prom.scalar
reads. With a chain filter injected it collapses regions instead.

Result: Mobula ~858ms avg < Codex ~1021ms avg cross-chain, reflecting
3/4 chain wins. All chains tab restored with honest data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant