diff --git a/README.md b/README.md index 75b4e1e1..cb7eabed 100644 --- a/README.md +++ b/README.md @@ -1,99 +1,128 @@ # OpenChainBench -> An open, reproducible benchmark series for crypto infrastructure — aggregators, bridges, price feeds, and more. Live at [openchainbench.xyz](https://openchainbench.xyz). +> Open, reproducible benchmarks for crypto infrastructure — aggregators, bridges, RPCs, price feeds. Same metric, same conditions, every provider. Live at [openchainbench.xyz](https://openchainbench.xyz). -OpenChainBench is a "field journal" — a research-paper-styled site that publishes one benchmark at a time, each one shipping with the script that produced it. The goal is to make performance an observable property of crypto infra, the way uptime is for SaaS. +OpenChainBench publishes one benchmark at a time, each one shipping with the script that produces its data. The goal is to make performance an observable property of crypto infra — measured in the open, by anyone who wants to add a provider or a metric. -The project is **sponsored by [Mobula](https://mobula.io)** and **edited independently**: every provider (sponsor included) goes through the same harness with the same inputs, and corrections are published in the open. +The project is community-run, MIT-licensed, and accepts PRs from any party including the providers it benchmarks. ## What's inside ``` -benchmarks/ Spec files — one YAML per report -├── README.md Spec reference + submission guide -└── *.yml A spec wires editorial metadata to PromQL -src/ -├── app/ Next.js 16 App Router -│ ├── page.tsx Front page -│ ├── benchmarks/ -│ │ ├── page.tsx Index -│ │ └── [slug]/page.tsx One paper-styled report per slug -│ ├── methodology / about / press / opengraph-image / icon -│ └── globals.css Paper theme -├── components/ -│ ├── site-header / site-footer -│ ├── byline · big-number · section-rule · figure -│ ├── range-chart Editorial range plot (p50→p99 + median) -│ ├── ledger-table Stock-exchange-style table w/ sparklines -│ ├── region-grid Tufte small multiples per region -│ └── sparkline 24h trend SVG -├── data/ -│ ├── benchmarks.ts Mock data + async loaders -│ └── site.ts Site constants -└── lib/ - ├── prometheus.ts HTTP API client (instant + range queries) - ├── spec.ts YAML loader, overlay-live-on-mock policy - └── format.ts / utils.ts +benchmarks/ Spec files — one YAML per published benchmark +├── aggregator-head-lag.yml +├── bridge-quote-latency.yml +├── bridge-fee.yml +└── README.md Spec format reference + submission guide + +harnesses/ The runners that produce the metrics +├── aggregator-head-lag/ Go service: WebSocket monitor for indexation lag +├── bridge-quote/ Go service: 4-bridge quote latency + fees +├── bridge-fee/ Go service: cost-percent comparison +└── README.md Contract for new harnesses + +src/ Next.js 16 site (App Router, ISR, Tailwind v4) +├── app/ Pages — overview, benchmarks index, [slug] reports +├── components/ time-series-chart, ledger-table, region-grid, … +├── data/ Spec loader (YAML → Prometheus → Benchmark[]) +└── lib/ Prometheus client, spec schema (Zod), formatting + +scripts/ pnpm validate, pnpm spec:dry-run +docs/ Methodology, ADRs, style guide ``` -Tailwind v4 with a paper aesthetic — cream background, Source Serif 4 for body, Inter for UI labels, JetBrains Mono for figures. +## How a benchmark gets data + +``` +[harness] ──── push metrics ────▶ [Prometheus] + ▲ │ + │ │ PromQL queries + │ runs 24/7 on Railway │ (defined in YAML) + │ ▼ + │ [benchmarks/.yml] + │ │ + │ │ resolved server-side + │ │ at request time + │ ▼ + │ [Next.js site] ── ISR 60s + │ │ + │ ▼ + │ openchainbench.xyz/benchmarks/ + └──── source code lives in harnesses//, deployed from this repo +``` + +The harness is the source of truth: it calls real provider endpoints, measures latency / cost / success, and pushes Prometheus metrics with the labels declared in the spec. The site never fakes numbers — if the harness stops emitting, the affected percentiles disappear from the page rather than fall back to placeholders. + +## Architecture + +| Layer | Where it runs | Why | +|---|---|---| +| Site (Next.js, ISR) | Vercel | Static pages with 60s revalidate, edge cache | +| Prometheus | Railway | Time-series DB, 24/7 | +| Harnesses (Go) | Railway | Long-running WebSockets, schedulers, on-chain signing | + +Vercel and Railway are intentionally split: Vercel can't host long-lived WebSocket connections or sign on-chain transactions; Railway can't serve a globally cached Next.js site at the same cost. They communicate over HTTPS — the site queries Prom URLs declared in each YAML spec. -## Running locally +## Running the site locally ```bash pnpm install -pnpm dev +pnpm dev # http://localhost:3000 ``` -Open [localhost:3000](http://localhost:3000). - -## How a benchmark gets data +The site reads every `benchmarks/*.yml` at request time. Specs whose Prometheus URL the runtime can't reach render as drafts (no numbers, methodology only). +```bash +pnpm validate # schema-lint every spec in benchmarks/ +pnpm spec:dry-run # query Prometheus and print numbers, no rendering +pnpm build # production build ``` -benchmarks/.yml ── PromQL ──► Prometheus - │ │ - │ (fail / empty / no URL) │ (live numbers) - ▼ ▼ -src/data/benchmarks.ts (mock fallback) ─► Benchmark[] - │ - ▼ - Paper report at /benchmarks/ + +## Running a harness locally + +Each harness has its own README with run instructions. They are independent Go programs (one per benchmark) that you can build with `go run ./cmd/...` or via the included Dockerfile. + +```bash +cd harnesses/aggregator-head-lag +cp .env.example .env # fill in API keys +docker-compose up -d # local Prom + monitor + Grafana ``` -Pages revalidate every minute (Next.js ISR). Live numbers replace mock numbers as soon as the harness starts emitting metrics with the labels the spec expects. The site never half-renders: any missing percentile and the page falls back to the mock so readers can't be misled about what's real. +Set `prom_url` in the corresponding YAML to your local Prom (`http://localhost:9090`) to render the site against your own data. ## Adding a benchmark -1. **Editorial mock.** Append an entry to `MOCK_BENCHMARKS` in `src/data/benchmarks.ts` with title, abstract, methodology, findings and placeholder numbers per provider. This is what readers see until the harness fills Prometheus. -2. **Harness.** Drop the script that emits metrics into `harnesses//` (TypeScript, Bun or Python — whatever fits). The harness runs continuously and pushes to Prometheus / writes to a Pushgateway. -3. **Spec.** Create `benchmarks/.yml` with the Prometheus URL and the PromQL queries (see `benchmarks/aggregator-quote-latency.yml`). -4. **PR.** The build picks the spec up automatically. - -The shape of a benchmark report is fixed: abstract → range chart → ledger table → region grid → findings → methodology → citation. Don't drift; readers learn the format and skim by it. +Full guide in [CONTRIBUTING.md](./CONTRIBUTING.md). Short version: -## Data provenance +1. **Open an issue** with the [new-benchmark template](https://github.com/OpenChainBench/OpenChainBench/issues/new?template=new-benchmark.md). Sketch the metric, providers, methodology — get feedback before you build. +2. **Write the spec** at `benchmarks/.yml`. Format documented in [`benchmarks/README.md`](./benchmarks/README.md), validated by `src/lib/spec-schema.ts`. +3. **Build the harness** in `harnesses//`. Any language works as long as it pushes Prometheus metrics with the labels your spec references. See the existing harnesses as reference. +4. **Open a PR.** CI runs schema validation, typecheck, lint, and build. Once green and merged: the site picks up the new spec automatically; a maintainer wires the harness into Railway (one-time setup per benchmark). -Each report links its harness in the source line. Every run stores raw transcripts (request + response, timestamps, region) so any single data point can be audited after the fact. If a number is wrong we publish a dated correction in place; future issues acknowledge it on the masthead. +Hosting trade-off: light harnesses (one HTTP poll loop, no secrets) can be deployed onto the OpenChainBench Railway. Harnesses that hold wallets, sign transactions, or otherwise represent capital must run from infra owned by the contributor — they push metrics to a public Prom endpoint and the site queries it the same way. -## Editorial policy +## Editorial conventions -- Mobula does not see results before publication and cannot block a report. -- The benchmark plan (providers, routes, cadence, timeout) is committed before each run. -- Where Mobula loses, we publish that. Where it wins, we explain why. -- All code is MIT, all reports are CC-BY-4.0. +- **No pre-determined winners.** Specs do not declare a "best" provider. The leader on every page is computed at render time from the lowest p50. +- **Tail before mean.** Headlines use p50 and p99. The arithmetic mean is reported in the table but never used as a takeaway. +- **State the timeout.** Failures are excluded from latency aggregates and counted toward success rate. Both numbers are reported. +- **Methodology first.** A spec without a written methodology is rejected. +- **Corrections in place.** If a number is wrong we publish a dated note on the affected report; future readers see it on the masthead. ## Stack -- Next.js 16 (App Router) on Vercel -- Tailwind v4 (CSS-only theme) -- Source Serif 4 / Inter / JetBrains Mono via `next/font` -- Static rendering — every benchmark page is pre-rendered, with ISR for live runs once the harness is wired in. +- Next.js 16 (App Router, ISR, Turbopack) on Vercel +- Tailwind v4 (CSS-only theme, `@theme` tokens) +- Source Serif 4 / Inter Tight / JetBrains Mono via `next/font` +- Zod for spec validation +- Prometheus HTTP API (instant + range queries) +- Go 1.24 for the existing harnesses (any language is acceptable) -## Social +## Links -- Twitter / X — [@openchainbench](https://twitter.com/openchainbench) -- Reddit — [r/openchainbench](https://reddit.com/r/openchainbench) -- GitHub — [mobula/openchainbench](https://github.com/mobula/openchainbench) +- Site — [openchainbench.xyz](https://openchainbench.xyz) +- Twitter — [@openchainbench](https://twitter.com/openchainbench) +- GitHub — [OpenChainBench/OpenChainBench](https://github.com/OpenChainBench/OpenChainBench) ## License diff --git a/benchmarks/bridge-fee.yml b/benchmarks/bridge-fee.yml index 4db9de11..6b1944f7 100644 --- a/benchmarks/bridge-fee.yml +++ b/benchmarks/bridge-fee.yml @@ -30,7 +30,7 @@ methodology: findings: [] -source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-fee +source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-monitor prometheus: url: https://prometheus-production-9ffe.up.railway.app diff --git a/benchmarks/bridge-quote-latency.yml b/benchmarks/bridge-quote-latency.yml index f5569cf4..99d1ff8a 100644 --- a/benchmarks/bridge-quote-latency.yml +++ b/benchmarks/bridge-quote-latency.yml @@ -27,7 +27,7 @@ methodology: findings: [] -source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-quote +source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/bridge-monitor prometheus: url: https://prometheus-production-9ffe.up.railway.app diff --git a/harnesses/README.md b/harnesses/README.md index 60ed723c..a9b6e3aa 100644 --- a/harnesses/README.md +++ b/harnesses/README.md @@ -1,29 +1,29 @@ # Harnesses -Each subdirectory holds the actual benchmark script for one report — the thing that runs continuously, calls providers, and pushes metrics to Prometheus. +Each subdirectory holds the runner for one or more benchmarks — the long-running process that calls providers, measures latency / cost / success, and pushes metrics to Prometheus. The OpenChainBench site queries that same Prometheus and renders the data. ``` harnesses/ -├── aggregator-quote/ (Bench № 001) -├── bridge-e2e/ (Bench № 002) -├── feed-freshness/ (Bench № 003) -└── rpc-head-lag/ (Bench № 004) +├── aggregator-head-lag/ Bench № 001 — WebSocket monitor (Go) +└── bridge-monitor/ Bench № 002 + № 003 — quote loop + execution (Go) ``` +A single harness can serve multiple benchmarks when the same set of measurements is consumed by more than one spec — `bridge-monitor` is the canonical example, producing both `bridge_quote_latency_ms` (read by `bridge-quote-latency.yml`) and `bridge_cost_percent` (read by `bridge-fee.yml`). + ## Contract -A harness is whatever you need it to be — Bun, Node, Python, Rust, a cron-runner shell script. The contract it must satisfy is small: +A harness can be written in any language. The contract it must satisfy is small: -| Concern | Requirement | -| -------------- | ----------------------------------------------------------------------------------------------- | -| Inputs | Read provider API keys from env vars, never commit them | -| Loop | Run continuously (or on a fixed cadence ≥ 24h) and push the same metric set every iteration | -| Metric names | Match the names in the matching `benchmarks/.yml` exactly | -| Labels | At minimum `provider`, `region`. Additional labels (chain, route) are encouraged | -| Push target | `pushgateway` or remote-write to the project's Prometheus | -| Timeouts | Documented, fail closed (count toward success rate) | -| Reproducibility| README explains how to run it locally with one command | -| License | MIT, same as the rest of the repo | +| Concern | Requirement | +| --- | --- | +| Inputs | Read provider API keys / wallet keys from env vars, never commit them | +| Loop | Run continuously and push the same metric set every iteration | +| Metric names | Match the names referenced in the matching `benchmarks/.yml` exactly | +| Labels | Include `provider` (or equivalent) and `region` at minimum; chain/route labels encouraged | +| Push target | Scrape endpoint exposed for Prometheus, or push to a remote-write endpoint | +| Timeouts | Documented; failures fail closed (counted toward success rate, excluded from latency aggregates) | +| Reproducibility | README explains how to run it locally with one command | +| License | MIT, same as the rest of the repo | ## Subdirectory layout @@ -31,19 +31,28 @@ A harness is expected to ship at minimum: ``` harnesses// -├── README.md What it measures, the providers, the labels, env vars, how to run -├── Dockerfile Container image for the runner -├── Makefile or run.sh `make run` (or `./run.sh`) starts the loop -└── … Source files in whatever language fits +├── README.md What it measures, providers, labels, env vars, how to run +├── Dockerfile Container image for the runner +├── .env.example Every env var the runner reads, with placeholders +└── … Source files in whatever language fits ``` -The point isn't uniformity for its own sake; it's that any contributor can `cd` into a folder and figure out what's happening in 30 seconds. +If the harness ships a full local stack (Prometheus + Grafana + Alertmanager) it should include a `docker-compose.yml` and a `Makefile` for the common targets (`make run`, `make logs`, `make stop`). The two existing harnesses do. + +## Hosting + +Two paths exist for getting a harness into production: + +1. **OpenChainBench Railway.** Light harnesses (one HTTP loop, no wallets, no signing) can be deployed onto the project's shared Railway. A maintainer wires the service after the PR is merged. +2. **Contributor-hosted.** Harnesses that hold wallets, sign transactions, or otherwise represent capital must run from infrastructure owned by the contributor. They push metrics to a publicly-reachable Prometheus endpoint and the site queries it the same way as the project-hosted harnesses. + +Either way the YAML spec's `prom_url` decides which Prometheus the site reads from — the data path is identical. ## Submitting a new harness -See [`/CONTRIBUTING.md`](../CONTRIBUTING.md) for the full submission flow. tldr: +See [`/CONTRIBUTING.md`](../CONTRIBUTING.md) for the full submission flow. Short version: -1. Add the editorial mock in `src/data/benchmarks.ts`. -2. Add the spec in `benchmarks/.yml`. -3. Add the harness here. +1. Open an issue with the new-benchmark template (sketch the metric, providers, methodology). +2. Write the spec at `benchmarks/.yml`. +3. Build the harness here at `harnesses//`. 4. Open a PR. diff --git a/harnesses/aggregator-head-lag/.gitignore b/harnesses/aggregator-head-lag/.gitignore new file mode 100644 index 00000000..ec3fb764 --- /dev/null +++ b/harnesses/aggregator-head-lag/.gitignore @@ -0,0 +1,34 @@ +# Environment variables (contains secrets) +.env + +# Build artifacts +benchmark +*.exe +*.dll +*.so +*.dylib + +# Test binaries +*.test +*.out + +# Go workspace file +go.work + +# IDE +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Logs +*.log + +# Temporary files +tmp/ +temp/ diff --git a/harnesses/aggregator-head-lag/Dockerfile b/harnesses/aggregator-head-lag/Dockerfile new file mode 100644 index 00000000..74ca8f0f --- /dev/null +++ b/harnesses/aggregator-head-lag/Dockerfile @@ -0,0 +1,36 @@ +# Build stage +FROM golang:1.24-alpine AS builder + +WORKDIR /app + +# Install dependencies +RUN apk add --no-cache git + +# Copy go mod files +COPY go.mod go.sum ./ +RUN go mod download + +# Copy source code +COPY . . + +# Build the binary +RUN CGO_ENABLED=0 GOOS=linux go build -o /app/monitor ./cmd/script + +# Runtime stage +FROM debian:bookworm-slim + +WORKDIR /app + +# Install runtime dependencies +RUN apt-get update && apt-get install -y \ + ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +# Copy binary from builder +COPY --from=builder /app/monitor /app/monitor + +# Expose metrics port +EXPOSE 2112 + +# Run the monitor +CMD ["/app/monitor"] diff --git a/harnesses/aggregator-head-lag/Makefile b/harnesses/aggregator-head-lag/Makefile new file mode 100644 index 00000000..7c1b2c68 --- /dev/null +++ b/harnesses/aggregator-head-lag/Makefile @@ -0,0 +1,151 @@ +# ============================================================================ +# Aggregator Latency Monitor with Grafana Dashboard +# ============================================================================ + +BINARY_NAME = latency_monitor +BINARY_PATH = bin/monitor +GO_FILES = ./cmd/script + +.PHONY: help +help: + @echo "Aggregator Latency Monitor - Grafana Dashboard" + @echo "==============================================" + @echo "" + @echo "Commands:" + @echo " make run - Start everything (Grafana + All monitors in background)" + @echo " make pulse - Start Mobula Pulse monitor only (foreground)" + @echo " make stop - Stop all services" + @echo " make logs - Follow monitor logs" + @echo " make status - Show status of all services" + @echo " make build - Build Go binary" + @echo " make clean - Stop services and remove binaries/logs" + @echo " make destroy - Remove everything including volumes (asks confirmation)" + @echo "" + @echo "Dashboard Access:" + @echo " Grafana: http://localhost:3000 (admin/admin)" + @echo " Prometheus: http://localhost:9090" + @echo " Metrics: http://localhost:2112/metrics" + @echo "" + +.PHONY: deps +deps: + @echo "📦 Downloading dependencies..." + @go mod tidy + @go mod download + @echo "✓ Dependencies ready" + @echo "" + +.PHONY: build +build: deps + @echo "🔨 Building $(BINARY_NAME)..." + @mkdir -p bin + @go build -o $(BINARY_PATH) $(GO_FILES) + @echo "✓ Build complete: $(BINARY_PATH)" + @echo "" + +.PHONY: start-grafana +start-grafana: + @echo "📊 Starting Grafana + Prometheus stack..." + @docker-compose up -d + @echo "✓ Grafana stack running" + @echo " → Grafana: http://localhost:3000 (admin/admin)" + @echo " → Prometheus: http://localhost:9090" + @echo "" + +.PHONY: run +run: build start-grafana + @echo "🚀 Starting Aggregator Latency Monitors in background..." + @echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + @echo " → Starting monitors (CoinGecko, Mobula Pulse, Mobula Trade, Codex)..." + @./$(BINARY_PATH) > monitor.log 2>&1 & echo $$! > monitor.pid + @sleep 2 + @if [ -f monitor.pid ]; then \ + echo " ✓ Monitors started (PID: $$(cat monitor.pid))"; \ + else \ + echo " ❌ Failed to start monitors"; \ + fi + @echo "" + @echo "✓ All monitors running" + @echo "✓ Monitoring: CoinGecko, Mobula (Pulse + Trade), Codex" + @echo "✓ Chains: Solana, BNB, Base, Monad" + @echo "✓ Metrics: http://localhost:2112/metrics" + @echo "✓ Logs: make logs" + @echo "✓ Stop: make stop" + @echo "" + +.PHONY: down +down: + @echo "🛑 Stopping services..." + @if [ -f monitor.pid ]; then \ + echo " → Stopping monitors (PID: $$(cat monitor.pid))..."; \ + kill $$(cat monitor.pid) 2>/dev/null || true; \ + rm -f monitor.pid; \ + fi + @echo " → Killing all monitor processes..." + @pkill -9 -f "bin/monitor" 2>/dev/null || true + @echo " → Stopping Docker containers..." + @docker-compose down 2>/dev/null || true + @docker stop prometheus grafana 2>/dev/null || true + @docker rm prometheus grafana 2>/dev/null || true + @echo "✓ All services stopped (volumes preserved)" + +.PHONY: stop +stop: down + +.PHONY: clean +clean: down + @echo "🧹 Cleaning binaries and logs..." + @rm -f $(BINARY_PATH) $(BINARY_NAME) monitor.log + @echo "✓ Clean complete" + +.PHONY: logs +logs: + @if [ -f monitor.log ]; then \ + tail -f monitor.log; \ + else \ + echo "❌ No log file found. Is the monitor running?"; \ + echo " Run 'make run' first"; \ + fi + +.PHONY: status +status: + @echo "📊 Service Status:" + @echo "" + @if [ -f monitor.pid ] && kill -0 $$(cat monitor.pid) 2>/dev/null; then \ + echo " ✓ Monitors: Running (PID: $$(cat monitor.pid))"; \ + else \ + echo " ✗ Monitors: Stopped"; \ + fi + @if docker-compose ps | grep -q "Up"; then \ + echo " ✓ Grafana: Running (http://localhost:3000)"; \ + echo " ✓ Prometheus: Running (http://localhost:9090)"; \ + else \ + echo " ✗ Grafana: Stopped"; \ + echo " ✗ Prometheus: Stopped"; \ + fi + @echo "" + +.PHONY: destroy +destroy: + @echo "⚠️ WARNING: This will remove all containers, volumes, and binaries!" + @echo "⚠️ All Grafana dashboards and Prometheus data will be lost!" + @echo "" + @read -p "Are you sure? [y/N] " -n 1 -r; \ + echo; \ + if [[ $$REPLY =~ ^[Yy]$$ ]]; then \ + echo "🗑️ Destroying everything..."; \ + if [ -f monitor.pid ]; then kill $$(cat monitor.pid) 2>/dev/null || true; rm -f monitor.pid; fi; \ + pkill -9 -f "bin/monitor" 2>/dev/null || true; \ + docker-compose down -v 2>/dev/null || true; \ + rm -f $(BINARY_PATH) $(BINARY_NAME) monitor.log monitor.pid; \ + echo "✓ Everything destroyed"; \ + else \ + echo "❌ Cancelled"; \ + fi + +.PHONY: pulse +pulse: + @echo "🚀 Starting Mobula Pulse V2 Monitor..." + @go run ./cmd/pulse/*.go + +.DEFAULT_GOAL := help diff --git a/harnesses/aggregator-head-lag/README.md b/harnesses/aggregator-head-lag/README.md index 59c9ded9..77575161 100644 --- a/harnesses/aggregator-head-lag/README.md +++ b/harnesses/aggregator-head-lag/README.md @@ -1,57 +1,147 @@ # Harness · aggregator-head-lag -> Watches reference pools across multiple chains and records the gap between an on-chain event and its appearance on each aggregator's data feed. +> Real-time monitor that produces the `head_lag_seconds` metric consumed by [`benchmarks/aggregator-head-lag.yml`](../../benchmarks/aggregator-head-lag.yml). **Bench**: [№ 001 · Aggregator Head Lag](../../benchmarks/aggregator-head-lag.yml) -## What it measures +## How it works -- **Head lag** — wall-clock time between an on-chain Swap event and the same event appearing on the aggregator's WebSocket / GraphQL / REST feed. -- **Presence** — share of expected sampling slots where a value was actually emitted (proxy for feed reachability). +The monitor connects to each aggregator's WebSocket / REST feed and measures latency by comparing: -## Aggregators +- The on-chain timestamp of a trade event (from the event payload) +- The wall-clock time at which the aggregator emitted the event -`mobula` · `codex` · `geckoterminal` +The delta is exported as a Prometheus gauge (`head_lag_seconds`), labelled by aggregator, chain and region. Failures surface as `head_lag_errors_total` counters. REST API latency, quote API latency and metadata coverage are recorded in parallel from the same process. -## Inputs +**Tracked aggregators**: GeckoTerminal · Mobula · Codex +**Supported chains**: Solana · Ethereum · BNB Chain · Base -- Reference: archive nodes per chain, peered to multiple sources, validated against block hash. -- Chains: Base, BNB Chain, Solana. -- Cadence: sampled every 15 seconds per aggregator × chain × region. -- Regions: `us-east`, `eu-west`, `sgp`. +## Metrics produced -## Metrics emitted +``` +head_lag_seconds{aggregator, chain, region} gauge +head_lag_milliseconds{aggregator, chain, region} gauge (× 1000 of seconds) +head_lag_errors_total{aggregator, chain, region, error_type} counter +mobula_head_lag_detailed_seconds{aggregator, chain, region, pool} gauge +mobula_processing_lag_seconds{...} gauge +mobula_network_lag_seconds{...} gauge +aggregator_head_block{aggregator, chain, region} gauge +blockchain_head_block{chain, region} gauge +rest_api_latency_milliseconds_{bucket,sum,count}{...} histogram +quote_api_latency_milliseconds_{bucket,sum,count}{...} histogram +metadata_coverage_total / metadata_coverage_success{...} counter +``` + +The Prometheus URL declared in the YAML spec is what the OpenChainBench site queries — the harness pushes here and the site reads from the same instance. + +## Run locally +Prerequisites: Go 1.24+, Docker, API keys for the aggregators you want to track. + +```bash +cp .env.example .env +# Fill in the keys (any missing key disables that aggregator's monitor) +docker-compose up -d ``` -head_lag_seconds{aggregator, chain, region} gauge -head_lag_milliseconds{aggregator, chain, region} gauge (× 1000 of seconds) -head_lag_errors_total{aggregator, chain, region, error_type} counter -mobula_head_lag_detailed_seconds{aggregator, chain, region, pool_address} gauge -mobula_processing_lag_seconds{...} gauge (server-side breakdown) -mobula_network_lag_seconds{...} gauge (network breakdown) -aggregator_head_block{aggregator, chain, region} gauge -blockchain_head_block{chain, region} gauge (canonical tip) + +Access: + +- Grafana: (admin / admin) +- Prometheus: +- Metrics endpoint: + +If the YAML spec at `benchmarks/aggregator-head-lag.yml` is repointed at `http://localhost:9090`, the OpenChainBench site rendered with `pnpm dev` will display your local data instead of production. + +## Run on Railway + +The deployed services live in this repo at `harnesses/aggregator-head-lag/`. To deploy: + +1. Create three services on Railway from this repo, root directory `harnesses/aggregator-head-lag/`: + - **monitor** — uses `Dockerfile` + - **prometheus** — root `harnesses/aggregator-head-lag/prometheus`, uses `Dockerfile` + - **alertmanager** — Docker image `prom/alertmanager:latest`, mounts `monitoring/alertmanager.yml` +2. Set environment variables on the monitor service (see `.env.example`). +3. Point the corresponding YAML's `prom_url` at the public Prometheus URL. + +## Environment variables + +| Var | Description | Required | +| --- | --- | --- | +| `MOBULA_API_KEY` | Mobula API key | optional | +| `COINGECKO_API_KEY` | CoinGecko Pro key (for GeckoTerminal feed) | optional | +| `DEFINED_SESSION_COOKIE` | Defined.fi session cookie (for Codex). Auto-scraped if absent. | optional | +| `MOBULA_WS_URL` | Override the Mobula WS endpoint | optional | +| `MONITOR_REGION` | Label written on every metric (e.g. `us-east`) | recommended | +| `GF_SECURITY_ADMIN_PASSWORD` | Grafana admin password (Docker only) | recommended | + +A monitor with no key for a given aggregator is skipped cleanly — the harness will run for whatever providers it can authenticate with. + +## Project layout + ``` +cmd/script/ Single Go binary: all monitors + Prometheus exporter + ├── main.go Entrypoint, supervises goroutines per aggregator + ├── config.go Env-var loader + ├── metrics.go Prometheus metric definitions + registration + ├── head_lag_monitor.go WebSocket monitor: on-chain ↔ feed timestamp delta + ├── mobula_*.go Mobula REST + WS monitors + ├── codex_*.go Codex REST monitor + ├── geckoterminal_monitor.go + ├── quote_api_monitor.go Mobula swap quoting latency + ├── metadata_coverage_monitor.go + ├── proxy.go HTTP transport + retries + └── log_buffer.go In-memory ring buffer (last N log lines) + +monitoring/ Local docker-compose stack + ├── prometheus.yml + ├── alert_rules.yml + ├── alertmanager.yml + └── grafana/ Datasource + dashboards provisioning -## Env vars +prometheus/ Production Prometheus (Railway service) + ├── Dockerfile + ├── prometheus.yml + ├── prometheus.staging.yml + └── alert_rules.yml + +grafana/ Production Grafana (Railway service) + ├── Dockerfile + ├── grafana-entrypoint.sh + ├── dashboards/ + └── provisioning/ + +Dockerfile Monitor binary (multi-stage Go build) +docker-compose.yml Local dev stack +Makefile `make run`, `make logs`, `make stop`, `make clean` +``` -| Var | Required | Notes | -| --------------------- | -------- | ------------------------------------ | -| `MOBULA_WS_URL` | yes | Mobula WebSocket endpoint | -| `CODEX_API_KEY` | yes | https://docs.codex.io | -| `GECKOTERMINAL_KEY` | optional | Public endpoints work without it | -| `ETH_RPC_URL` | yes | Archive node for canonical reference | -| `SOL_RPC_URL` | yes | | -| `BSC_RPC_URL` | yes | | -| `PROMETHEUS_PUSH_URL` | yes | Prometheus push gateway | -| `MONITOR_REGION` | yes | one of `us-east`, `eu-west`, `sgp` | +## Adding an aggregator -## Running locally +1. Create `cmd/script/_monitor.go` mirroring an existing file (e.g. `geckoterminal_monitor.go`). +2. Implement the WebSocket / REST loop and call `RecordLatency("", chain, latencyMs)` (or the appropriate metric registrar in `metrics.go`). +3. Add the API key field to `Config` in `config.go` and to `.env.example`. +4. Start the monitor goroutine in `main.go`. +5. Update the YAML spec at `benchmarks/aggregator-head-lag.yml` to reference the new aggregator under `providers:` and add its query under `prometheus.providers.`. +6. Add a Grafana panel if you want it visualised internally. -> Implementation TBD. This README describes the contract; the runner code -> lives in the team's separate harness repo and pushes metrics to the -> Prometheus instance referenced in the spec YAML. +## Troubleshooting ```bash -make run REGION=eu-west +# Are metrics being produced? +curl http://localhost:2112/metrics | grep head_lag_seconds + +# Is Prometheus scraping? +open http://localhost:9090/targets # all targets should be UP + +# Restart the stack +docker-compose down && docker-compose up -d --build + +# Tail monitor logs +docker-compose logs -f monitor ``` + +If a specific aggregator silently emits nothing: most often the API key is missing or rate-limited. Check the monitor logs for `[AGGREGATOR][skip]` lines. + +## License + +MIT — same as the rest of OpenChainBench. diff --git a/harnesses/aggregator-head-lag/assets/logo.png b/harnesses/aggregator-head-lag/assets/logo.png new file mode 100644 index 00000000..e40972bd Binary files /dev/null and b/harnesses/aggregator-head-lag/assets/logo.png differ diff --git a/harnesses/aggregator-head-lag/cmd/script/cleanup.go b/harnesses/aggregator-head-lag/cmd/script/cleanup.go new file mode 100644 index 00000000..fb1d5883 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/cleanup.go @@ -0,0 +1,137 @@ +package main + +import ( + "fmt" + "log" + "net/http" + "os" + "path/filepath" + "time" +) + +// setupCleanupEndpoint adds an admin endpoint for selective data cleanup +func setupCleanupEndpoint(mux *http.ServeMux) { + adminToken := os.Getenv("ADMIN_CLEANUP_TOKEN") + if adminToken == "" { + log.Println("[CLEANUP] Warning: ADMIN_CLEANUP_TOKEN not set - cleanup endpoint will be disabled") + return + } + + mux.HandleFunc("/admin/cleanup", func(w http.ResponseWriter, r *http.Request) { + // Only allow DELETE method + if r.Method != "DELETE" { + http.Error(w, "Method not allowed - use DELETE", http.StatusMethodNotAllowed) + return + } + + // Check authorization + authHeader := r.Header.Get("Authorization") + expectedAuth := "Bearer " + adminToken + if authHeader != expectedAuth { + log.Printf("[CLEANUP] Unauthorized cleanup attempt from %s", r.RemoteAddr) + http.Error(w, "Unauthorized", http.StatusUnauthorized) + return + } + + // Parse the 'before' timestamp parameter + beforeStr := r.URL.Query().Get("before") + if beforeStr == "" { + http.Error(w, "Missing 'before' parameter (use RFC3339 format: 2026-04-10T00:00:00Z)", http.StatusBadRequest) + return + } + + before, err := time.Parse(time.RFC3339, beforeStr) + if err != nil { + http.Error(w, fmt.Sprintf("Invalid timestamp format: %v (use RFC3339: 2026-04-10T00:00:00Z)", err), http.StatusBadRequest) + return + } + + // Get data paths from environment or use defaults + prometheusPath := os.Getenv("PROMETHEUS_DATA_PATH") + if prometheusPath == "" { + prometheusPath = "/data/prometheus" + } + + grafanaPath := os.Getenv("GRAFANA_DATA_PATH") + if grafanaPath == "" { + grafanaPath = "/data/grafana" + } + + log.Printf("[CLEANUP] Starting cleanup: deleting data before %s", before.Format(time.RFC3339)) + log.Printf("[CLEANUP] Prometheus path: %s", prometheusPath) + log.Printf("[CLEANUP] Grafana path: %s", grafanaPath) + + totalDeleted := 0 + + // Cleanup Prometheus data + if _, err := os.Stat(prometheusPath); err == nil { + deleted, err := cleanupDirectory(prometheusPath, before) + if err != nil { + log.Printf("[CLEANUP] Error cleaning Prometheus data: %v", err) + http.Error(w, fmt.Sprintf("Prometheus cleanup failed: %v", err), http.StatusInternalServerError) + return + } + totalDeleted += deleted + log.Printf("[CLEANUP] Deleted %d items from Prometheus", deleted) + } else { + log.Printf("[CLEANUP] Prometheus path not found: %s", prometheusPath) + } + + // Cleanup Grafana data + if _, err := os.Stat(grafanaPath); err == nil { + deleted, err := cleanupDirectory(grafanaPath, before) + if err != nil { + log.Printf("[CLEANUP] Error cleaning Grafana data: %v", err) + http.Error(w, fmt.Sprintf("Grafana cleanup failed: %v", err), http.StatusInternalServerError) + return + } + totalDeleted += deleted + log.Printf("[CLEANUP] Deleted %d items from Grafana", deleted) + } else { + log.Printf("[CLEANUP] Grafana path not found: %s", grafanaPath) + } + + response := fmt.Sprintf("✅ Cleanup complete\n\nDeleted %d files/directories before %s\n\nPaths cleaned:\n- %s\n- %s\n", + totalDeleted, before.Format(time.RFC3339), prometheusPath, grafanaPath) + + log.Printf("[CLEANUP] Cleanup complete: %d items deleted", totalDeleted) + w.Header().Set("Content-Type", "text/plain") + w.WriteHeader(http.StatusOK) + fmt.Fprint(w, response) + }) + + log.Printf("[CLEANUP] Admin cleanup endpoint enabled at /admin/cleanup") +} + +// cleanupDirectory removes files and directories older than the specified time +func cleanupDirectory(dir string, before time.Time) (int, error) { + deleted := 0 + + entries, err := os.ReadDir(dir) + if err != nil { + return 0, fmt.Errorf("failed to read directory: %w", err) + } + + for _, entry := range entries { + path := filepath.Join(dir, entry.Name()) + info, err := entry.Info() + if err != nil { + log.Printf("[CLEANUP] Warning: failed to get info for %s: %v", path, err) + continue + } + + // Check if modification time is before the cutoff + if info.ModTime().Before(before) { + log.Printf("[CLEANUP] Deleting: %s (modified: %s)", path, info.ModTime().Format(time.RFC3339)) + + if err := os.RemoveAll(path); err != nil { + log.Printf("[CLEANUP] Warning: failed to delete %s: %v", path, err) + continue + } + + deleted++ + } + } + + return deleted, nil +} diff --git a/harnesses/aggregator-head-lag/cmd/script/codex_rest_monitor.go b/harnesses/aggregator-head-lag/cmd/script/codex_rest_monitor.go new file mode 100644 index 00000000..f75f0c00 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/codex_rest_monitor.go @@ -0,0 +1,245 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "log" + "net/http" + "strings" + "sync" + "time" +) + +const ( + codexRESTBaseURL = "https://graph.codex.io/graphql" +) + +// Track rate limiting to avoid spamming JWT generation +var ( + codexRESTRateLimitedUntil time.Time + codexRESTRateLimitMutex sync.RWMutex +) + +// Chains for REST monitoring - aligned with all monitors +var codexRESTChains = []struct { + networkID int + chainName string + poolAddress string +}{ + {1399811149, "solana", "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm"}, // SOL/USDC Raydium + {8453, "base", "0x4c36388be6f416a29c8d8eee81c771ce6be14b18"}, // WETH/USDC Base + {56, "bnb", "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16"}, // WBNB/BUSD PancakeSwap +} + +type CodexGraphQLRequest struct { + Query string `json:"query"` + Variables map[string]interface{} `json:"variables"` +} + +type CodexGraphQLResponse struct { + Data map[string]interface{} `json:"data"` + Errors []struct { + Message string `json:"message"` + } `json:"errors"` +} + +// callCodexGraphQLAPI makes a GraphQL query to Codex API +func callCodexGraphQLAPI(apiKey string, poolAddress string, networkID int, chainName string) (float64, int, error) { + // Create HTTP client with proxy support (forces new connection for IP rotation) + client := getProxyHTTPClient() + + // Build GraphQL query - filterPairs is reliable and works for all chains + // This query filters pairs by network and returns one result to measure latency + query := ` + query FilterPairs($networkId: [Int!]) { + filterPairs(filters: { network: $networkId }, limit: 1) { + results { + pair { + address + token0 + token1 + } + } + } + } + ` + + // Build request body with variables + reqBody := CodexGraphQLRequest{ + Query: query, + Variables: map[string]interface{}{ + "networkId": []int{networkID}, + }, + } + + bodyBytes, err := json.Marshal(reqBody) + if err != nil { + return 0, 0, fmt.Errorf("failed to marshal request: %w", err) + } + + // Build request + req, err := http.NewRequest("POST", codexRESTBaseURL, bytes.NewBuffer(bodyBytes)) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + + // Add headers + req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", apiKey)) + req.Header.Set("Content-Type", "application/json") + + // Measure latency + startTime := time.Now() + resp, err := client.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + // Read response body + body, _ := io.ReadAll(resp.Body) + + // Try to parse response + var graphqlResp CodexGraphQLResponse + if err := json.Unmarshal(body, &graphqlResp); err != nil { + log.Printf("[CODEX-REST][%s] Response parse warning: %v (status: %d)", chainName, err, resp.StatusCode) + } + + // Check for GraphQL errors + if len(graphqlResp.Errors) > 0 { + log.Printf("[CODEX-REST][%s] GraphQL errors: %v", chainName, graphqlResp.Errors[0].Message) + + // Check if it's an authentication error + if graphqlResp.Errors[0].Message == "User is not authenticated" { + return latencyMs, resp.StatusCode, fmt.Errorf("authentication error: %s", graphqlResp.Errors[0].Message) + } + } + + return latencyMs, resp.StatusCode, nil +} + +// monitorCodexREST continuously monitors Codex GraphQL API latency +func monitorCodexREST(config *Config, stopChan <-chan struct{}) { + fmt.Println("Starting Codex REST API monitor...") + fmt.Printf(" Monitoring %d chains with 20s interval\n", len(codexRESTChains)) + fmt.Printf(" Endpoint: POST /graphql (GraphQL)\n") + fmt.Println() + + if config.DefinedSessionCookie == "" { + fmt.Println("DEFINED_SESSION_COOKIE not set in .env file. Skipping Codex REST monitor.") + return + } + + // Create ticker for 20 second intervals + ticker := time.NewTicker(20 * time.Second) + defer ticker.Stop() + + // Run once immediately + performCodexRESTChecks(config) + + // Then run every 20 seconds + for { + select { + case <-stopChan: + fmt.Println("Codex REST monitor stopped") + return + case <-ticker.C: + performCodexRESTChecks(config) + } + } +} + +// performCodexRESTChecks performs GraphQL API calls to all chains +func performCodexRESTChecks(config *Config) { + timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") + + // Check if we're still rate limited + codexRESTRateLimitMutex.RLock() + if time.Now().Before(codexRESTRateLimitedUntil) { + waitTime := time.Until(codexRESTRateLimitedUntil) + codexRESTRateLimitMutex.RUnlock() + fmt.Printf("[CODEX-REST] Skipping checks - rate limited for another %v\n", waitTime.Round(time.Second)) + return + } + codexRESTRateLimitMutex.RUnlock() + + // Generate JWT token from session cookie + jwtToken, err := GetDefinedJWTToken(config.DefinedSessionCookie) + if err != nil { + fmt.Printf("[CODEX-REST] Failed to get JWT token: %v\n", err) + + // If rate limited, stop trying for 10 minutes + if strings.Contains(err.Error(), "rate limited (429)") || strings.Contains(err.Error(), "too many token requests") { + codexRESTRateLimitMutex.Lock() + codexRESTRateLimitedUntil = time.Now().Add(10 * time.Minute) + codexRESTRateLimitMutex.Unlock() + fmt.Printf("[CODEX-REST] ⏱️ Rate limited - pausing checks for 10 minutes\n") + } + return + } + + for _, chain := range codexRESTChains { + latencyMs, statusCode, err := callCodexGraphQLAPI( + jwtToken, + chain.poolAddress, + chain.networkID, + chain.chainName, + ) + + if err != nil { + // Check if it's an auth error + if err.Error() == "authentication error: User is not authenticated" { + fmt.Println("[CODEX-REST] Authentication error - invalidating JWT cache") + InvalidateTokenCache() + } + + // Record error + errorType := "request_error" + if statusCode >= 500 { + errorType = "server_error" + } else if statusCode >= 400 { + errorType = "client_error" + } else if statusCode == 0 { + errorType = "timeout_error" + } + + RecordRESTError("codex", "graphql", chain.chainName, errorType, config.MonitorRegion) + + fmt.Printf("[CODEX-REST][%s][%s] ERROR | Latency: %.0fms | Status: %d | Error: %v\n", + timestamp, + chain.chainName, + latencyMs, + statusCode, + err, + ) + continue + } + + // Record successful latency measurement + RecordRESTLatency("codex", "graphql", chain.chainName, latencyMs, statusCode, config.MonitorRegion) + + // Log the result + statusEmoji := "✓" + if statusCode >= 400 { + statusEmoji = "✗" + } else if statusCode >= 300 { + statusEmoji = "⚠" + } + + fmt.Printf("[CODEX-REST][%s][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, + chain.chainName, + statusEmoji, + latencyMs, + statusCode, + ) + } +} + +// runCodexRESTMonitor is the entry point for the Codex REST monitor +func runCodexRESTMonitor(config *Config, stopChan <-chan struct{}) { + monitorCodexREST(config, stopChan) +} diff --git a/harnesses/aggregator-head-lag/cmd/script/config.go b/harnesses/aggregator-head-lag/cmd/script/config.go new file mode 100644 index 00000000..a3fd5114 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/config.go @@ -0,0 +1,85 @@ +package main + +import ( + "bufio" + "fmt" + "os" + "strings" +) + +type Config struct { + CoinGeckoAPIKey string + MobulaAPIKey string + DefinedSessionCookie string + MonitorRegion string // Deployment region: us-west, us-east, singapore, etc. + MobulaWSURL string // Mobula fast-trade WebSocket endpoint (allows staging to use EU-specific cluster) +} + +func loadEnv() (*Config, error) { + config := &Config{} + + // First, try to load from environment variables (for production/Railway) + config.CoinGeckoAPIKey = strings.TrimSpace(os.Getenv("COINGECKO_API_KEY")) + config.MobulaAPIKey = strings.TrimSpace(os.Getenv("MOBULA_API_KEY")) + config.DefinedSessionCookie = strings.TrimSpace(os.Getenv("DEFINED_SESSION_COOKIE")) + config.MonitorRegion = strings.TrimSpace(os.Getenv("MONITOR_REGION")) + config.MobulaWSURL = strings.TrimSpace(os.Getenv("MOBULA_WS_URL")) + + // Default to "unknown" if not set + if config.MonitorRegion == "" { + config.MonitorRegion = "unknown" + } + + // Default Mobula WS endpoint (global). Staging can override to wss://api-prod-eu.mobula.io + if config.MobulaWSURL == "" { + config.MobulaWSURL = "wss://api.mobula.io" + } + + // If all env vars are set, return early (production mode) + if config.CoinGeckoAPIKey != "" || config.MobulaAPIKey != "" || config.DefinedSessionCookie != "" { + return config, nil + } + + // Otherwise, try to load from .env file (for local development) + file, err := os.Open(".env") + if err != nil { + // If no .env file and no env vars, that's OK - services will just be skipped + return config, nil + } + defer file.Close() + + scanner := bufio.NewScanner(file) + for scanner.Scan() { + line := strings.TrimSpace(scanner.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + + parts := strings.SplitN(line, "=", 2) + if len(parts) != 2 { + continue + } + + key, value := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1]) + switch key { + case "COINGECKO_API_KEY": + if config.CoinGeckoAPIKey == "" { + config.CoinGeckoAPIKey = value + } + case "MOBULA_API_KEY": + if config.MobulaAPIKey == "" { + config.MobulaAPIKey = value + } + case "DEFINED_SESSION_COOKIE": + if config.DefinedSessionCookie == "" { + config.DefinedSessionCookie = value + } + } + } + + if err := scanner.Err(); err != nil { + return nil, fmt.Errorf("error reading .env file: %w", err) + } + + return config, nil +} diff --git a/harnesses/aggregator-head-lag/cmd/script/defined_auth.go b/harnesses/aggregator-head-lag/cmd/script/defined_auth.go new file mode 100644 index 00000000..716f05f0 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/defined_auth.go @@ -0,0 +1,197 @@ +package main + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "sync" + "time" +) + +type DefinedTokenResponse struct { + Data struct { + CreateApiTokens []struct { + Token string `json:"token"` + } `json:"createApiTokens"` + } `json:"data"` +} + +// JWT token cache to avoid rate limiting +type tokenCache struct { + mu sync.RWMutex + token string + expiresAt time.Time + lastRefresh time.Time +} + +var globalTokenCache = &tokenCache{} + +// decodeJWTExpiration extracts the expiration time from a JWT token +func decodeJWTExpiration(token string) (time.Time, error) { + parts := strings.Split(token, ".") + if len(parts) != 3 { + return time.Time{}, fmt.Errorf("invalid JWT format") + } + + // Decode payload (second part) + payload, err := base64.RawURLEncoding.DecodeString(parts[1]) + if err != nil { + return time.Time{}, fmt.Errorf("failed to decode JWT payload: %w", err) + } + + var claims struct { + Exp int64 `json:"exp"` + } + if err := json.Unmarshal(payload, &claims); err != nil { + return time.Time{}, fmt.Errorf("failed to unmarshal JWT claims: %w", err) + } + + if claims.Exp == 0 { + return time.Time{}, fmt.Errorf("no expiration in JWT") + } + + return time.Unix(claims.Exp, 0), nil +} + +// GetDefinedJWTToken returns a cached JWT token or generates a new one if expired +func GetDefinedJWTToken(sessionCookie string) (string, error) { + globalTokenCache.mu.RLock() + + // Check if we have a valid cached token + // Renew 1 hour before expiration to be safe + if globalTokenCache.token != "" && time.Now().Before(globalTokenCache.expiresAt.Add(-1*time.Hour)) { + token := globalTokenCache.token + globalTokenCache.mu.RUnlock() + return token, nil + } + globalTokenCache.mu.RUnlock() + + // Need to refresh token + globalTokenCache.mu.Lock() + defer globalTokenCache.mu.Unlock() + + // Double-check after acquiring write lock + if globalTokenCache.token != "" && time.Now().Before(globalTokenCache.expiresAt.Add(-1*time.Hour)) { + return globalTokenCache.token, nil + } + + // Generate new token + token, err := generateDefinedJWTToken(sessionCookie) + if err != nil { + return "", err + } + + // Decode expiration from token + expiresAt, err := decodeJWTExpiration(token) + if err != nil { + fmt.Printf("[DEFINED-AUTH] Warning: Could not decode token expiration: %v. Will cache for 24h.\n", err) + expiresAt = time.Now().Add(24 * time.Hour) + } + + // Cache the token + globalTokenCache.token = token + globalTokenCache.expiresAt = expiresAt + globalTokenCache.lastRefresh = time.Now() + + timeUntilExpiry := time.Until(expiresAt) + fmt.Printf("[DEFINED-AUTH] JWT token refreshed. Expires in %.1fh (at %s)\n", + timeUntilExpiry.Hours(), expiresAt.Format("2006-01-02 15:04:05")) + + return token, nil +} + +// generateDefinedJWTToken generates a new JWT token from Defined.fi session cookie +func generateDefinedJWTToken(sessionCookie string) (string, error) { + fmt.Println("[DEFINED-AUTH] Generating new JWT token from Defined.fi (local)...") + fmt.Println("[DEFINED-AUTH] Creating new HTTP client with fresh TCP connection (no keepalive)") + + // Create a new HTTP client with fresh connection for each request. + // CRITICAL: route through HTTP_PROXY/HTTPS_PROXY (webshare rotating proxy) + // so each JWT mint hits a fresh IP. Direct from the container IP gets us + // stuck on Vercel's bot ban (429 loop) when the container restarts often. + transport := &http.Transport{ + DisableKeepAlives: true, + MaxIdleConnsPerHost: 0, + Proxy: http.ProxyFromEnvironment, + } + client := &http.Client{ + Timeout: 10 * time.Second, + Transport: transport, + } + + reqBody := map[string]interface{}{ + "operationName": "CreateApiToken", + "query": "mutation CreateApiToken { createApiTokens(input: { count: 1 }) { token } }", + "variables": map[string]interface{}{}, + } + + bodyBytes, _ := json.Marshal(reqBody) + req, _ := http.NewRequest("POST", "https://www.defined.fi/api", bytes.NewBuffer(bodyBytes)) + + req.Header.Set("Accept", "application/json") + req.Header.Set("Accept-Language", "en-US,en;q=0.9") + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Origin", "https://www.defined.fi") + req.Header.Set("Referer", "https://www.defined.fi/") + req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36") + req.Header.Set("sec-ch-ua", `"Not_A Brand";v="8", "Chromium";v="131", "Google Chrome";v="131"`) + req.Header.Set("sec-ch-ua-mobile", "?0") + req.Header.Set("sec-ch-ua-platform", `"macOS"`) + req.Header.Set("sec-fetch-dest", "empty") + req.Header.Set("sec-fetch-mode", "cors") + req.Header.Set("sec-fetch-site", "same-origin") + req.AddCookie(&http.Cookie{Name: "session", Value: sessionCookie}) + + fmt.Println("[DEFINED-AUTH] Sending POST request to https://www.defined.fi/api...") + resp, err := client.Do(req) + if err != nil { + fmt.Printf("[DEFINED-AUTH] ❌ Request failed: %v\n", err) + return "", fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + respBody, _ := io.ReadAll(resp.Body) + fmt.Printf("[DEFINED-AUTH] Response status: %d\n", resp.StatusCode) + + if resp.StatusCode == 429 { + // Parse retry-after header if available + retryAfter := resp.Header.Get("Retry-After") + fmt.Printf("[DEFINED-AUTH] ⚠ Rate limited! Retry-After: %s\n", retryAfter) + if retryAfter != "" { + return "", fmt.Errorf("rate limited (429), retry after: %s", retryAfter) + } + return "", fmt.Errorf("rate limited (429), too many token requests - will retry later") + } + + if resp.StatusCode != 200 { + fmt.Printf("[DEFINED-AUTH] ❌ Unexpected status %d: %s\n", resp.StatusCode, string(respBody[:min(len(respBody), 100)])) + return "", fmt.Errorf("unexpected status %d: %s", resp.StatusCode, string(respBody[:min(len(respBody), 100)])) + } + + var tokenResp DefinedTokenResponse + if err := json.Unmarshal(respBody, &tokenResp); err != nil { + fmt.Printf("[DEFINED-AUTH] ❌ Failed to decode response: %v\n", err) + return "", fmt.Errorf("failed to decode: %w", err) + } + + if len(tokenResp.Data.CreateApiTokens) == 0 { + fmt.Println("[DEFINED-AUTH] ❌ No token in response") + return "", fmt.Errorf("no token returned") + } + + fmt.Printf("[DEFINED-AUTH] ✅ JWT token generated successfully (length: %d)\n", len(tokenResp.Data.CreateApiTokens[0].Token)) + return tokenResp.Data.CreateApiTokens[0].Token, nil +} + +// InvalidateTokenCache forces a token refresh on next request +func InvalidateTokenCache() { + globalTokenCache.mu.Lock() + defer globalTokenCache.mu.Unlock() + globalTokenCache.token = "" + globalTokenCache.expiresAt = time.Time{} + fmt.Println("[DEFINED-AUTH] Token cache invalidated - will refresh on next request") +} diff --git a/harnesses/aggregator-head-lag/cmd/script/geckoterminal_monitor.go b/harnesses/aggregator-head-lag/cmd/script/geckoterminal_monitor.go new file mode 100644 index 00000000..c19e7edd --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/geckoterminal_monitor.go @@ -0,0 +1,263 @@ +package main + +import ( + "encoding/json" + "fmt" + "log" + "sync" + "time" + + "github.com/gorilla/websocket" +) + +// ============================================================================ +// GeckoTerminal WebSocket Monitor +// ============================================================================ + +const ( + geckoWSURL = "wss://cables.geckoterminal.com/cable" + geckoOrigin = "https://www.geckoterminal.com" + geckoUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" +) + +// GeckoTerminal pools (pool_id extracted via reverse engineering) +var geckoTerminalPools = []struct { + Name string + Network string + PoolID string + Chain string +}{ + { + Name: "SOL/USDC Raydium", + Network: "solana", + PoolID: "162715608", + Chain: "solana", + }, + { + Name: "WETH/USDC Base", + Network: "base", + PoolID: "162840764", + Chain: "base", + }, + { + Name: "WBNB/BUSD PancakeSwap", + Network: "bsc", + PoolID: "24", + Chain: "bnb", + }, +} + +// ActionCable message structures +type GeckoActionCableMessage struct { + Type string `json:"type,omitempty"` + Command string `json:"command,omitempty"` + Identifier string `json:"identifier,omitempty"` + Message json.RawMessage `json:"message,omitempty"` +} + +type GeckoChannelIdentifier struct { + Channel string `json:"channel"` + PoolID string `json:"pool_id,omitempty"` +} + +// Swap data from SwapChannel +type GeckoSwapData struct { + Data struct { + BlockTimestamp int64 `json:"block_timestamp"` // On-chain timestamp (ms) + TxHash string `json:"tx_hash"` + // Other fields available but not needed for head lag + } `json:"data"` + Type string `json:"type"` // "newSwap" +} + +func runGeckoTerminalHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { + defer wg.Done() + + fmt.Println("[HEAD-LAG][GECKO] Starting WebSocket monitor...") + + reconnectDelay := 5 * time.Second + maxReconnectDelay := 60 * time.Second + + for { + select { + case <-stopChan: + fmt.Println("[HEAD-LAG][GECKO] Monitor stopped") + return + default: + err := connectAndMonitorGecko(config, stopChan) + if err != nil { + log.Printf("[HEAD-LAG][GECKO] Connection error: %v. Reconnecting in %v...", err, reconnectDelay) + + select { + case <-stopChan: + return + case <-time.After(reconnectDelay): + reconnectDelay = reconnectDelay * 2 + if reconnectDelay > maxReconnectDelay { + reconnectDelay = maxReconnectDelay + } + } + } else { + reconnectDelay = 5 * time.Second + } + } + } +} + +func connectAndMonitorGecko(config *Config, stopChan <-chan struct{}) error { + headers := map[string][]string{ + "Origin": {geckoOrigin}, + "User-Agent": {geckoUserAgent}, + } + + conn, _, err := getProxyDialer().Dial(geckoWSURL, headers) + if err != nil { + return fmt.Errorf("dial failed: %w", err) + } + defer conn.Close() + + // Channel for messages + done := make(chan struct{}) + + // Read messages goroutine + go func() { + defer close(done) + for { + _, message, err := conn.ReadMessage() + if err != nil { + return + } + + handleGeckoMessage(config, conn, message) + } + }() + + // Wait for welcome message + time.Sleep(2 * time.Second) + + // Subscribe to SwapChannel for all monitored pools + for _, pool := range geckoTerminalPools { + subscribeToGeckoSwapChannel(conn, pool.PoolID, pool.Name) + time.Sleep(100 * time.Millisecond) + } + + fmt.Printf("[HEAD-LAG][GECKO] Subscribed to %d pools\n", len(geckoTerminalPools)) + + // Heartbeat ticker + pingTicker := time.NewTicker(25 * time.Second) + defer pingTicker.Stop() + + // Read messages + for { + select { + case <-stopChan: + return nil + case <-done: + return fmt.Errorf("connection closed by server") + case <-pingTicker.C: + // Server sends pings, we respond with pongs (handled in handleGeckoMessage) + conn.SetWriteDeadline(time.Now().Add(10 * time.Second)) + } + } +} + +func handleGeckoMessage(config *Config, conn *websocket.Conn, message []byte) { + var msg GeckoActionCableMessage + if err := json.Unmarshal(message, &msg); err != nil { + return + } + + switch msg.Type { + case "welcome": + // Connection established + + case "ping": + // Respond to ping with pong + pong := GeckoActionCableMessage{ + Type: "pong", + } + conn.WriteJSON(pong) + + case "confirm_subscription": + // Subscription confirmed + + case "reject_subscription": + log.Printf("[HEAD-LAG][GECKO] Subscription rejected: %s", msg.Identifier) + + default: + // Handle data messages + if msg.Message != nil { + handleGeckoDataMessage(config, msg.Identifier, msg.Message) + } + } +} + +func handleGeckoDataMessage(config *Config, identifier string, message json.RawMessage) { + // Parse swap data + var swapData GeckoSwapData + if err := json.Unmarshal(message, &swapData); err != nil { + return + } + + if swapData.Type != "newSwap" { + return + } + + // Extract channel info to get pool + var channelIdent GeckoChannelIdentifier + if err := json.Unmarshal([]byte(identifier), &channelIdent); err != nil { + return + } + + // Find which pool this is + var poolChain string + for _, pool := range geckoTerminalPools { + if pool.PoolID == channelIdent.PoolID { + poolChain = pool.Chain + break + } + } + + if poolChain == "" { + return + } + + // Calculate head lag + receiveTime := time.Now().UTC() + onChainTime := time.UnixMilli(swapData.Data.BlockTimestamp) + lagMs := receiveTime.Sub(onChainTime).Milliseconds() + lagSeconds := float64(lagMs) / 1000.0 + + // Record metrics with tx hash + RecordHeadLag("geckoterminal", poolChain, lagMs, lagSeconds, config.MonitorRegion, swapData.Data.TxHash) + + // Log occasionally (not every trade) + if lagMs > 10000 || time.Now().Second()%30 == 0 { + timestamp := receiveTime.Format("15:04:05") + txHash := swapData.Data.TxHash + if len(txHash) > 12 { + txHash = txHash[:10] + "..." + } + fmt.Printf("[HEAD-LAG][GECKO][%s][%s] Lag: %.2fs | Tx: %s\n", + timestamp, poolChain, lagSeconds, txHash) + } +} + +func subscribeToGeckoSwapChannel(conn *websocket.Conn, poolID, poolName string) { + identifier := GeckoChannelIdentifier{ + Channel: "SwapChannel", + PoolID: poolID, + } + + identifierJSON, _ := json.Marshal(identifier) + + subscribeMsg := GeckoActionCableMessage{ + Command: "subscribe", + Identifier: string(identifierJSON), + } + + if err := conn.WriteJSON(subscribeMsg); err != nil { + log.Printf("[HEAD-LAG][GECKO] Error subscribing to %s: %v", poolName, err) + return + } +} diff --git a/harnesses/aggregator-head-lag/cmd/script/head_lag_monitor.go b/harnesses/aggregator-head-lag/cmd/script/head_lag_monitor.go new file mode 100644 index 00000000..7f1dfef9 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/head_lag_monitor.go @@ -0,0 +1,673 @@ +package main + +import ( + "encoding/json" + "fmt" + "log" + "strings" + "sync" + "time" + + "github.com/gorilla/websocket" +) + +// ============================================================================ +// Head Lag Monitor +// Measures indexation latency: time between on-chain event and WebSocket receipt +// ============================================================================ + +// Pool configurations for head lag monitoring +type HeadLagPool struct { + Name string // Human readable name + Blockchain string // For Mobula: "evm:1", "solana", etc. + NetworkID int // For Codex: 1, 1399811149, etc. + Address string // Pool address + ChainName string // Normalized chain name for metrics +} + +// Pools to monitor - high activity pools for accurate lag measurement +var headLagPools = []HeadLagPool{ + { + Name: "SOL/USDC Raydium", + Blockchain: "solana", + NetworkID: 1399811149, + Address: "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm", + ChainName: "solana", + }, + { + Name: "WETH/USDC Base", + Blockchain: "evm:8453", + NetworkID: 8453, + Address: "0x4c36388be6f416a29c8d8eee81c771ce6be14b18", + ChainName: "base", + }, + { + Name: "WBNB/BUSD PancakeSwap", + Blockchain: "evm:56", + NetworkID: 56, + Address: "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16", + ChainName: "bnb", + }, +} + +// ============================================================================ +// Mobula WebSocket Monitor +// ============================================================================ + +type MobulaTradeEvent struct { + Blockchain string `json:"blockchain"` + Date int64 `json:"date"` // On-chain timestamp (ms) + Timestamp int64 `json:"timestamp"` // When Mobula processed it (ms) + Hash string `json:"hash"` + Pair string `json:"pair"` + Type string `json:"type"` + TokenPrice float64 `json:"tokenPrice"` +} + +func runMobulaHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { + defer wg.Done() + + if config.MobulaAPIKey == "" { + fmt.Println("[HEAD-LAG][MOBULA] API key not set, skipping") + return + } + + fmt.Println("[HEAD-LAG][MOBULA] Starting WebSocket monitor...") + + reconnectDelay := 5 * time.Second + maxReconnectDelay := 60 * time.Second + + for { + select { + case <-stopChan: + fmt.Println("[HEAD-LAG][MOBULA] Monitor stopped") + return + default: + err := connectAndMonitorMobula(config, stopChan) + if err != nil { + log.Printf("[HEAD-LAG][MOBULA] Connection error: %v. Reconnecting in %v...", err, reconnectDelay) + + select { + case <-stopChan: + return + case <-time.After(reconnectDelay): + reconnectDelay = reconnectDelay * 2 + if reconnectDelay > maxReconnectDelay { + reconnectDelay = maxReconnectDelay + } + } + } else { + // Reset delay on clean disconnect + reconnectDelay = 5 * time.Second + } + } + } +} + +func connectAndMonitorMobula(config *Config, stopChan <-chan struct{}) error { + // Use direct connection for Mobula (no proxy needed, avoids latency spikes) + dialer := &websocket.Dialer{} + conn, _, err := dialer.Dial(config.MobulaWSURL, nil) + if err != nil { + return fmt.Errorf("dial failed: %w", err) + } + defer conn.Close() + + // Build subscription items + var items []map[string]interface{} + for _, pool := range headLagPools { + items = append(items, map[string]interface{}{ + "blockchain": pool.Blockchain, + "address": pool.Address, + }) + } + + // Subscribe to fast-trade + subscribeMsg := map[string]interface{}{ + "type": "fast-trade", + "authorization": config.MobulaAPIKey, + "payload": map[string]interface{}{ + "assetMode": false, + "items": items, + }, + } + + if err := conn.WriteJSON(subscribeMsg); err != nil { + return fmt.Errorf("subscribe failed: %w", err) + } + + fmt.Printf("[HEAD-LAG][MOBULA] Subscribed to %d pools\n", len(items)) + + // Start ping goroutine + pingDone := make(chan struct{}) + go func() { + ticker := time.NewTicker(25 * time.Second) + defer ticker.Stop() + for { + select { + case <-pingDone: + return + case <-ticker.C: + if err := conn.WriteJSON(map[string]string{"event": "ping"}); err != nil { + return + } + } + } + }() + defer close(pingDone) + + // Read messages + for { + select { + case <-stopChan: + return nil + default: + conn.SetReadDeadline(time.Now().Add(60 * time.Second)) + _, message, err := conn.ReadMessage() + if err != nil { + return fmt.Errorf("read failed: %w", err) + } + + // Parse message + var trade MobulaTradeEvent + if err := json.Unmarshal(message, &trade); err != nil { + continue + } + + // Skip non-trade messages (pong, etc) + if trade.Hash == "" || trade.Date == 0 { + continue + } + + // Calculate head lag + receiveTime := time.Now().UTC() + onChainTime := time.UnixMilli(trade.Date) + mobulaProcessTime := time.UnixMilli(trade.Timestamp) + + // Total lag: on-chain → WebSocket receipt + totalLagMs := receiveTime.Sub(onChainTime).Milliseconds() + + // Drop WebSocket replays / clock-skew events: not real indexation latency + // (Mobula WS occasionally replays old trades on reconnect; those would otherwise fire alerts) + if totalLagMs < 0 || totalLagMs > 30000 { + continue + } + + // Mobula processing latency: on-chain → Mobula processed + mobulaLagMs := mobulaProcessTime.Sub(onChainTime).Milliseconds() + + // Network latency: Mobula processed → WebSocket receipt + networkLagMs := receiveTime.Sub(mobulaProcessTime).Milliseconds() + + lagSeconds := float64(totalLagMs) / 1000.0 + + // Get chain name from pool config + chainName := getChainNameFromBlockchain(trade.Blockchain) + + // Record basic metric + RecordHeadLag("mobula", chainName, totalLagMs, lagSeconds, config.MonitorRegion, trade.Hash) + + // Track latest tx per pool so alert annotations can link to it + RecordMobulaLastTx(chainName, config.MonitorRegion, trade.Pair, trade.Hash) + + // Record detailed metric with breakdown + RecordMobulaHeadLagDetailed( + chainName, + config.MonitorRegion, + trade.Pair, + trade.Hash, + totalLagMs, + mobulaLagMs, + networkLagMs, + onChainTime.Format("2006-01-02T15:04:05Z"), + mobulaProcessTime.Format("2006-01-02T15:04:05Z"), + receiveTime.Format("2006-01-02T15:04:05Z"), + ) + + // Enhanced logging for spikes + if totalLagMs > 3000 { + timestamp := receiveTime.Format("15:04:05") + fmt.Printf("[HEAD-LAG][MOBULA][%s][%s] 🚨 SPIKE DETECTED!\n", timestamp, chainName) + fmt.Printf(" Total Lag: %.2fs (%.0fms)\n", lagSeconds, float64(totalLagMs)) + fmt.Printf(" ├─ Mobula Processing: %.0fms (on-chain → Mobula)\n", float64(mobulaLagMs)) + fmt.Printf(" └─ Network Latency: %.0fms (Mobula → WebSocket)\n", float64(networkLagMs)) + fmt.Printf(" On-chain: %s | Mobula: %s | Received: %s\n", + onChainTime.Format("15:04:05.000"), + mobulaProcessTime.Format("15:04:05.000"), + receiveTime.Format("15:04:05.000")) + fmt.Printf(" Tx: %s\n", trade.Hash) + } else if time.Now().Second()%30 == 0 { + // Log normal latency occasionally + timestamp := receiveTime.Format("15:04:05") + fmt.Printf("[HEAD-LAG][MOBULA][%s][%s] Lag: %.2fs | Tx: %s\n", + timestamp, chainName, lagSeconds, trade.Hash[:12]+"...") + } + } + } +} + +func getChainNameFromBlockchain(blockchain string) string { + switch blockchain { + case "Ethereum", "evm:1": + return "ethereum" + case "Solana", "solana": + return "solana" + case "Base", "evm:8453": + return "base" + case "BNB Smart Chain (BEP20)", "BSC", "evm:56": + return "bnb" + default: + return blockchain + } +} + +// ============================================================================ +// Codex WebSocket Monitor (using Defined.fi session auth) +// ============================================================================ + +type CodexWSMessage struct { + Type string `json:"type"` + ID string `json:"id,omitempty"` + Payload map[string]interface{} `json:"payload,omitempty"` +} + +type CodexEventData struct { + Data struct { + OnEventsCreated *struct { + Address string `json:"address"` + NetworkID int `json:"networkId"` + Events []struct { + BlockNumber int64 `json:"blockNumber"` + Timestamp int64 `json:"timestamp"` + TransactionHash string `json:"transactionHash"` + EventType string `json:"eventType"` + } `json:"events"` + } `json:"onEventsCreated,omitempty"` + OnUnconfirmedEventsCreated *struct { + Address string `json:"address"` + NetworkID int `json:"networkId"` + Events []struct { + BlockNumber int64 `json:"blockNumber"` + Timestamp int64 `json:"timestamp"` + TransactionHash string `json:"transactionHash"` + EventType string `json:"eventType"` + } `json:"events"` + } `json:"onUnconfirmedEventsCreated,omitempty"` + } `json:"data"` +} + +func runCodexHeadLagMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { + defer wg.Done() + + fmt.Println("[HEAD-LAG][CODEX] Starting WebSocket monitor (via Defined.fi auth)...") + + // Backoff strategy: + // - 10s base delay (proxy IP rotation window) + // - Doubles on consecutive net errors, capped at 60s (NOT 5min — too long + // for the rotating-proxy case; we want to keep probing for fresh IPs) + // - Every 10 consecutive failures, force-reset to 10s AND invalidate the JWT + // cache (covers silent token expiry that surfaces as net errors) + const baseDelay = 10 * time.Second + const maxReconnectDelay = 60 * time.Second + const forceResetEvery = 10 + + reconnectDelay := baseDelay + consecutiveFailures := 0 + attemptNum := 0 + + for { + select { + case <-stopChan: + fmt.Println("[HEAD-LAG][CODEX] Monitor stopped") + return + default: + attemptNum++ + log.Printf("[HEAD-LAG][CODEX] 🔄 Connection attempt #%d (proxy rotation enabled)", attemptNum) + + err := connectAndMonitorCodex(config, stopChan) + if err != nil { + consecutiveFailures++ + log.Printf("[HEAD-LAG][CODEX] ❌ Connection attempt #%d failed (%d consecutive): %v", attemptNum, consecutiveFailures, err) + + if strings.Contains(err.Error(), "rate limited (429)") || strings.Contains(err.Error(), "too many token requests") { + log.Printf("[HEAD-LAG][CODEX] ⚠️ Rate limited on JWT — invalidating cache, retrying in 30s") + InvalidateTokenCache() + reconnectDelay = 30 * time.Second + } else if strings.Contains(err.Error(), "authentication") || strings.Contains(err.Error(), "401") || strings.Contains(err.Error(), "4401") || strings.Contains(err.Error(), "4403") { + log.Printf("[HEAD-LAG][CODEX] 🔑 Auth error — invalidating JWT cache, retrying in 30s") + InvalidateTokenCache() + reconnectDelay = 30 * time.Second + } else { + // Net/IO error: exponential backoff, capped at 60s. + reconnectDelay *= 2 + if reconnectDelay < baseDelay { + reconnectDelay = baseDelay + } + if reconnectDelay > maxReconnectDelay { + reconnectDelay = maxReconnectDelay + } + } + + // Periodic force-reset: stale JWT silently rejected can manifest + // as net errors (server-closed connection). Every 10 failures, + // invalidate the token AND drop back to base delay so we probe + // fast on a fresh state. + if consecutiveFailures%forceResetEvery == 0 { + log.Printf("[HEAD-LAG][CODEX] 🔁 %d consecutive failures — force-reset (invalidate JWT + base delay)", consecutiveFailures) + InvalidateTokenCache() + reconnectDelay = baseDelay + } + + log.Printf("[HEAD-LAG][CODEX] ⏳ Reconnecting in %v... (next attempt: #%d)", reconnectDelay, attemptNum+1) + select { + case <-stopChan: + return + case <-time.After(reconnectDelay): + } + } else { + log.Printf("[HEAD-LAG][CODEX] ✅ Connection closed cleanly after attempt #%d", attemptNum) + reconnectDelay = baseDelay + consecutiveFailures = 0 + attemptNum = 0 + } + } + } +} + +func connectAndMonitorCodex(config *Config, stopChan <-chan struct{}) error { + log.Printf("[HEAD-LAG][CODEX] Step 1/4: Generating JWT token...") + jwtToken, err := GetDefinedJWTToken(config.DefinedSessionCookie) + if err != nil { + return fmt.Errorf("failed to get JWT token: %w", err) + } + + log.Printf("[HEAD-LAG][CODEX] Step 2/4: Creating proxy dialer...") + dialer := getProxyDialerWithSubprotocols([]string{"graphql-transport-ws"}) + + log.Printf("[HEAD-LAG][CODEX] Step 3/4: Connecting to wss://graph.codex.io/graphql...") + conn, resp, err := dialer.Dial("wss://graph.codex.io/graphql", nil) + if err != nil { + if resp != nil { + return fmt.Errorf("dial failed (HTTP %d): %w", resp.StatusCode, err) + } + return fmt.Errorf("dial failed: %w", err) + } + defer conn.Close() + + // Verify proxy IP rotation (for debugging) + if outboundIP, err := getOutboundIP(); err == nil { + log.Printf("[HEAD-LAG][CODEX] Step 3/4: ✅ WebSocket connection established via proxy IP: %s", outboundIP) + } else { + log.Printf("[HEAD-LAG][CODEX] Step 3/4: ✅ WebSocket connection established (IP check failed: %v)", err) + } + + // Connection init with JWT Bearer token + log.Printf("[HEAD-LAG][CODEX] Step 4/4: Sending connection_init with JWT...") + initMsg := map[string]interface{}{ + "type": "connection_init", + "payload": map[string]interface{}{ + "Authorization": fmt.Sprintf("Bearer %s", jwtToken), + }, + } + if err := conn.WriteJSON(initMsg); err != nil { + return fmt.Errorf("init failed: %w", err) + } + + // Wait for ack + conn.SetReadDeadline(time.Now().Add(10 * time.Second)) + _, msg, err := conn.ReadMessage() + if err != nil { + return fmt.Errorf("ack read failed: %w", err) + } + + var ackMsg CodexWSMessage + if err := json.Unmarshal(msg, &ackMsg); err != nil || ackMsg.Type != "connection_ack" { + return fmt.Errorf("unexpected ack: %s", string(msg)) + } + log.Printf("[HEAD-LAG][CODEX] Step 4/4: ✅ Received connection_ack") + + // Subscribe to each pool (use onUnconfirmedEventsCreated for Solana, onEventsCreated for others) + log.Printf("[HEAD-LAG][CODEX] Subscribing to %d pools...", len(headLagPools)) + for i, pool := range headLagPools { + subID := fmt.Sprintf("headlag_%d", i) + + var subMsg map[string]interface{} + + // Solana: use onUnconfirmedEventsCreated (pre-finalized, lowest latency) + if pool.ChainName == "solana" { + poolID := fmt.Sprintf("%s:%d", pool.Address, pool.NetworkID) + subMsg = map[string]interface{}{ + "type": "subscribe", + "id": subID, + "payload": map[string]interface{}{ + "query": `subscription OnPoolEvents($id: String!) { + onUnconfirmedEventsCreated(id: $id, quoteToken: token0) { + address + networkId + events { + blockNumber + timestamp + transactionHash + eventType + } + } + }`, + "variables": map[string]interface{}{ + "id": poolID, + }, + }, + } + } else { + // EVM chains: use onEventsCreated (confirmed events) + subMsg = map[string]interface{}{ + "type": "subscribe", + "id": subID, + "payload": map[string]interface{}{ + "query": `subscription OnPoolEvents($address: String!, $networkId: Int!) { + onEventsCreated(address: $address, networkId: $networkId) { + address + networkId + events { + blockNumber + timestamp + transactionHash + eventType + } + } + }`, + "variables": map[string]interface{}{ + "address": pool.Address, + "networkId": pool.NetworkID, + }, + }, + } + } + + if err := conn.WriteJSON(subMsg); err != nil { + return fmt.Errorf("subscribe to %s failed: %w", pool.Name, err) + } + + time.Sleep(100 * time.Millisecond) // Small delay between subscriptions + } + + log.Printf("[HEAD-LAG][CODEX] ✅ Subscribed to %d pools", len(headLagPools)) + log.Printf("[HEAD-LAG][CODEX] 🎉 Connection fully established! Waiting for events...") + + // graphql-transport-ws keepalive: send {"type":"ping"} every 20s. + // Without this, the rotating proxy (or Codex itself) silently kills idle + // TCP after ~3min, manifesting as "i/o timeout" or "1006 unexpected EOF". + // Writes to the websocket conn are serialized via writeMu since we now have + // two goroutines writing (this ping loop + the subscribe path on next reconnect). + var writeMu sync.Mutex + pingDone := make(chan struct{}) + go func() { + t := time.NewTicker(20 * time.Second) + defer t.Stop() + for { + select { + case <-pingDone: + return + case <-t.C: + writeMu.Lock() + err := conn.WriteJSON(map[string]string{"type": "ping"}) + writeMu.Unlock() + if err != nil { + return // read loop will see the failure too and exit cleanly + } + } + } + }() + defer close(pingDone) + + // Read messages + for { + select { + case <-stopChan: + return nil + default: + // 90s deadline: longer than ping interval so a single missed pong doesn't + // kill us, but short enough to fail fast if the server actually went away. + conn.SetReadDeadline(time.Now().Add(90 * time.Second)) + _, message, err := conn.ReadMessage() + if err != nil { + return fmt.Errorf("read failed: %w", err) + } + + // Parse message + var wsMsg CodexWSMessage + if err := json.Unmarshal(message, &wsMsg); err != nil { + continue + } + + // graphql-transport-ws keepalive: server may push ping; reply pong. + // Server may also push pong in response to our ping — silently consume. + if wsMsg.Type == "ping" { + writeMu.Lock() + _ = conn.WriteJSON(map[string]string{"type": "pong"}) + writeMu.Unlock() + continue + } + if wsMsg.Type == "pong" { + continue + } + + // Skip non-data messages + if wsMsg.Type != "next" || wsMsg.Payload == nil { + continue + } + + // Parse event data + payloadBytes, _ := json.Marshal(wsMsg.Payload) + var eventData CodexEventData + if err := json.Unmarshal(payloadBytes, &eventData); err != nil { + continue + } + + // Handle both onEventsCreated and onUnconfirmedEventsCreated + var events []struct { + BlockNumber int64 `json:"blockNumber"` + Timestamp int64 `json:"timestamp"` + TransactionHash string `json:"transactionHash"` + EventType string `json:"eventType"` + } + var networkID int + + if eventData.Data.OnEventsCreated != nil { + events = eventData.Data.OnEventsCreated.Events + networkID = eventData.Data.OnEventsCreated.NetworkID + } else if eventData.Data.OnUnconfirmedEventsCreated != nil { + events = eventData.Data.OnUnconfirmedEventsCreated.Events + networkID = eventData.Data.OnUnconfirmedEventsCreated.NetworkID + } + + if len(events) == 0 { + continue + } + + for _, event := range events { + if event.EventType != "Swap" || event.TransactionHash == "" { + continue + } + + // Calculate head lag + receiveTime := time.Now().UTC() + onChainTime := time.Unix(event.Timestamp, 0) + lagMs := receiveTime.Sub(onChainTime).Milliseconds() + lagSeconds := float64(lagMs) / 1000.0 + + // Get chain name + chainName := getChainNameFromNetworkID(networkID) + + // Record metrics with tx hash + RecordHeadLag("codex", chainName, lagMs, lagSeconds, config.MonitorRegion, event.TransactionHash) + RecordCodexBlockNumber(chainName, event.BlockNumber, config.MonitorRegion) + + // Enhanced logging for spikes + if lagMs > 3000 { + timestamp := receiveTime.Format("15:04:05") + fmt.Printf("[HEAD-LAG][CODEX][%s][%s] 🚨 SPIKE DETECTED!\n", timestamp, chainName) + fmt.Printf(" Total Lag: %.2fs (%.0fms)\n", lagSeconds, float64(lagMs)) + fmt.Printf(" On-chain: %s | Received: %s\n", + onChainTime.Format("15:04:05.000"), + receiveTime.Format("15:04:05.000")) + fmt.Printf(" Block: %d | Tx: %s\n", event.BlockNumber, event.TransactionHash) + } else if time.Now().Second()%30 == 0 { + // Log normal latency occasionally + timestamp := receiveTime.Format("15:04:05") + fmt.Printf("[HEAD-LAG][CODEX][%s][%s] Lag: %.2fs | Block: %d | Tx: %s\n", + timestamp, chainName, lagSeconds, event.BlockNumber, event.TransactionHash[:12]+"...") + } + } + } + } +} + +func getChainNameFromNetworkID(networkID int) string { + switch networkID { + case 1: + return "ethereum" + case 1399811149: + return "solana" + case 8453: + return "base" + case 56: + return "bnb" + default: + return fmt.Sprintf("network_%d", networkID) + } +} + +// ============================================================================ +// Main Head Lag Monitor +// ============================================================================ + +func runHeadLagMonitor(config *Config, stopChan <-chan struct{}) { + fmt.Println() + fmt.Println("╔══════════════════════════════════════════════════════════════╗") + fmt.Println("║ HEAD LAG MONITOR (WebSocket-based) ║") + fmt.Println("╠══════════════════════════════════════════════════════════════╣") + fmt.Println("║ Measures: Time between on-chain event and WebSocket receipt ║") + fmt.Println("║ Providers: Mobula + Pulse + Codex + GeckoTerminal ║") + fmt.Printf("║ Pools: %d high-activity pools across 3 chains ║\n", len(headLagPools)) + fmt.Printf("║ Region: %s ║\n", config.MonitorRegion) + fmt.Println("╚══════════════════════════════════════════════════════════════╝") + fmt.Println() + + var wg sync.WaitGroup + + // Start Mobula fast-trade monitor + wg.Add(1) + go runMobulaHeadLagMonitor(config, stopChan, &wg) + + // Start Codex monitor + wg.Add(1) + go runCodexHeadLagMonitor(config, stopChan, &wg) + + // Start GeckoTerminal monitor + wg.Add(1) + go runGeckoTerminalHeadLagMonitor(config, stopChan, &wg) + + // Wait for all to finish + wg.Wait() + fmt.Println("[HEAD-LAG] All monitors stopped") +} diff --git a/harnesses/aggregator-head-lag/cmd/script/log_buffer.go b/harnesses/aggregator-head-lag/cmd/script/log_buffer.go new file mode 100644 index 00000000..0f7455e0 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/log_buffer.go @@ -0,0 +1,102 @@ +package main + +import ( + "bufio" + "fmt" + "io" + "net/http" + "os" + "strconv" + "sync" + "time" +) + +// logBuffer keeps the last N log lines in memory for debug fetching via /logs. +// Captures BOTH log.* and fmt.Print* output (stdout is dup'd via a pipe). +type logBuffer struct { + mu sync.Mutex + lines []string + max int +} + +const logBufferMax = 5000 + +var globalLogBuffer = &logBuffer{max: logBufferMax} + +func (b *logBuffer) push(line string) { + entry := time.Now().UTC().Format("2006-01-02T15:04:05.000Z") + " " + line + b.mu.Lock() + if len(b.lines) >= b.max { + b.lines = append(b.lines[1:], entry) + } else { + b.lines = append(b.lines, entry) + } + b.mu.Unlock() +} + +func (b *logBuffer) Snapshot(tail int) []string { + b.mu.Lock() + defer b.mu.Unlock() + if tail <= 0 || tail >= len(b.lines) { + out := make([]string, len(b.lines)) + copy(out, b.lines) + return out + } + start := len(b.lines) - tail + out := make([]string, tail) + copy(out, b.lines[start:]) + return out +} + +// installLogCapture replaces os.Stdout with the write-end of a pipe, then +// spawns a goroutine that fan-outs every line to the real stdout AND the +// in-memory ring buffer. This catches fmt.Println/Printf as well as log.Printf. +// +// Call exactly once, very early in main(). +func installLogCapture() { + originalStdout := os.Stdout + r, w, err := os.Pipe() + if err != nil { + // Fallback: don't intercept. /logs will be empty but everything else still works. + fmt.Fprintf(originalStdout, "[log_buffer] failed to create pipe: %v (logs endpoint will be empty)\n", err) + return + } + os.Stdout = w + + go func() { + scanner := bufio.NewScanner(r) + // Allow long lines (default is 64KB, bump to 1MB for safety) + buf := make([]byte, 0, 1024*1024) + scanner.Buffer(buf, 1024*1024) + for scanner.Scan() { + line := scanner.Text() + fmt.Fprintln(originalStdout, line) + globalLogBuffer.push(line) + } + // Pipe closed (process shutdown) — drain anything left + _, _ = io.Copy(originalStdout, r) + }() +} + +// setupLogsEndpoint exposes GET /logs?tail=N (default 500, max logBufferMax). +// If LOGS_TOKEN env var is set, requires header `X-Logs-Token` to match. +// Otherwise the endpoint is open — only safe for Railway-internal access. +func setupLogsEndpoint(mux *http.ServeMux) { + expectedToken := os.Getenv("LOGS_TOKEN") + mux.HandleFunc("/logs", func(w http.ResponseWriter, r *http.Request) { + if expectedToken != "" && r.Header.Get("X-Logs-Token") != expectedToken { + http.Error(w, "forbidden", http.StatusForbidden) + return + } + tail := 500 + if t := r.URL.Query().Get("tail"); t != "" { + if n, err := strconv.Atoi(t); err == nil && n > 0 { + tail = n + } + } + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + for _, l := range globalLogBuffer.Snapshot(tail) { + fmt.Fprintln(w, l) + } + }) +} diff --git a/harnesses/aggregator-head-lag/cmd/script/main.go b/harnesses/aggregator-head-lag/cmd/script/main.go new file mode 100644 index 00000000..d124aa1b --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/main.go @@ -0,0 +1,98 @@ +package main + +import ( + "fmt" + "os" + "os/signal" + "sync" + "syscall" +) + +func main() { + installLogCapture() // must be first — captures all subsequent stdout into ring buffer for /logs + fmt.Println("=== Aggregator Indexation Lag Monitor ===") + fmt.Println("Measuring real-time indexation lag (head lag) for blockchain data APIs") + fmt.Println("Press Ctrl+C to stop") + fmt.Println() + + config, err := loadEnv() + if err != nil { + fmt.Printf("Error: %v\n", err) + os.Exit(1) + } + + // Use session cookie from environment (scraping requires GUI, doesn't work on Railway) + if config.DefinedSessionCookie == "" { + fmt.Println("Warning: DEFINED_SESSION_COOKIE not set in environment") + fmt.Println("Codex REST and WebSocket monitors will not work") + } else { + fmt.Printf("Using DEFINED_SESSION_COOKIE from environment (length: %d)\n", len(config.DefinedSessionCookie)) + } + + fmt.Println("Metrics will be exposed on :2112/metrics for Prometheus") + fmt.Println() + + sigChan := make(chan os.Signal, 1) + signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM) + + var wg sync.WaitGroup + stopChan := make(chan struct{}) + + wg.Add(1) + go func() { + defer wg.Done() + fmt.Println("Starting Prometheus metrics server on :2112") + if err := StartMetricsServer(":2112"); err != nil { + fmt.Printf("Metrics server error: %v\n", err) + } + }() + + // Mobula REST API monitor + wg.Add(1) + go func() { + defer wg.Done() + runMobulaRESTMonitor(config, stopChan) + }() + + // Codex REST API monitor + wg.Add(1) + go func() { + defer wg.Done() + runCodexRESTMonitor(config, stopChan) + }() + + // Quote API latency monitor (Jupiter, Li.Fi, 1inch, KyberSwap) + wg.Add(1) + go func() { + defer wg.Done() + runQuoteAPIMonitor(config, stopChan) + }() + + // Metadata coverage monitor (Mobula vs Codex) + wg.Add(1) + go func() { + defer wg.Done() + runMetadataCoverageMonitor(config, stopChan) + }() + + // Head lag monitor (blockchain head vs aggregator indexed head) + wg.Add(1) + go func() { + defer wg.Done() + runHeadLagMonitor(config, stopChan) + }() + + // Mobula Fast-Trade monitor (for comparison with Pulse V2) + wg.Add(1) + go func() { + defer wg.Done() + runMobulaFastTradeMonitor(config, stopChan) + }() + + <-sigChan + fmt.Println("\n\nShutting down monitors...") + close(stopChan) + + wg.Wait() + fmt.Println("All monitors stopped") +} diff --git a/harnesses/aggregator-head-lag/cmd/script/metadata_coverage_monitor.go b/harnesses/aggregator-head-lag/cmd/script/metadata_coverage_monitor.go new file mode 100644 index 00000000..672b9f3f --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/metadata_coverage_monitor.go @@ -0,0 +1,729 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "sync" + "time" +) + +// ============================================================================ +// Metadata Coverage Monitor +// Measures metadata and logo coverage across providers (Mobula, Codex) +// ============================================================================ + +const ( + mobulaTokenDetailsURL = "https://api.mobula.io/api/2/token/details" + codexGraphQLURL = "https://graph.codex.io/graphql" + jupiterTokenPageURL = "https://jup.ag/tokens/" +) + +// TokenToCheck represents a token discovered via Pulse that needs metadata checking +type TokenToCheck struct { + Address string + ChainID string // e.g., "solana", "evm:1", "evm:8453" + Symbol string + Name string + DetectedAt time.Time +} + +// MetadataFields represents the fields we check for coverage +type MetadataFields struct { + HasLogo bool + HasName bool + HasSymbol bool + HasDescription bool + HasTwitter bool + HasWebsite bool + HasTelegram bool + LogoURL string + ResponseTimeMs float64 + Error string +} + +// ProviderCoverage holds coverage stats for a single provider +type ProviderCoverage struct { + Provider string + TotalChecks int + LogoCount int + NameCount int + SymbolCount int + DescCount int + TwitterCount int + WebsiteCount int + TelegramCount int + ErrorCount int + TotalLatencyMs float64 +} + +// MetadataCoverageStats holds overall stats +type MetadataCoverageStats struct { + mu sync.Mutex + Mobula ProviderCoverage + Codex ProviderCoverage + Jupiter ProviderCoverage + LastPrint time.Time +} + +var ( + coverageStats = &MetadataCoverageStats{ + Mobula: ProviderCoverage{Provider: "mobula"}, + Codex: ProviderCoverage{Provider: "codex"}, + Jupiter: ProviderCoverage{Provider: "jupiter"}, + } + tokenQueue = make(chan TokenToCheck, 500) + metadataClient = &http.Client{Timeout: 10 * time.Second} +) + +// ============================================================================ +// Mobula API - Token Details +// ============================================================================ + +type MobulaTokenDetailsResponse struct { + Data MobulaTokenData `json:"data"` +} + +type MobulaTokenData struct { + Address string `json:"address"` + Name string `json:"name"` + Symbol string `json:"symbol"` + Logo string `json:"logo"` + Description string `json:"description"` + Socials MobulaSocials `json:"socials"` +} + +type MobulaSocials struct { + Twitter string `json:"twitter"` + Website string `json:"website"` + Telegram string `json:"telegram"` +} + +func checkMobulaMetadata(token TokenToCheck, apiKey string) MetadataFields { + result := MetadataFields{} + + // Build URL with query params + params := url.Values{} + params.Add("address", token.Address) + + // Convert chainID to Mobula format + blockchain := token.ChainID + if blockchain == "solana:solana" { + blockchain = "solana" + } + params.Add("blockchain", blockchain) + + fullURL := fmt.Sprintf("%s?%s", mobulaTokenDetailsURL, params.Encode()) + + req, err := http.NewRequest("GET", fullURL, nil) + if err != nil { + result.Error = fmt.Sprintf("request_create_error: %v", err) + return result + } + + req.Header.Set("Accept", "application/json") + if apiKey != "" { + req.Header.Set("Authorization", apiKey) + } + + startTime := time.Now() + resp, err := metadataClient.Do(req) + result.ResponseTimeMs = float64(time.Since(startTime).Milliseconds()) + + if err != nil { + result.Error = fmt.Sprintf("request_error: %v", err) + return result + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + result.Error = fmt.Sprintf("status_%d", resp.StatusCode) + return result + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + result.Error = fmt.Sprintf("read_error: %v", err) + return result + } + + var response MobulaTokenDetailsResponse + if err := json.Unmarshal(body, &response); err != nil { + result.Error = fmt.Sprintf("parse_error: %v", err) + return result + } + + data := response.Data + + // Check each field + result.HasName = data.Name != "" + result.HasSymbol = data.Symbol != "" + result.HasLogo = data.Logo != "" + result.LogoURL = data.Logo + result.HasDescription = data.Description != "" + result.HasTwitter = data.Socials.Twitter != "" + result.HasWebsite = data.Socials.Website != "" + result.HasTelegram = data.Socials.Telegram != "" + + return result +} + +// ============================================================================ +// Codex API - GraphQL Token Query +// ============================================================================ + +// Note: CodexGraphQLRequest is defined in codex_rest_monitor.go + +// CodexTokenResponse represents the response from token query +// Returns EnhancedToken with socialLinks and info +// https://docs.codex.io/api-reference/queries/token +type CodexTokenResponse struct { + Data struct { + Token CodexEnhancedToken `json:"token"` + } `json:"data"` + Errors []struct { + Message string `json:"message"` + } `json:"errors"` +} + +// CodexEnhancedToken matches the EnhancedToken type from Codex API +type CodexEnhancedToken struct { + Address string `json:"address"` + Name string `json:"name"` + Symbol string `json:"symbol"` + Decimals int `json:"decimals"` + NetworkID int `json:"networkId"` + Info *CodexTokenInfo `json:"info"` + SocialLinks *CodexSocialLinks `json:"socialLinks"` +} + +// CodexTokenInfo contains metadata about the token +type CodexTokenInfo struct { + ImageThumbUrl string `json:"imageThumbUrl"` + ImageSmallUrl string `json:"imageSmallUrl"` + ImageLargeUrl string `json:"imageLargeUrl"` + Description string `json:"description"` + CirculatingSupply string `json:"circulatingSupply"` + TotalSupply string `json:"totalSupply"` +} + +// CodexSocialLinks contains social media links for the token +type CodexSocialLinks struct { + Twitter string `json:"twitter"` + Website string `json:"website"` + Telegram string `json:"telegram"` + Discord string `json:"discord"` + Github string `json:"github"` +} + +func getCodexNetworkID(chainID string) int { + switch chainID { + case "solana", "solana:solana": + return 1399811149 + case "evm:1": + return 1 + case "evm:8453": + return 8453 + case "evm:56": + return 56 + case "evm:42161": + return 42161 + default: + return 0 + } +} + +func checkCodexMetadata(token TokenToCheck, sessionCookie string) MetadataFields { + result := MetadataFields{} + + networkID := getCodexNetworkID(token.ChainID) + if networkID == 0 { + result.Error = "unsupported_chain" + return result + } + + // Use session cookie directly + apiKey := sessionCookie + + // Use token query which returns EnhancedToken with socialLinks and info + // https://docs.codex.io/api-reference/queries/token + query := `query GetToken($address: String!, $networkId: Int!) { + token(input: { address: $address, networkId: $networkId }) { + address + name + symbol + decimals + networkId + info { + imageThumbUrl + imageSmallUrl + imageLargeUrl + description + circulatingSupply + totalSupply + } + socialLinks { + twitter + website + telegram + discord + github + } + } + }` + + reqBody := CodexGraphQLRequest{ + Query: query, + Variables: map[string]interface{}{ + "address": token.Address, + "networkId": networkID, + }, + } + + jsonBody, err := json.Marshal(reqBody) + if err != nil { + result.Error = fmt.Sprintf("marshal_error: %v", err) + return result + } + + req, err := http.NewRequest("POST", codexGraphQLURL, bytes.NewBuffer(jsonBody)) + if err != nil { + result.Error = fmt.Sprintf("request_create_error: %v", err) + return result + } + + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json") + if apiKey != "" { + req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", apiKey)) + } + + startTime := time.Now() + resp, err := metadataClient.Do(req) + result.ResponseTimeMs = float64(time.Since(startTime).Milliseconds()) + + if err != nil { + result.Error = fmt.Sprintf("request_error: %v", err) + return result + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + result.Error = fmt.Sprintf("status_%d", resp.StatusCode) + return result + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + result.Error = fmt.Sprintf("read_error: %v", err) + return result + } + + var response CodexTokenResponse + if err := json.Unmarshal(body, &response); err != nil { + result.Error = fmt.Sprintf("parse_error: %v", err) + return result + } + + if len(response.Errors) > 0 { + result.Error = fmt.Sprintf("graphql_error: %s", response.Errors[0].Message) + return result + } + + data := response.Data.Token + + // Check if token was found + if data.Address == "" { + result.Error = "token_not_found" + return result + } + + // Check each field based on EnhancedToken + // https://docs.codex.io/api-reference/queries/token + result.HasName = data.Name != "" + result.HasSymbol = data.Symbol != "" + + // Check logo from info + if data.Info != nil { + result.HasLogo = data.Info.ImageThumbUrl != "" || data.Info.ImageSmallUrl != "" || data.Info.ImageLargeUrl != "" + if data.Info.ImageLargeUrl != "" { + result.LogoURL = data.Info.ImageLargeUrl + } else if data.Info.ImageSmallUrl != "" { + result.LogoURL = data.Info.ImageSmallUrl + } else { + result.LogoURL = data.Info.ImageThumbUrl + } + result.HasDescription = data.Info.Description != "" + } + + // Check social links + if data.SocialLinks != nil { + result.HasTwitter = data.SocialLinks.Twitter != "" + result.HasWebsite = data.SocialLinks.Website != "" + result.HasTelegram = data.SocialLinks.Telegram != "" + } + + return result +} + +// ============================================================================ +// Jupiter - Scraping from frontend (Solana only) +// ============================================================================ + +// JupiterNextData represents the __NEXT_DATA__ JSON structure +type JupiterNextData struct { + Props struct { + PageProps struct { + DehydratedState struct { + Queries []struct { + State struct { + Data JupiterTokenData `json:"data"` + } `json:"state"` + } `json:"queries"` + } `json:"dehydratedState"` + } `json:"pageProps"` + } `json:"props"` +} + +// JupiterTokenData represents token data from Jupiter +type JupiterTokenData struct { + ID string `json:"id"` + Name string `json:"name"` + Symbol string `json:"symbol"` + Icon string `json:"icon"` + Decimals int `json:"decimals"` +} + +func checkJupiterMetadata(token TokenToCheck) MetadataFields { + result := MetadataFields{} + + // Jupiter only supports Solana + if token.ChainID != "solana" && token.ChainID != "solana:solana" { + result.Error = "unsupported_chain" + return result + } + + // Scrape the token page + pageURL := jupiterTokenPageURL + token.Address + + req, err := http.NewRequest("GET", pageURL, nil) + if err != nil { + result.Error = fmt.Sprintf("request_create_error: %v", err) + return result + } + + req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36") + req.Header.Set("Accept", "text/html,application/xhtml+xml") + + startTime := time.Now() + resp, err := metadataClient.Do(req) + result.ResponseTimeMs = float64(time.Since(startTime).Milliseconds()) + + if err != nil { + result.Error = fmt.Sprintf("request_error: %v", err) + return result + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + result.Error = fmt.Sprintf("status_%d", resp.StatusCode) + return result + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + result.Error = fmt.Sprintf("read_error: %v", err) + return result + } + + // Extract __NEXT_DATA__ JSON from HTML + htmlContent := string(body) + startMarker := `` + + startIdx := -1 + for i := 0; i < len(htmlContent)-len(startMarker); i++ { + if htmlContent[i:i+len(startMarker)] == startMarker { + startIdx = i + len(startMarker) + break + } + } + + if startIdx == -1 { + result.Error = "next_data_not_found" + return result + } + + endIdx := -1 + for i := startIdx; i < len(htmlContent)-len(endMarker); i++ { + if htmlContent[i:i+len(endMarker)] == endMarker { + endIdx = i + break + } + } + + if endIdx == -1 { + result.Error = "next_data_end_not_found" + return result + } + + jsonData := htmlContent[startIdx:endIdx] + + var nextData JupiterNextData + if err := json.Unmarshal([]byte(jsonData), &nextData); err != nil { + result.Error = fmt.Sprintf("parse_error: %v", err) + return result + } + + // Find token data in queries + var tokenData JupiterTokenData + for _, query := range nextData.Props.PageProps.DehydratedState.Queries { + if query.State.Data.ID == token.Address { + tokenData = query.State.Data + break + } + } + + if tokenData.ID == "" { + result.Error = "token_not_found" + return result + } + + // Check fields - Jupiter only has basic on-chain data + result.HasName = tokenData.Name != "" + result.HasSymbol = tokenData.Symbol != "" + result.HasLogo = tokenData.Icon != "" + result.LogoURL = tokenData.Icon + // Jupiter doesn't have description or socials + result.HasDescription = false + result.HasTwitter = false + result.HasWebsite = false + result.HasTelegram = false + + return result +} + +// ============================================================================ +// Stats and Reporting +// ============================================================================ + +func updateStats(provider string, fields MetadataFields) { + coverageStats.mu.Lock() + defer coverageStats.mu.Unlock() + + var stats *ProviderCoverage + switch provider { + case "mobula": + stats = &coverageStats.Mobula + case "codex": + stats = &coverageStats.Codex + case "jupiter": + stats = &coverageStats.Jupiter + default: + return + } + + stats.TotalChecks++ + stats.TotalLatencyMs += fields.ResponseTimeMs + + if fields.Error != "" { + stats.ErrorCount++ + return + } + + if fields.HasLogo { + stats.LogoCount++ + } + if fields.HasName { + stats.NameCount++ + } + if fields.HasSymbol { + stats.SymbolCount++ + } + if fields.HasDescription { + stats.DescCount++ + } + if fields.HasTwitter { + stats.TwitterCount++ + } + if fields.HasWebsite { + stats.WebsiteCount++ + } + if fields.HasTelegram { + stats.TelegramCount++ + } +} + +func printCoverageStats() { + coverageStats.mu.Lock() + defer coverageStats.mu.Unlock() + + timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") + + fmt.Printf("\n") + fmt.Printf("╔══════════════════════════════════════════════════════════════════════════════╗\n") + fmt.Printf("║ METADATA COVERAGE STATS - %s ║\n", timestamp) + fmt.Printf("╠══════════════════════════════════════════════════════════════════════════════╣\n") + fmt.Printf("║ Provider │ Checks │ Logo │ Name │ Symbol│ Desc │Twitter│Website│Telegram│ Errors │\n") + fmt.Printf("╠══════════════════════════════════════════════════════════════════════════════╣\n") + + for _, stats := range []*ProviderCoverage{&coverageStats.Mobula, &coverageStats.Codex, &coverageStats.Jupiter} { + if stats.TotalChecks == 0 { + fmt.Printf("║ %-8s │ %6d │ - │ - │ - │ - │ - │ - │ - │ %6d ║\n", + stats.Provider, stats.TotalChecks, stats.ErrorCount) + continue + } + + successChecks := stats.TotalChecks - stats.ErrorCount + if successChecks == 0 { + successChecks = 1 // Avoid division by zero + } + + fmt.Printf("║ %-8s │ %6d │ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%%│ %5.1f%% │ %6d ║\n", + stats.Provider, + stats.TotalChecks, + float64(stats.LogoCount)/float64(successChecks)*100, + float64(stats.NameCount)/float64(successChecks)*100, + float64(stats.SymbolCount)/float64(successChecks)*100, + float64(stats.DescCount)/float64(successChecks)*100, + float64(stats.TwitterCount)/float64(successChecks)*100, + float64(stats.WebsiteCount)/float64(successChecks)*100, + float64(stats.TelegramCount)/float64(successChecks)*100, + stats.ErrorCount, + ) + } + + fmt.Printf("╚══════════════════════════════════════════════════════════════════════════════╝\n") + fmt.Printf("\n") + + coverageStats.LastPrint = time.Now() +} + +func chainNameFromPulseChainID(chainID string) string { + switch chainID { + case "solana:solana": + return "solana" + case "evm:56": + return "bnb" + case "evm:8453": + return "base" + case "evm:143": + return "monad" + default: + return chainID + } +} + +func checkTokenMetadata(token TokenToCheck, config *Config) { + chainName := chainNameFromPulseChainID(token.ChainID) + + // Check Mobula + mobulaResult := checkMobulaMetadata(token, config.MobulaAPIKey) + updateStats("mobula", mobulaResult) + + // Record Prometheus metrics for Mobula + RecordMetadataCoverage("mobula", chainName, "logo", mobulaResult.HasLogo, config.MonitorRegion) + RecordMetadataCoverage("mobula", chainName, "description", mobulaResult.HasDescription, config.MonitorRegion) + RecordMetadataCoverage("mobula", chainName, "twitter", mobulaResult.HasTwitter, config.MonitorRegion) + RecordMetadataCoverage("mobula", chainName, "website", mobulaResult.HasWebsite, config.MonitorRegion) + RecordMetadataLatency("mobula", chainName, mobulaResult.ResponseTimeMs, config.MonitorRegion) + + // Check Codex + codexResult := checkCodexMetadata(token, config.DefinedSessionCookie) + updateStats("codex", codexResult) + + // Record Prometheus metrics for Codex + RecordMetadataCoverage("codex", chainName, "logo", codexResult.HasLogo, config.MonitorRegion) + RecordMetadataCoverage("codex", chainName, "description", codexResult.HasDescription, config.MonitorRegion) + RecordMetadataCoverage("codex", chainName, "twitter", codexResult.HasTwitter, config.MonitorRegion) + RecordMetadataCoverage("codex", chainName, "website", codexResult.HasWebsite, config.MonitorRegion) + RecordMetadataLatency("codex", chainName, codexResult.ResponseTimeMs, config.MonitorRegion) + + // Check Jupiter (Solana only - scraping frontend) + var jupiterResult MetadataFields + if token.ChainID == "solana" || token.ChainID == "solana:solana" { + jupiterResult = checkJupiterMetadata(token) + updateStats("jupiter", jupiterResult) + + // Record Prometheus metrics for Jupiter + RecordMetadataCoverage("jupiter", chainName, "logo", jupiterResult.HasLogo, config.MonitorRegion) + RecordMetadataCoverage("jupiter", chainName, "description", jupiterResult.HasDescription, config.MonitorRegion) + RecordMetadataCoverage("jupiter", chainName, "twitter", jupiterResult.HasTwitter, config.MonitorRegion) + RecordMetadataCoverage("jupiter", chainName, "website", jupiterResult.HasWebsite, config.MonitorRegion) + RecordMetadataLatency("jupiter", chainName, jupiterResult.ResponseTimeMs, config.MonitorRegion) + } + + // Single condensed log line + boolToIcon := func(b bool) string { + if b { + return "✓" + } + return "✗" + } + + jupiterLogo := "-" + if token.ChainID == "solana" || token.ChainID == "solana:solana" { + jupiterLogo = boolToIcon(jupiterResult.HasLogo) + } + + fmt.Printf("[META] %s/%s | M:%s%s%s | C:%s%s%s | J:%s\n", + token.Symbol, chainName, + boolToIcon(mobulaResult.HasLogo), boolToIcon(mobulaResult.HasDescription), boolToIcon(mobulaResult.HasTwitter), + boolToIcon(codexResult.HasLogo), boolToIcon(codexResult.HasDescription), boolToIcon(codexResult.HasTwitter), + jupiterLogo) + + // Print stats every 50 checks (reduced from 10) + coverageStats.mu.Lock() + totalChecks := coverageStats.Mobula.TotalChecks + coverageStats.mu.Unlock() + + if totalChecks > 0 && totalChecks%50 == 0 { + printCoverageStats() + } +} + +// QueueTokenForMetadataCheck adds a token to the check queue +func QueueTokenForMetadataCheck(token TokenToCheck) { + select { + case tokenQueue <- token: + // Token queued successfully + default: + // Queue full, skip this token + fmt.Printf("[METADATA] Queue full, skipping token: %s\n", token.Address) + } +} + +// runMetadataCoverageMonitor starts the metadata coverage monitoring +func runMetadataCoverageMonitor(config *Config, stopChan <-chan struct{}) { + fmt.Println("Starting Metadata Coverage Monitor...") + fmt.Println(" Comparing metadata coverage: Mobula vs Codex vs Jupiter") + fmt.Println(" Fields tracked: Logo, Name, Symbol, Description, Twitter, Website, Telegram") + fmt.Println(" Note: Jupiter only supports Solana and has no description/socials") + fmt.Println(" Waiting for new tokens from Pulse stream...") + fmt.Println() + + // Stats printer ticker - print every 5 minutes + statsTicker := time.NewTicker(5 * time.Minute) + defer statsTicker.Stop() + + for { + select { + case <-stopChan: + fmt.Println("Metadata Coverage monitor stopped") + printCoverageStats() // Print final stats + return + + case token := <-tokenQueue: + // Small delay to let the token get indexed + time.Sleep(2 * time.Second) + checkTokenMetadata(token, config) + + case <-statsTicker.C: + printCoverageStats() + } + } +} + diff --git a/harnesses/aggregator-head-lag/cmd/script/metrics.go b/harnesses/aggregator-head-lag/cmd/script/metrics.go new file mode 100644 index 00000000..288e1b16 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/metrics.go @@ -0,0 +1,421 @@ +package main + +import ( + "fmt" + "net/http" + "sync" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +var ( + // Pool discovery latency metric + poolDiscoveryLatency *prometheus.GaugeVec + poolDiscoveryErrors *prometheus.CounterVec + + // REST API latency metrics + restAPILatency *prometheus.HistogramVec + restAPIErrors *prometheus.CounterVec + restAPIStatusCodes *prometheus.CounterVec + + // Quote API latency metrics + quoteAPILatency *prometheus.HistogramVec + quoteAPIErrors *prometheus.CounterVec + quoteAPIStatusCodes *prometheus.CounterVec + + // Metadata coverage metrics + metadataCoverageTotal *prometheus.CounterVec + metadataCoverageSuccess *prometheus.CounterVec + metadataAPILatency *prometheus.HistogramVec + + // Head lag metrics + headLagBlocks *prometheus.GaugeVec + headLagSeconds *prometheus.GaugeVec + blockchainHead *prometheus.GaugeVec + aggregatorHead *prometheus.GaugeVec + headLagErrors *prometheus.CounterVec + + // Fast-trade latency (for comparison with Pulse V2) + fastTradeLatency *prometheus.GaugeVec + + // Mobula detailed head lag (with breakdown) + mobulaHeadLagDetailed *prometheus.GaugeVec + mobulaProcessingLagSeconds *prometheus.GaugeVec + mobulaNetworkLagSeconds *prometheus.GaugeVec + + // Latest tx_hash seen per pool. Kept at most 1 series per (chain, pool_address) + // via explicit Delete of the previous label set in RecordMobulaLastTx. + // Alert annotations query this gauge to include a clickable tx link. + mobulaLastTxHash *prometheus.GaugeVec + mobulaLastTxMu sync.Mutex + mobulaLastTxSeen = make(map[string]string) // key: "chain:pool_address" -> last tx_hash +) + +func init() { + poolDiscoveryLatency = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "pool_discovery_latency_milliseconds", + Help: "Time from pool creation on-chain to first trade detection (pool discovery latency)", + }, + []string{"aggregator", "chain", "region"}, + ) + prometheus.MustRegister(poolDiscoveryLatency) + + poolDiscoveryErrors = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "pool_discovery_errors_total", + Help: "Total number of errors when fetching pool discovery data", + }, + []string{"aggregator", "error_type", "region"}, + ) + prometheus.MustRegister(poolDiscoveryErrors) + + // REST API latency histogram with buckets optimized for API response times + restAPILatency = prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Name: "rest_api_latency_milliseconds", + Help: "REST API response latency in milliseconds", + Buckets: []float64{50, 100, 200, 500, 1000, 2000, 5000, 10000}, + }, + []string{"aggregator", "endpoint", "chain", "region"}, + ) + prometheus.MustRegister(restAPILatency) + + // REST API errors counter + restAPIErrors = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "rest_api_errors_total", + Help: "Total number of REST API errors", + }, + []string{"aggregator", "endpoint", "chain", "error_type", "region"}, + ) + prometheus.MustRegister(restAPIErrors) + + // REST API status codes counter + restAPIStatusCodes = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "rest_api_status_codes_total", + Help: "Total count of REST API responses by status code", + }, + []string{"aggregator", "endpoint", "chain", "status_code", "region"}, + ) + prometheus.MustRegister(restAPIStatusCodes) + + // Quote API latency histogram + quoteAPILatency = prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Name: "quote_api_latency_milliseconds", + Help: "Quote API response latency in milliseconds", + Buckets: []float64{50, 100, 200, 300, 500, 750, 1000, 1500, 2000, 3000, 5000}, + }, + []string{"provider", "chain", "region"}, + ) + prometheus.MustRegister(quoteAPILatency) + + // Quote API errors counter + quoteAPIErrors = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "quote_api_errors_total", + Help: "Total number of Quote API errors", + }, + []string{"provider", "chain", "error_type", "region"}, + ) + prometheus.MustRegister(quoteAPIErrors) + + // Quote API status codes counter + quoteAPIStatusCodes = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "quote_api_status_codes_total", + Help: "Total count of Quote API responses by status code", + }, + []string{"provider", "chain", "status_code", "region"}, + ) + prometheus.MustRegister(quoteAPIStatusCodes) + + // Metadata coverage - total checks per provider/chain/field + metadataCoverageTotal = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "metadata_coverage_checks_total", + Help: "Total number of metadata coverage checks", + }, + []string{"provider", "chain", "field", "region"}, + ) + prometheus.MustRegister(metadataCoverageTotal) + + // Metadata coverage - successful (field present) checks + metadataCoverageSuccess = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "metadata_coverage_success_total", + Help: "Total number of successful metadata coverage checks (field present)", + }, + []string{"provider", "chain", "field", "region"}, + ) + prometheus.MustRegister(metadataCoverageSuccess) + + // Metadata API latency + metadataAPILatency = prometheus.NewHistogramVec( + prometheus.HistogramOpts{ + Name: "metadata_api_latency_milliseconds", + Help: "Metadata API response latency in milliseconds", + Buckets: []float64{50, 100, 200, 500, 1000, 2000, 5000, 10000}, + }, + []string{"provider", "chain", "region"}, + ) + prometheus.MustRegister(metadataAPILatency) + + // Head lag - milliseconds behind (raw value) + headLagBlocks = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "head_lag_milliseconds", + Help: "Indexation latency in milliseconds (time between on-chain event and WebSocket receipt)", + }, + []string{"aggregator", "chain", "region"}, + ) + prometheus.MustRegister(headLagBlocks) + + // Head lag - seconds behind (converted from ms) + headLagSeconds = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "head_lag_seconds", + Help: "Indexation latency in seconds (time between on-chain event and WebSocket receipt)", + }, + []string{"aggregator", "chain", "region"}, + ) + prometheus.MustRegister(headLagSeconds) + + // Blockchain head block number (source of truth) + blockchainHead = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "blockchain_head_block", + Help: "Latest block number on the blockchain (source of truth)", + }, + []string{"chain", "region"}, + ) + prometheus.MustRegister(blockchainHead) + + // Aggregator head block number (what they have indexed) + aggregatorHead = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "aggregator_head_block", + Help: "Latest block number indexed by the aggregator", + }, + []string{"aggregator", "chain", "region"}, + ) + prometheus.MustRegister(aggregatorHead) + + // Head lag errors counter + headLagErrors = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "head_lag_errors_total", + Help: "Total number of errors when fetching head lag data", + }, + []string{"aggregator", "chain", "error_type", "region"}, + ) + prometheus.MustRegister(headLagErrors) + + // Fast-trade latency (separate from head_lag for comparison) + fastTradeLatency = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "fast_trade_latency_milliseconds", + Help: "Fast-trade WebSocket latency in milliseconds (for comparison with Pulse V2)", + }, + []string{"aggregator", "chain", "region"}, + ) + prometheus.MustRegister(fastTradeLatency) + + // Mobula head lag detailed (fixed-cardinality; breakdown exposed as separate gauges) + mobulaHeadLagDetailed = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "mobula_head_lag_detailed_seconds", + Help: "Mobula total head lag in seconds (on-chain -> WebSocket receipt)", + }, + []string{"aggregator", "chain", "region", "pool_address"}, + ) + prometheus.MustRegister(mobulaHeadLagDetailed) + + mobulaProcessingLagSeconds = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "mobula_processing_lag_seconds", + Help: "Mobula processing latency (on-chain -> Mobula processed)", + }, + []string{"aggregator", "chain", "region", "pool_address"}, + ) + prometheus.MustRegister(mobulaProcessingLagSeconds) + + mobulaNetworkLagSeconds = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "mobula_network_lag_seconds", + Help: "Network latency Mobula processed -> WebSocket receipt", + }, + []string{"aggregator", "chain", "region", "pool_address"}, + ) + prometheus.MustRegister(mobulaNetworkLagSeconds) + + // Gauge value is always 1; tx_hash is carried as a label so alert templates + // can query the latest tx per pool. Exactly ONE series per (chain, pool_address) + // is kept alive at any time — see RecordMobulaLastTx for the delete/set dance. + mobulaLastTxHash = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Name: "mobula_last_tx_hash", + Help: "Latest Mobula trade tx_hash per pool (value=1, tx_hash label rotated)", + }, + []string{"aggregator", "chain", "region", "pool_address", "tx_hash"}, + ) + prometheus.MustRegister(mobulaLastTxHash) +} + +// RecordMobulaLastTx overwrites the single live series for (chain, pool_address) +// with a new tx_hash label. Prevents cardinality leak by deleting the previous +// label set before setting the new one. +func RecordMobulaLastTx(chain, region, poolAddress, txHash string) { + if txHash == "" { + return + } + key := chain + ":" + poolAddress + mobulaLastTxMu.Lock() + defer mobulaLastTxMu.Unlock() + + if prev, ok := mobulaLastTxSeen[key]; ok && prev != txHash { + mobulaLastTxHash.DeleteLabelValues("mobula", chain, region, poolAddress, prev) + } + mobulaLastTxHash.WithLabelValues("mobula", chain, region, poolAddress, txHash).Set(1) + mobulaLastTxSeen[key] = txHash +} + +func RecordPoolDiscoveryLatency(aggregator string, chain string, latencyMs float64, region string) { + // Filter out invalid values: negative or > 2 minutes (120000ms) + if latencyMs < 0 || latencyMs > 120000 { + return + } + + poolDiscoveryLatency.WithLabelValues(aggregator, chain, region).Set(latencyMs) +} + +// RecordPoolDiscoveryError records an error when fetching pool discovery data +func RecordPoolDiscoveryError(aggregator string, errorType string, region string) { + poolDiscoveryErrors.WithLabelValues(aggregator, errorType, region).Inc() +} + +// RecordRESTLatency records the latency of a REST API call +func RecordRESTLatency(aggregator string, endpoint string, chain string, latencyMs float64, statusCode int, region string) { + // Record latency in histogram + restAPILatency.WithLabelValues(aggregator, endpoint, chain, region).Observe(latencyMs) + + // Record status code + restAPIStatusCodes.WithLabelValues(aggregator, endpoint, chain, fmt.Sprintf("%d", statusCode), region).Inc() +} + +// RecordRESTError records a REST API error +func RecordRESTError(aggregator string, endpoint string, chain string, errorType string, region string) { + restAPIErrors.WithLabelValues(aggregator, endpoint, chain, errorType, region).Inc() +} + +// RecordQuoteAPILatency records the latency of a Quote API call +func RecordQuoteAPILatency(provider string, chain string, latencyMs float64, statusCode int, region string) { + // Record latency in histogram + quoteAPILatency.WithLabelValues(provider, chain, region).Observe(latencyMs) + + // Record status code + quoteAPIStatusCodes.WithLabelValues(provider, chain, fmt.Sprintf("%d", statusCode), region).Inc() +} + +// RecordQuoteAPIError records a Quote API error +func RecordQuoteAPIError(provider string, chain string, errorType string, region string) { + quoteAPIErrors.WithLabelValues(provider, chain, errorType, region).Inc() +} + +// RecordMetadataCoverage records metadata coverage for a specific field +func RecordMetadataCoverage(provider string, chain string, field string, present bool, region string) { + metadataCoverageTotal.WithLabelValues(provider, chain, field, region).Inc() + if present { + metadataCoverageSuccess.WithLabelValues(provider, chain, field, region).Inc() + } +} + +// RecordMetadataLatency records the latency of a metadata API call +func RecordMetadataLatency(provider string, chain string, latencyMs float64, region string) { + metadataAPILatency.WithLabelValues(provider, chain, region).Observe(latencyMs) +} + +// RecordHeadLag records the head lag for an aggregator on a specific chain +func RecordHeadLag(aggregator string, chain string, lagBlocks int64, lagSeconds float64, region string, txHash string) { + // Filter out aberrant values (> 30s likely means connection issues, not real lag) + if lagSeconds < 0 || lagSeconds > 30 { + return + } + + headLagBlocks.WithLabelValues(aggregator, chain, region).Set(float64(lagBlocks)) + headLagSeconds.WithLabelValues(aggregator, chain, region).Set(lagSeconds) + // tx_hash is logged but not stored as a metric label to avoid cardinality explosion +} + +// RecordBlockchainHead records the current blockchain head block number +func RecordBlockchainHead(chain string, blockNumber int64, region string) { + blockchainHead.WithLabelValues(chain, region).Set(float64(blockNumber)) +} + +// RecordAggregatorHead records the aggregator's indexed head block number +func RecordAggregatorHead(aggregator string, chain string, blockNumber int64, region string) { + aggregatorHead.WithLabelValues(aggregator, chain, region).Set(float64(blockNumber)) +} + +// RecordHeadLagError records an error when fetching head lag data +func RecordHeadLagError(aggregator string, chain string, errorType string, region string) { + headLagErrors.WithLabelValues(aggregator, chain, errorType, region).Inc() +} + +// RecordCodexBlockNumber records the block number from Codex events +func RecordCodexBlockNumber(chain string, blockNumber int64, region string) { + aggregatorHead.WithLabelValues("codex", chain, region).Set(float64(blockNumber)) +} + +// RecordFastTradeLatency records fast-trade WebSocket latency +func RecordFastTradeLatency(aggregator string, chain string, latencyMs float64, region string) { + // Filter out invalid values + if latencyMs < 0 || latencyMs > 120000 { + return + } + fastTradeLatency.WithLabelValues(aggregator, chain, region).Set(latencyMs) +} + +// RecordMobulaHeadLagDetailed records Mobula head lag with breakdown on fixed-cardinality gauges +func RecordMobulaHeadLagDetailed( + chain string, + region string, + poolAddress string, + txHash string, + totalLagMs int64, + mobulaProcessingMs int64, + networkLatencyMs int64, + onChainTime string, + mobulaTime string, + receivedTime string, +) { + // Drop replays/clock-skew: > 30s is not a real indexation latency + if totalLagMs < 0 || totalLagMs > 30000 { + return + } + + mobulaHeadLagDetailed.WithLabelValues("mobula", chain, region, poolAddress). + Set(float64(totalLagMs) / 1000.0) + mobulaProcessingLagSeconds.WithLabelValues("mobula", chain, region, poolAddress). + Set(float64(mobulaProcessingMs) / 1000.0) + mobulaNetworkLagSeconds.WithLabelValues("mobula", chain, region, poolAddress). + Set(float64(networkLatencyMs) / 1000.0) +} + +func StartMetricsServer(addr string) error { + mux := http.NewServeMux() + + // Prometheus metrics endpoint + mux.Handle("/metrics", promhttp.Handler()) + + // Admin cleanup endpoint + setupCleanupEndpoint(mux) + + // Debug: tail of in-memory log ring + setupLogsEndpoint(mux) + + return http.ListenAndServe(addr, mux) +} diff --git a/harnesses/aggregator-head-lag/cmd/script/mobula_fast_trade_monitor.go b/harnesses/aggregator-head-lag/cmd/script/mobula_fast_trade_monitor.go new file mode 100644 index 00000000..9629ef2e --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/mobula_fast_trade_monitor.go @@ -0,0 +1,225 @@ +package main + +import ( + "encoding/json" + "fmt" + "time" + + _ "github.com/gorilla/websocket" // Used by proxy.go getProxyDialer +) + +// ============================================================================ +// Mobula Fast-Trade Monitor (separate from head lag) +// Monitors fast-trade latency for comparison with Pulse V2 +// ============================================================================ + +type MobulaFastTradeEvent struct { + Blockchain string `json:"blockchain"` + Date int64 `json:"date"` // On-chain timestamp (ms) + Timestamp int64 `json:"timestamp"` // Mobula processing timestamp (ms) + Hash string `json:"hash"` + Pair string `json:"pair"` + Address string `json:"address"` // Pool address + Type string `json:"type"` + TokenPrice float64 `json:"tokenPrice"` +} + +// Monitored pools (same chains as Pulse V2) +var fastTradePools = []struct { + Blockchain string + Address string + ChainName string +}{ + { + Blockchain: "solana", + Address: "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm", // SOL/USDC Raydium + ChainName: "solana", + }, + { + Blockchain: "evm:8453", + Address: "0x4c36388be6f416a29c8d8eee81c771ce6be14b18", // WETH/USDC Base + ChainName: "base", + }, + { + Blockchain: "evm:56", + Address: "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16", // WBNB/BUSD PancakeSwap + ChainName: "bnb", + }, +} + +func runMobulaFastTradeMonitor(config *Config, stopChan <-chan struct{}) { + fmt.Println() + fmt.Println("╔═══════════════════════════════════════════════════════════╗") + fmt.Println("║ MOBULA FAST-TRADE MONITOR (Comparison) ║") + fmt.Println("╠═══════════════════════════════════════════════════════════╣") + fmt.Println("║ Purpose: Compare with Pulse V2 discovery latency ║") + fmt.Println("║ Monitoring: High-volume pools on 4 chains ║") + fmt.Printf("║ Pools: %d monitored pools ║\n", len(fastTradePools)) + fmt.Println("╚═══════════════════════════════════════════════════════════╝") + fmt.Println() + + if config.MobulaAPIKey == "" { + fmt.Println("[FAST-TRADE][MOBULA] API key not set, skipping") + return + } + + reconnectDelay := 5 * time.Second + maxReconnectDelay := 60 * time.Second + + for { + select { + case <-stopChan: + fmt.Println("[FAST-TRADE][MOBULA] Monitor stopped") + return + default: + err := connectAndMonitorFastTrade(config, stopChan) + if err != nil { + fmt.Printf("[FAST-TRADE][MOBULA] Connection error: %v. Reconnecting in %v...\n", err, reconnectDelay) + + select { + case <-stopChan: + return + case <-time.After(reconnectDelay): + reconnectDelay = reconnectDelay * 2 + if reconnectDelay > maxReconnectDelay { + reconnectDelay = maxReconnectDelay + } + } + } else { + reconnectDelay = 5 * time.Second + } + } + } +} + +func connectAndMonitorFastTrade(config *Config, stopChan <-chan struct{}) error { + conn, _, err := getProxyDialer().Dial(config.MobulaWSURL, nil) + if err != nil { + return fmt.Errorf("dial failed: %w", err) + } + defer conn.Close() + + // Build subscription items + var items []map[string]interface{} + for _, pool := range fastTradePools { + items = append(items, map[string]interface{}{ + "blockchain": pool.Blockchain, + "address": pool.Address, + }) + } + + // Subscribe to fast-trade + subscribeMsg := map[string]interface{}{ + "type": "fast-trade", + "authorization": config.MobulaAPIKey, + "payload": map[string]interface{}{ + "assetMode": false, + "items": items, + }, + } + + if err := conn.WriteJSON(subscribeMsg); err != nil { + return fmt.Errorf("subscribe failed: %w", err) + } + + fmt.Printf("[FAST-TRADE][MOBULA] Subscribed to %d pools\n", len(items)) + + // Start ping goroutine + pingDone := make(chan struct{}) + go func() { + ticker := time.NewTicker(25 * time.Second) + defer ticker.Stop() + for { + select { + case <-pingDone: + return + case <-ticker.C: + if err := conn.WriteJSON(map[string]string{"event": "ping"}); err != nil { + return + } + } + } + }() + defer close(pingDone) + + // Read messages + for { + select { + case <-stopChan: + return nil + default: + conn.SetReadDeadline(time.Now().Add(60 * time.Second)) + _, message, err := conn.ReadMessage() + if err != nil { + return fmt.Errorf("read failed: %w", err) + } + + // Parse message + var trade MobulaFastTradeEvent + if err := json.Unmarshal(message, &trade); err != nil { + continue + } + + // Skip non-trade messages (pong, etc) + if trade.Hash == "" || trade.Date == 0 { + continue + } + + // Calculate fast-trade latency + receiveTime := time.Now().UTC() + onChainTime := time.UnixMilli(trade.Date) + mobulaProcessTime := time.UnixMilli(trade.Timestamp) + + // Total latency: on-chain → WebSocket receipt + totalLagMs := receiveTime.Sub(onChainTime).Milliseconds() + + // Mobula processing latency: on-chain → Mobula processed + mobulaLagMs := mobulaProcessTime.Sub(onChainTime).Milliseconds() + + // Network latency: Mobula processed → WebSocket receipt + networkLagMs := receiveTime.Sub(mobulaProcessTime).Milliseconds() + + lagSeconds := float64(totalLagMs) / 1000.0 + + // Get chain name + chainName := getChainNameFromBlockchainFastTrade(trade.Blockchain) + + // Record metric with "fast-trade" source label + RecordFastTradeLatency("mobula", chainName, float64(totalLagMs), config.MonitorRegion) + + // Enhanced logging for spikes + if totalLagMs > 3000 { + timestamp := receiveTime.Format("15:04:05") + fmt.Printf("[FAST-TRADE][MOBULA][%s][%s] 🚨 SPIKE DETECTED!\n", timestamp, chainName) + fmt.Printf(" Total Latency: %.2fs (%.0fms)\n", lagSeconds, float64(totalLagMs)) + fmt.Printf(" ├─ Mobula Processing: %.0fms (on-chain → Mobula)\n", float64(mobulaLagMs)) + fmt.Printf(" └─ Network Latency: %.0fms (Mobula → WebSocket)\n", float64(networkLagMs)) + fmt.Printf(" Tx: %s\n", trade.Hash) + fmt.Printf(" Pool: %s\n", trade.Address) + fmt.Printf(" On-chain time: %s\n", onChainTime.Format("15:04:05.000")) + fmt.Printf(" Mobula processed: %s\n", mobulaProcessTime.Format("15:04:05.000")) + fmt.Printf(" Received: %s\n", receiveTime.Format("15:04:05.000")) + } else if time.Now().Second()%30 == 0 { + // Log normal latency occasionally + timestamp := receiveTime.Format("15:04:05") + fmt.Printf("[FAST-TRADE][MOBULA][%s][%s] Latency: %.2fs | Tx: %s\n", + timestamp, chainName, lagSeconds, trade.Hash[:12]+"...") + } + } + } +} + +func getChainNameFromBlockchainFastTrade(blockchain string) string { + switch blockchain { + case "Ethereum", "evm:1": + return "ethereum" + case "Solana", "solana": + return "solana" + case "Base", "evm:8453": + return "base" + case "BNB Smart Chain (BEP20)", "BSC", "evm:56": + return "bnb" + default: + return blockchain + } +} diff --git a/harnesses/aggregator-head-lag/cmd/script/mobula_rest_monitor.go b/harnesses/aggregator-head-lag/cmd/script/mobula_rest_monitor.go new file mode 100644 index 00000000..52315efe --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/mobula_rest_monitor.go @@ -0,0 +1,185 @@ +package main + +import ( + "encoding/json" + "fmt" + "io" + "log" + "net/http" + "time" +) + +const ( + mobulaRESTBaseURL = "https://api.mobula.io" +) + +// Chains for REST monitoring - aligned with all monitors +var mobulaRESTChains = []struct { + blockchain string + blockchainID string + chainName string + poolAddress string +}{ + {"Solana", "solana", "solana", "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm"}, // SOL/USDC + {"Base", "8453", "base", "0x4c36388be6f416a29c8d8eee81c771ce6be14b18"}, // WETH/USDC Base + {"BSC", "56", "bnb", "0x58F876857a02D6762E0101bb5C46A8c1ED44Dc16"}, // WBNB/BUSD PancakeSwap +} + +type MobulaMarketDataResponse struct { + Data []struct { + Volume float64 `json:"volume"` + Open float64 `json:"open"` + High float64 `json:"high"` + Low float64 `json:"low"` + Close float64 `json:"close"` + Time int64 `json:"time"` + } `json:"data"` +} + +// callMobulaMarketDataAPI makes a REST call to Mobula's market history/pair endpoint +func callMobulaMarketDataAPI(apiKey string, poolAddress string, blockchain string, chainName string) (float64, int, error) { + endpoint := fmt.Sprintf("%s/api/1/market/history/pair", mobulaRESTBaseURL) + + // Create HTTP client with timeout + client := &http.Client{ + Timeout: 10 * time.Second, + } + + // Build request + req, err := http.NewRequest("GET", endpoint, nil) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + + // Add query parameters + // Get last 1 hour of data with 1 minute candles + to := time.Now().UnixMilli() + from := time.Now().Add(-1 * time.Hour).UnixMilli() + + q := req.URL.Query() + q.Add("address", poolAddress) + q.Add("blockchain", blockchain) + q.Add("period", "1min") + q.Add("from", fmt.Sprintf("%d", from)) + q.Add("to", fmt.Sprintf("%d", to)) + q.Add("amount", "5") // Just get 5 candles, we don't care about data + req.URL.RawQuery = q.Encode() + + // Add headers + req.Header.Set("Authorization", apiKey) + req.Header.Set("Content-Type", "application/json") + + // Measure latency + startTime := time.Now() + resp, err := client.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + // Read response body for debugging + body, _ := io.ReadAll(resp.Body) + + // Try to parse response + var marketData MobulaMarketDataResponse + if err := json.Unmarshal(body, &marketData); err != nil { + // Not a critical error, we still measured latency + log.Printf("[MOBULA-REST][%s] Response parse warning: %v (status: %d)", chainName, err, resp.StatusCode) + } + + return latencyMs, resp.StatusCode, nil +} + +// monitorMobulaREST continuously monitors Mobula REST API latency +func monitorMobulaREST(config *Config, stopChan <-chan struct{}) { + fmt.Println("Starting Mobula REST API monitor...") + fmt.Printf(" Monitoring %d chains with 20s interval\n", len(mobulaRESTChains)) + fmt.Printf(" Endpoint: /api/1/market/history/pair\n") + fmt.Println() + + if config.MobulaAPIKey == "" { + fmt.Println("MOBULA_API_KEY not set in .env file. Skipping Mobula REST monitor.") + return + } + + // Create ticker for 20 second intervals + ticker := time.NewTicker(20 * time.Second) + defer ticker.Stop() + + // Run once immediately + performMobulaRESTChecks(config) + + // Then run every 20 seconds + for { + select { + case <-stopChan: + fmt.Println("Mobula REST monitor stopped") + return + case <-ticker.C: + performMobulaRESTChecks(config) + } + } +} + +// performMobulaRESTChecks performs REST API calls to all chains +func performMobulaRESTChecks(config *Config) { + timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") + + for _, chain := range mobulaRESTChains { + latencyMs, statusCode, err := callMobulaMarketDataAPI( + config.MobulaAPIKey, + chain.poolAddress, + chain.blockchainID, + chain.chainName, + ) + + if err != nil { + // Record error + errorType := "request_error" + if statusCode >= 500 { + errorType = "server_error" + } else if statusCode >= 400 { + errorType = "client_error" + } else if statusCode == 0 { + errorType = "timeout_error" + } + + RecordRESTError("mobula", "market_data", chain.chainName, errorType, config.MonitorRegion) + + fmt.Printf("[MOBULA-REST][%s][%s] ERROR | Latency: %.0fms | Status: %d | Error: %v\n", + timestamp, + chain.chainName, + latencyMs, + statusCode, + err, + ) + continue + } + + // Record successful latency measurement + RecordRESTLatency("mobula", "market_data", chain.chainName, latencyMs, statusCode, config.MonitorRegion) + + // Log the result + statusEmoji := "✓" + if statusCode >= 400 { + statusEmoji = "✗" + } else if statusCode >= 300 { + statusEmoji = "⚠" + } + + fmt.Printf("[MOBULA-REST][%s][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, + chain.chainName, + statusEmoji, + latencyMs, + statusCode, + ) + } +} + +// runMobulaRESTMonitor is the entry point for the Mobula REST monitor +func runMobulaRESTMonitor(config *Config, stopChan <-chan struct{}) { + monitorMobulaREST(config, stopChan) +} diff --git a/harnesses/aggregator-head-lag/cmd/script/moralis_rest_monitor.go b/harnesses/aggregator-head-lag/cmd/script/moralis_rest_monitor.go new file mode 100644 index 00000000..6a646883 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/moralis_rest_monitor.go @@ -0,0 +1,219 @@ +package main + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "sync" + "time" +) + +// ============================================================================ +// Moralis REST API Monitor +// Triggered by WebSocket trades to measure indexation lag +// ============================================================================ + +type MoralisOHLCVResponse struct { + PairAddress string `json:"pairAddress"` + Result []struct { + Timestamp string `json:"timestamp"` + Close float64 `json:"close"` + Volume float64 `json:"volume"` + Trades int `json:"trades"` + } `json:"result"` +} + +type MoralisMonitorPool struct { + Name string + Chain string // Chain name for metrics + ChainID string // Chain ID for Moralis API (hex for EVM, "solana" for Solana) + PairAddress string + IsEVM bool +} + +// Map WebSocket pool addresses to Moralis pairs +var moralisPairMapping = map[string]MoralisMonitorPool{ + // Solana + "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm": { + Name: "SOL/USDC Raydium", + Chain: "solana", + ChainID: "solana", + PairAddress: "7qbRF6YsyGuLUVs6Y1q64bdVrfe4ZcUUz1JRdoVNUJnm", + IsEVM: false, + }, + // Base + "0x4c36388be6f416a29c8d8eee81c771ce6be14b18": { + Name: "WETH/USDC Base", + Chain: "base", + ChainID: "0x2105", + PairAddress: "0x4c36388be6f416a29c8d8eee81c771ce6be14b18", + IsEVM: true, + }, + // BNB + "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16": { + Name: "WBNB/BUSD PancakeSwap", + Chain: "bnb", + ChainID: "0x38", + PairAddress: "0x58f876857a02d6762e0101bb5c46a8c1ed44dc16", + IsEVM: true, + }, +} + +var ( + moralisCheckQueue = make(chan TradeCheckRequest, 1000) + moralisHttpClient = &http.Client{Timeout: 10 * time.Second} +) + +type TradeCheckRequest struct { + PairAddress string + OnChainTime time.Time + TransactionHash string +} + +func runMoralisRESTMonitor(config *Config, stopChan <-chan struct{}, wg *sync.WaitGroup) { + defer wg.Done() + + fmt.Println("[HEAD-LAG][MORALIS-REST] Starting triggered REST monitor...") + fmt.Println("[HEAD-LAG][MORALIS-REST] Will check Moralis API when trades arrive via WebSocket") + + // Start worker to process check requests + for { + select { + case <-stopChan: + fmt.Println("[HEAD-LAG][MORALIS-REST] Monitor stopped") + return + case req := <-moralisCheckQueue: + checkMoralisForTrade(config, req) + } + } +} + +// TriggerMoralisCheck is called when a trade is detected via WebSocket +// It queues a check to see if Moralis has indexed it yet +func TriggerMoralisCheck(pairAddress string, onChainTime time.Time, txHash string) { + // Normalize address + pairAddress = strings.ToLower(pairAddress) + + // Check if we monitor this pair + if _, exists := moralisPairMapping[pairAddress]; !exists { + return + } + + select { + case moralisCheckQueue <- TradeCheckRequest{ + PairAddress: pairAddress, + OnChainTime: onChainTime, + TransactionHash: txHash, + }: + default: + // Queue full, skip + } +} + +func checkMoralisForTrade(config *Config, req TradeCheckRequest) { + pool, exists := moralisPairMapping[req.PairAddress] + if !exists { + return + } + + // Skip if no Moralis API key configured + // TODO: Add MORALIS_API_KEY to config + // For now, skip Moralis checks silently + return + + // Build URL using correct Moralis Web3 Data API + url := fmt.Sprintf("https://deep-index.moralis.io/api/v2.2/pairs/%s/ohlcv", pool.PairAddress) + + // Query from slightly before the on-chain trade to now + toDate := time.Now().UTC() + fromDate := req.OnChainTime.Add(-2 * time.Minute) // Start 2 minutes before trade + + httpReq, err := http.NewRequest("GET", url, nil) + if err != nil { + RecordHeadLagError("moralis", pool.Chain, "request_creation_failed", config.MonitorRegion) + return + } + + q := httpReq.URL.Query() + if pool.IsEVM { + q.Add("chain", pool.ChainID) + } + q.Add("to_date", fmt.Sprintf("%d", toDate.Unix())) + q.Add("from_date", fmt.Sprintf("%d", fromDate.Unix())) + q.Add("timeframe", "1m") + httpReq.URL.RawQuery = q.Encode() + + // Set headers with API key + // httpReq.Header.Set("X-API-Key", config.MoralisAPIKey) + httpReq.Header.Set("Accept", "application/json") + + // Make request + checkTime := time.Now() + resp, err := moralisHttpClient.Do(httpReq) + if err != nil { + RecordHeadLagError("moralis", pool.Chain, "request_failed", config.MonitorRegion) + return + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + RecordHeadLagError("moralis", pool.Chain, fmt.Sprintf("http_%d", resp.StatusCode), config.MonitorRegion) + return + } + + // Parse response + body, err := io.ReadAll(resp.Body) + if err != nil { + RecordHeadLagError("moralis", pool.Chain, "read_body_failed", config.MonitorRegion) + return + } + + var data MoralisOHLCVResponse + if err := json.Unmarshal(body, &data); err != nil { + RecordHeadLagError("moralis", pool.Chain, "json_parse_failed", config.MonitorRegion) + return + } + + if len(data.Result) == 0 { + // No data yet - trade not indexed + RecordHeadLagError("moralis", pool.Chain, "trade_not_found", config.MonitorRegion) + return + } + + // Find the candle that contains our on-chain trade time + // The candle timestamp is the START of the 1-minute window + tradeMinute := req.OnChainTime.Truncate(time.Minute) + + found := false + for _, candle := range data.Result { + candleTime, err := time.Parse("2006-01-02T15:04:05.000Z", candle.Timestamp) + if err != nil { + continue + } + + // Check if this candle contains our trade + // Candle at 09:05:00 contains trades from 09:05:00 to 09:05:59 + if candleTime.Equal(tradeMinute) || candleTime.Before(tradeMinute) && candleTime.Add(time.Minute).After(req.OnChainTime) { + // Found! Calculate lag + lagMs := checkTime.Sub(req.OnChainTime).Milliseconds() + lagSeconds := float64(lagMs) / 1000.0 + + // Record metrics with tx hash + RecordHeadLag("moralis", pool.Chain, lagMs, lagSeconds, config.MonitorRegion, req.TransactionHash) + + // Log + fmt.Printf("[HEAD-LAG][MORALIS][%s][%s] Trade found! Lag: %.2fs | Tx: %s | Candle: %s\n", + checkTime.Format("15:04:05"), pool.Chain, lagSeconds, req.TransactionHash[:16], candle.Timestamp) + + found = true + break + } + } + + if !found { + // Trade happened but not in any candle yet + RecordHeadLagError("moralis", pool.Chain, "trade_not_in_candles", config.MonitorRegion) + } +} diff --git a/harnesses/aggregator-head-lag/cmd/script/proxy.go b/harnesses/aggregator-head-lag/cmd/script/proxy.go new file mode 100644 index 00000000..ac6c38aa --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/proxy.go @@ -0,0 +1,103 @@ +package main + +import ( + "context" + "io" + "net" + "net/http" + "net/url" + "os" + "time" + + "github.com/gorilla/websocket" +) + +// getProxyDialer returns a websocket.Dialer configured with proxy from env +// IMPORTANT: Forces new connection for each dial to enable proxy IP rotation +func getProxyDialer() *websocket.Dialer { + dialer := &websocket.Dialer{ + // Force new TCP connection for each dial (no keep-alive reuse) + // This ensures rotating proxy assigns a new IP from the pool + NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) { + netDialer := &net.Dialer{ + Timeout: 30 * time.Second, + KeepAlive: -1, // Disable keep-alive to prevent connection reuse + } + return netDialer.DialContext(ctx, network, addr) + }, + HandshakeTimeout: 30 * time.Second, + } + + // Check for proxy configuration from environment + proxyURL := os.Getenv("HTTP_PROXY") + if proxyURL == "" { + proxyURL = os.Getenv("HTTPS_PROXY") + } + + if proxyURL != "" { + parsedURL, err := url.Parse(proxyURL) + if err == nil { + dialer.Proxy = http.ProxyURL(parsedURL) + } + } + + return dialer +} + +// getProxyDialerWithSubprotocols returns a websocket.Dialer with proxy and custom subprotocols +// Each call creates a FRESH dialer to ensure proxy IP rotation works correctly +func getProxyDialerWithSubprotocols(subprotocols []string) *websocket.Dialer { + dialer := getProxyDialer() + dialer.Subprotocols = subprotocols + return dialer +} + +// getProxyHTTPClient returns an http.Client configured with proxy from env +// IMPORTANT: Creates new client for each call to enable proxy IP rotation +func getProxyHTTPClient() *http.Client { + client := &http.Client{ + Timeout: 10 * time.Second, + Transport: &http.Transport{ + // Disable keep-alive to prevent connection reuse (enables proxy rotation) + DisableKeepAlives: true, + DialContext: (&net.Dialer{ + Timeout: 30 * time.Second, + KeepAlive: -1, // Disable keep-alive + }).DialContext, + }, + } + + // Check for proxy configuration from environment + proxyURL := os.Getenv("HTTP_PROXY") + if proxyURL == "" { + proxyURL = os.Getenv("HTTPS_PROXY") + } + + if proxyURL != "" { + parsedURL, err := url.Parse(proxyURL) + if err == nil { + client.Transport.(*http.Transport).Proxy = http.ProxyURL(parsedURL) + } + } + + return client +} + +// getOutboundIP fetches the current outbound IP address through the proxy +// Used to verify proxy IP rotation is working correctly +func getOutboundIP() (string, error) { + client := getProxyHTTPClient() + + resp, err := client.Get("https://api.ipify.org?format=text") + if err != nil { + return "", err + } + defer resp.Body.Close() + + body, err := io.ReadAll(resp.Body) + if err != nil { + return "", err + } + + return string(body), nil +} diff --git a/harnesses/aggregator-head-lag/cmd/script/quote_api_monitor.go b/harnesses/aggregator-head-lag/cmd/script/quote_api_monitor.go new file mode 100644 index 00000000..f4470f47 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/quote_api_monitor.go @@ -0,0 +1,493 @@ +package main + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "time" +) + +// Quote API endpoints +const ( + // Free APIs (no API key required) + jupiterPublicURL = "https://public.jupiterapi.com/quote" // Free, 10 req/sec, Solana only + mobulaSwapURL = "https://api.mobula.io/api/2/swap/quoting" // Solana only for now + openOceanQuoteURL = "https://open-api.openocean.finance/v3" + paraSwapQuoteURL = "https://apiv5.paraswap.io/prices" + kyberSwapQuoteURL = "https://aggregator-api.kyberswap.com" + lifiQuoteURL = "https://li.quest/v1/quote" +) + +// Dummy wallet addresses for APIs that require fromAddress +const dummyWalletAddressEVM = "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045" // Vitalik's address (EVM) +const dummyWalletAddressSolana = "HN7cABqLq46Es1jh92dQQisAq662SmxELLLsHHe4YWrH" // Random Solana wallet + +// Chain configurations for quote testing +type QuoteChainConfig struct { + Name string + ChainID string // Numeric chain ID + OpenOceanChain string // OpenOcean chain key + KyberChainKey string // KyberSwap chain key + TokenIn string // Input token address + TokenOut string // Output token address + TokenInSymbol string + TokenOutSymbol string + Amount string // Amount in smallest unit + Decimals int +} + +// Solana config for Jupiter +var solanaConfig = QuoteChainConfig{ + Name: "solana", + TokenIn: "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v", // USDC + TokenOut: "So11111111111111111111111111111111111111112", // SOL + TokenInSymbol: "USDC", + TokenOutSymbol: "SOL", + Amount: "100000000", // 100 USDC (6 decimals) + Decimals: 6, +} + +// EVM chains config +var evmQuoteChains = []QuoteChainConfig{ + { + Name: "base", + ChainID: "8453", + OpenOceanChain: "8453", + KyberChainKey: "base", + TokenIn: "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913", // USDC on Base + TokenOut: "0x4200000000000000000000000000000000000006", // WETH on Base + TokenInSymbol: "USDC", + TokenOutSymbol: "WETH", + Amount: "100000000", // 100 USDC (6 decimals) + Decimals: 6, + }, + { + Name: "bnb", + ChainID: "56", + OpenOceanChain: "56", + KyberChainKey: "bsc", + TokenIn: "0x8AC76a51cc950d9822D68b83fE1Ad97B32Cd580d", // USDC on BSC (18 decimals) + TokenOut: "0xbb4CdB9CBd36B01bD1cBaEBF2De08d9173bc095c", // WBNB + TokenInSymbol: "USDC", + TokenOutSymbol: "WBNB", + Amount: "100000000000000000000", // 100 USDC (18 decimals on BSC) + Decimals: 18, + }, +} + +// HTTP client with timeout +var quoteHTTPClient = &http.Client{ + Timeout: 15 * time.Second, +} + +// ============================================================================ +// Mobula Swap Quoting API (Solana + Base + Arbitrum, requires API key) +// ============================================================================ + +func callMobulaSwapQuoteAPI(chainID string, chainName string, tokenIn string, tokenOut string, amount string, apiKey string) (float64, int, error) { + // Use appropriate wallet address based on chain + walletAddress := dummyWalletAddressEVM + if chainName == "solana" { + walletAddress = dummyWalletAddressSolana + } + + params := url.Values{} + params.Add("chainId", chainID) + params.Add("tokenIn", tokenIn) + params.Add("tokenOut", tokenOut) + params.Add("amount", amount) + params.Add("walletAddress", walletAddress) + params.Add("slippage", "1") + + fullURL := fmt.Sprintf("%s?%s", mobulaSwapURL, params.Encode()) + + req, err := http.NewRequest("GET", fullURL, nil) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + req.Header.Set("Accept", "application/json") + if apiKey != "" { + req.Header.Set("Authorization", apiKey) + } + + startTime := time.Now() + resp, err := quoteHTTPClient.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + // Read body to check for errors + body, _ := io.ReadAll(resp.Body) + + // Check for API errors in response body + var result map[string]interface{} + if err := json.Unmarshal(body, &result); err == nil { + if errMsg, ok := result["error"]; ok && errMsg != nil { + // Return 400 to indicate API error (even if HTTP was 200) + return latencyMs, 400, nil + } + } + + return latencyMs, resp.StatusCode, nil +} + +// ============================================================================ +// Jupiter Public API (Solana only, FREE - 10 req/sec) +// ============================================================================ + +func callJupiterPublicQuoteAPI() (float64, int, error) { + params := url.Values{} + params.Add("inputMint", solanaConfig.TokenIn) + params.Add("outputMint", solanaConfig.TokenOut) + params.Add("amount", solanaConfig.Amount) + params.Add("slippageBps", "50") + + fullURL := fmt.Sprintf("%s?%s", jupiterPublicURL, params.Encode()) + + req, err := http.NewRequest("GET", fullURL, nil) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + req.Header.Set("Accept", "application/json") + + startTime := time.Now() + resp, err := quoteHTTPClient.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + _, _ = io.ReadAll(resp.Body) + + return latencyMs, resp.StatusCode, nil +} + +// ============================================================================ +// OpenOcean API (Multi-chain, FREE) +// ============================================================================ + +func callOpenOceanQuoteAPI(chain QuoteChainConfig) (float64, int, error) { + endpoint := fmt.Sprintf("%s/%s/quote", openOceanQuoteURL, chain.OpenOceanChain) + + params := url.Values{} + params.Add("inTokenAddress", chain.TokenIn) + params.Add("outTokenAddress", chain.TokenOut) + params.Add("amount", chain.Amount) + params.Add("gasPrice", "5") + + fullURL := fmt.Sprintf("%s?%s", endpoint, params.Encode()) + + req, err := http.NewRequest("GET", fullURL, nil) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + req.Header.Set("Accept", "application/json") + + startTime := time.Now() + resp, err := quoteHTTPClient.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + _, _ = io.ReadAll(resp.Body) + + return latencyMs, resp.StatusCode, nil +} + +// ============================================================================ +// ParaSwap API (Multi-chain, FREE) +// ============================================================================ + +func callParaSwapQuoteAPI(chain QuoteChainConfig) (float64, int, error) { + params := url.Values{} + params.Add("srcToken", chain.TokenIn) + params.Add("destToken", chain.TokenOut) + params.Add("amount", chain.Amount) + params.Add("srcDecimals", fmt.Sprintf("%d", chain.Decimals)) + params.Add("destDecimals", "18") // Native tokens are 18 decimals + params.Add("network", chain.ChainID) + + fullURL := fmt.Sprintf("%s?%s", paraSwapQuoteURL, params.Encode()) + + req, err := http.NewRequest("GET", fullURL, nil) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + req.Header.Set("Accept", "application/json") + + startTime := time.Now() + resp, err := quoteHTTPClient.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + _, _ = io.ReadAll(resp.Body) + + return latencyMs, resp.StatusCode, nil +} + +// ============================================================================ +// Li.Fi API (Multi-chain, FREE) +// ============================================================================ + +func callLifiQuoteAPI(chain QuoteChainConfig) (float64, int, error) { + params := url.Values{} + params.Add("fromChain", chain.ChainID) + params.Add("toChain", chain.ChainID) // Same chain swap + params.Add("fromToken", chain.TokenIn) + params.Add("toToken", chain.TokenOut) + params.Add("fromAmount", chain.Amount) + params.Add("fromAddress", dummyWalletAddressEVM) // Required by Li.Fi + + fullURL := fmt.Sprintf("%s?%s", lifiQuoteURL, params.Encode()) + + req, err := http.NewRequest("GET", fullURL, nil) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + req.Header.Set("Accept", "application/json") + + startTime := time.Now() + resp, err := quoteHTTPClient.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + _, _ = io.ReadAll(resp.Body) + + return latencyMs, resp.StatusCode, nil +} + +// ============================================================================ +// KyberSwap API (Multi-chain, FREE) +// ============================================================================ + +func callKyberSwapQuoteAPI(chain QuoteChainConfig) (float64, int, error) { + endpoint := fmt.Sprintf("%s/%s/api/v1/routes", kyberSwapQuoteURL, chain.KyberChainKey) + + params := url.Values{} + params.Add("tokenIn", chain.TokenIn) + params.Add("tokenOut", chain.TokenOut) + params.Add("amountIn", chain.Amount) + + fullURL := fmt.Sprintf("%s?%s", endpoint, params.Encode()) + + req, err := http.NewRequest("GET", fullURL, nil) + if err != nil { + return 0, 0, fmt.Errorf("failed to create request: %w", err) + } + req.Header.Set("Accept", "application/json") + + startTime := time.Now() + resp, err := quoteHTTPClient.Do(req) + latencyMs := float64(time.Since(startTime).Milliseconds()) + + if err != nil { + return latencyMs, 0, fmt.Errorf("request failed: %w", err) + } + defer resp.Body.Close() + + _, _ = io.ReadAll(resp.Body) + + return latencyMs, resp.StatusCode, nil +} + + +// ============================================================================ +// Main monitoring function +// ============================================================================ + +func performQuoteAPIChecks(config *Config) { + timestamp := time.Now().UTC().Format("2006-01-02 15:04:05") + + fmt.Printf("\n[QUOTE-API][%s] === Starting quote API latency checks ===\n", timestamp) + + // ========== SOLANA QUOTES ========== + + // Mobula (Solana) + latencyMs, statusCode, err := callMobulaSwapQuoteAPI( + "solana", + "solana", + solanaConfig.TokenIn, + solanaConfig.TokenOut, + "100", // 100 USDC + config.MobulaAPIKey, + ) + if err != nil || statusCode >= 400 { + RecordQuoteAPIError("mobula", "solana", getErrorType(statusCode), config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][mobula][solana] %s | Latency: %.0fms | Status: %d\n", + timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) + } else { + RecordQuoteAPILatency("mobula", "solana", latencyMs, statusCode, config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][mobula][solana] %s | Latency: %.0fms | Status: %d\n", + timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) + } + + // Jupiter (Solana only - FREE public API) + latencyMs, statusCode, err = callJupiterPublicQuoteAPI() + if err != nil || statusCode >= 400 { + RecordQuoteAPIError("jupiter", "solana", getErrorType(statusCode), config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][jupiter][solana] %s | Latency: %.0fms | Status: %d\n", + timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) + } else { + RecordQuoteAPILatency("jupiter", "solana", latencyMs, statusCode, config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][jupiter][solana] %s | Latency: %.0fms | Status: %d\n", + timestamp, getStatusEmoji(statusCode), latencyMs, statusCode) + } + + // ========== EVM QUOTES ========== + + // Test EVM chains with FREE APIs: Mobula (Base), OpenOcean, ParaSwap, Li.Fi, KyberSwap + for _, chain := range evmQuoteChains { + // Mobula (Base - chain where MobulaRouter is deployed) + if chain.Name == "base" { + latencyMs, statusCode, err := callMobulaSwapQuoteAPI( + "evm:"+chain.ChainID, + chain.Name, + chain.TokenIn, + chain.TokenOut, + "100", // 100 USDC + config.MobulaAPIKey, + ) + if err != nil || statusCode >= 400 { + RecordQuoteAPIError("mobula", chain.Name, getErrorType(statusCode), config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][mobula][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } else { + RecordQuoteAPILatency("mobula", chain.Name, latencyMs, statusCode, config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][mobula][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } + } + + // OpenOcean (FREE) + latencyMs, statusCode, err := callOpenOceanQuoteAPI(chain) + if err != nil || statusCode >= 400 { + RecordQuoteAPIError("openocean", chain.Name, getErrorType(statusCode), config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][openocean][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } else { + RecordQuoteAPILatency("openocean", chain.Name, latencyMs, statusCode, config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][openocean][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } + + // ParaSwap (FREE) + latencyMs, statusCode, err = callParaSwapQuoteAPI(chain) + if err != nil || statusCode >= 400 { + RecordQuoteAPIError("paraswap", chain.Name, getErrorType(statusCode), config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][paraswap][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } else { + RecordQuoteAPILatency("paraswap", chain.Name, latencyMs, statusCode, config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][paraswap][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } + + // Li.Fi (FREE) + latencyMs, statusCode, err = callLifiQuoteAPI(chain) + if err != nil || statusCode >= 400 { + RecordQuoteAPIError("lifi", chain.Name, getErrorType(statusCode), config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][lifi][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } else { + RecordQuoteAPILatency("lifi", chain.Name, latencyMs, statusCode, config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][lifi][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } + + // KyberSwap (FREE) + latencyMs, statusCode, err = callKyberSwapQuoteAPI(chain) + if err != nil || statusCode >= 400 { + RecordQuoteAPIError("kyberswap", chain.Name, getErrorType(statusCode), config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][kyberswap][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } else { + RecordQuoteAPILatency("kyberswap", chain.Name, latencyMs, statusCode, config.MonitorRegion) + fmt.Printf("[QUOTE-API][%s][kyberswap][%s] %s | Latency: %.0fms | Status: %d\n", + timestamp, chain.Name, getStatusEmoji(statusCode), latencyMs, statusCode) + } + } + + // Jupiter (Solana) - Requires API key, skip if not available + // TODO: Add JUPITER_API_KEY to config if needed + // latencyMs, statusCode, err := callJupiterQuoteAPI("") + // ... + + fmt.Printf("[QUOTE-API][%s] === Quote API checks completed ===\n\n", timestamp) +} + +func getErrorType(statusCode int) string { + if statusCode >= 500 { + return "server_error" + } else if statusCode >= 400 { + return "client_error" + } else if statusCode == 0 { + return "timeout_error" + } + return "request_error" +} + +func getStatusEmoji(statusCode int) string { + if statusCode >= 400 { + return "✗" + } else if statusCode >= 300 { + return "⚠" + } + return "✓" +} + +// runQuoteAPIMonitor starts the quote API latency monitoring +func runQuoteAPIMonitor(config *Config, stopChan <-chan struct{}) { + fmt.Println("Starting Quote API Latency Monitor...") + fmt.Println(" Comparing: Mobula, Jupiter, OpenOcean, ParaSwap, Li.Fi, KyberSwap") + fmt.Println(" Mobula: Solana + Base") + fmt.Println(" Jupiter: Solana") + fmt.Println(" Others: Ethereum, Base, BNB") + fmt.Println(" Test: 100 USDC → Native token quote") + fmt.Println(" Interval: 30 seconds") + fmt.Println() + + // Create ticker for 30 second intervals + ticker := time.NewTicker(30 * time.Second) + defer ticker.Stop() + + // Run once immediately + performQuoteAPIChecks(config) + + // Then run every 30 seconds + for { + select { + case <-stopChan: + fmt.Println("Quote API monitor stopped") + return + case <-ticker.C: + performQuoteAPIChecks(config) + } + } +} + +// Helper to pretty print JSON for debugging +func prettyPrintJSON(data []byte) { + var prettyJSON map[string]interface{} + if err := json.Unmarshal(data, &prettyJSON); err == nil { + formatted, _ := json.MarshalIndent(prettyJSON, "", " ") + fmt.Printf("%s\n", formatted) + } +} diff --git a/harnesses/aggregator-head-lag/cmd/script/scrape_session.go b/harnesses/aggregator-head-lag/cmd/script/scrape_session.go new file mode 100644 index 00000000..4607912c --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/scrape_session.go @@ -0,0 +1,61 @@ +package main + +import ( + "context" + "fmt" + "time" + + "github.com/chromedp/cdproto/network" + "github.com/chromedp/chromedp" +) + +func ScrapeDefinedSessionCookie() (string, error) { + // Create Chrome context with headless mode + opts := append(chromedp.DefaultExecAllocatorOptions[:], + chromedp.Flag("headless", true), + chromedp.Flag("disable-gpu", true), + chromedp.Flag("no-sandbox", true), + chromedp.Flag("disable-dev-shm-usage", true), + chromedp.UserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"), + ) + + allocCtx, cancel := chromedp.NewExecAllocator(context.Background(), opts...) + defer cancel() + + ctx, cancel := chromedp.NewContext(allocCtx) + defer cancel() + + ctx, cancel = context.WithTimeout(ctx, 30*time.Second) + defer cancel() + + var sessionCookie string + + err := chromedp.Run(ctx, + chromedp.Navigate("https://www.defined.fi/"), + chromedp.WaitVisible(`body`, chromedp.ByQuery), + chromedp.Sleep(5*time.Second), + + chromedp.ActionFunc(func(ctx context.Context) error { + cookieParams, err := network.GetCookies().Do(ctx) + if err != nil { + return fmt.Errorf("failed to get cookies: %w", err) + } + + for _, cookie := range cookieParams { + if cookie.Name == "session" { + sessionCookie = cookie.Value + return nil + } + } + + return fmt.Errorf("session cookie not found") + }), + ) + + if err != nil { + return "", fmt.Errorf("failed to scrape session cookie: %w", err) + } + + return sessionCookie, nil +} + diff --git a/harnesses/aggregator-head-lag/cmd/script/update_metrics_calls.sh b/harnesses/aggregator-head-lag/cmd/script/update_metrics_calls.sh new file mode 100755 index 00000000..bd84baa2 --- /dev/null +++ b/harnesses/aggregator-head-lag/cmd/script/update_metrics_calls.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# Auto-update all RecordX calls to include region parameter + +FILES="head_lag_monitor.go codex_rest_monitor.go mobula_rest_monitor.go quote_api_monitor.go geckoterminal_monitor.go moralis_rest_monitor.go metadata_coverage_monitor.go" + +for file in $FILES; do + if [ -f "$file" ]; then + echo "Updating $file..." + # Use perl for multiline regex + perl -i -pe 's/RecordHeadLag\(([^)]+)\)/RecordHeadLag($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordRESTLatency\(([^)]+)\)/RecordRESTLatency($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordRESTError\(([^)]+)\)/RecordRESTError($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordQuoteAPILatency\(([^)]+)\)/RecordQuoteAPILatency($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordQuoteAPIError\(([^)]+)\)/RecordQuoteAPIError($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordHeadLagError\(([^)]+)\)/RecordHeadLagError($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordCodexBlockNumber\(([^)]+)\)/RecordCodexBlockNumber($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordMetadataCoverage\(([^)]+)\)/RecordMetadataCoverage($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordMetadataLatency\(([^)]+)\)/RecordMetadataLatency($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordPoolDiscoveryLatency\(([^)]+)\)/RecordPoolDiscoveryLatency($1, config.MonitorRegion)/g' "$file" + perl -i -pe 's/RecordPoolDiscoveryError\(([^)]+)\)/RecordPoolDiscoveryError($1, config.MonitorRegion)/g' "$file" + fi +done + +echo "✓ All files updated" diff --git a/harnesses/aggregator-head-lag/docker-compose.yml b/harnesses/aggregator-head-lag/docker-compose.yml new file mode 100644 index 00000000..7536daee --- /dev/null +++ b/harnesses/aggregator-head-lag/docker-compose.yml @@ -0,0 +1,82 @@ +services: + monitor: + build: + context: . + dockerfile: Dockerfile + container_name: latency_monitor + ports: + - "2112:2112" + environment: + - COINGECKO_API_KEY=${COINGECKO_API_KEY} + - MOBULA_API_KEY=${MOBULA_API_KEY} + - DEFINED_SESSION_COOKIE=${DEFINED_SESSION_COOKIE} + networks: + - monitoring + restart: unless-stopped + + prometheus: + image: prom/prometheus:latest + container_name: prometheus + ports: + - "9090:9090" + volumes: + - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml + - ./monitoring/alert_rules.yml:/etc/prometheus/alert_rules.yml + - prometheus_data:/prometheus + command: + - '--config.file=/etc/prometheus/prometheus.yml' + - '--storage.tsdb.path=/prometheus' + - '--web.console.libraries=/etc/prometheus/console_libraries' + - '--web.console.templates=/etc/prometheus/consoles' + - '--web.enable-lifecycle' + networks: + - monitoring + restart: unless-stopped + depends_on: + - monitor + + alertmanager: + image: prom/alertmanager:latest + container_name: alertmanager + ports: + - "9093:9093" + volumes: + - ./monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml + command: + - '--config.file=/etc/alertmanager/alertmanager.yml' + - '--storage.path=/alertmanager' + networks: + - monitoring + restart: unless-stopped + + grafana: + image: grafana/grafana:latest + container_name: grafana + ports: + - "3000:3000" + volumes: + - ./monitoring/grafana/provisioning:/etc/grafana/provisioning + - ./monitoring/grafana/dashboards:/dashboards-source:ro + - ./grafana-entrypoint.sh:/grafana-entrypoint.sh:ro + entrypoint: ["/bin/sh", "/grafana-entrypoint.sh"] + environment: + - GF_SECURITY_ADMIN_USER=admin + - GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD:-admin} + - GF_USERS_ALLOW_SIGN_UP=false + - GF_SERVER_ROOT_URL=${GF_SERVER_ROOT_URL:-http://localhost:3000} + - GF_INSTALL_PLUGINS=grafana-clock-panel + - GF_AUTH_ANONYMOUS_ENABLED=true + - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer + - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/head_lag.json + networks: + - monitoring + depends_on: + - prometheus + restart: unless-stopped + +networks: + monitoring: + driver: bridge + +volumes: + prometheus_data: diff --git a/harnesses/aggregator-head-lag/go.mod b/harnesses/aggregator-head-lag/go.mod new file mode 100644 index 00000000..ed17b543 --- /dev/null +++ b/harnesses/aggregator-head-lag/go.mod @@ -0,0 +1,28 @@ +module mobula_latency_competitor + +go 1.24.4 + +require ( + github.com/chromedp/cdproto v0.0.0-20250724212937-08a3db8b4327 + github.com/chromedp/chromedp v0.14.2 + github.com/gorilla/websocket v1.5.3 + github.com/prometheus/client_golang v1.23.2 +) + +require ( + github.com/beorn7/perks v1.0.1 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/chromedp/sysutil v1.1.0 // indirect + github.com/go-json-experiment/json v0.0.0-20250725192818-e39067aee2d2 // indirect + github.com/gobwas/httphead v0.1.0 // indirect + github.com/gobwas/pool v0.2.1 // indirect + github.com/gobwas/ws v1.4.0 // indirect + github.com/kr/text v0.2.0 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/prometheus/client_model v0.6.2 // indirect + github.com/prometheus/common v0.66.1 // indirect + github.com/prometheus/procfs v0.16.1 // indirect + go.yaml.in/yaml/v2 v2.4.2 // indirect + golang.org/x/sys v0.35.0 // indirect + google.golang.org/protobuf v1.36.8 // indirect +) diff --git a/harnesses/aggregator-head-lag/go.sum b/harnesses/aggregator-head-lag/go.sum new file mode 100644 index 00000000..f594bc89 --- /dev/null +++ b/harnesses/aggregator-head-lag/go.sum @@ -0,0 +1,67 @@ +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/chromedp/cdproto v0.0.0-20250724212937-08a3db8b4327 h1:UQ4AU+BGti3Sy/aLU8KVseYKNALcX9UXY6DfpwQ6J8E= +github.com/chromedp/cdproto v0.0.0-20250724212937-08a3db8b4327/go.mod h1:NItd7aLkcfOA/dcMXvl8p1u+lQqioRMq/SqDp71Pb/k= +github.com/chromedp/chromedp v0.14.2 h1:r3b/WtwM50RsBZHMUm9fsNhhzRStTHrKdr2zmwbZSzM= +github.com/chromedp/chromedp v0.14.2/go.mod h1:rHzAv60xDE7VNy/MYtTUrYreSc0ujt2O1/C3bzctYBo= +github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM= +github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/go-json-experiment/json v0.0.0-20250725192818-e39067aee2d2 h1:iizUGZ9pEquQS5jTGkh4AqeeHCMbfbjeb0zMt0aEFzs= +github.com/go-json-experiment/json v0.0.0-20250725192818-e39067aee2d2/go.mod h1:TiCD2a1pcmjd7YnhGH0f/zKNcCD06B029pHhzV23c2M= +github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU= +github.com/gobwas/httphead v0.1.0/go.mod h1:O/RXo79gxV8G+RqlR/otEwx4Q36zl9rqC5u12GKvMCM= +github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og= +github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw= +github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs= +github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= +github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= +github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= +github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80 h1:6Yzfa6GP0rIo/kULo2bwGEkFvCePZ3qHDDTC3/J9Swo= +github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw= +github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= +github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= +github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= +github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= +github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs= +github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA= +github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg= +github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= +go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI= +golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= +google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc= +google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/harnesses/aggregator-head-lag/grafana-entrypoint.sh b/harnesses/aggregator-head-lag/grafana-entrypoint.sh new file mode 100755 index 00000000..ccf63ca7 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana-entrypoint.sh @@ -0,0 +1,29 @@ +#!/bin/sh + +echo "=== GRAFANA ENTRYPOINT SCRIPT STARTING ===" +echo "Working directory: $(pwd)" +echo "User: $(whoami)" + +# Create dashboards directory +mkdir -p /var/lib/grafana/dashboards + +# Debug: print environment variables +echo "Checking environment..." +env | grep -E '(RAILWAY|HIDE_QUOTE)' || echo "No RAILWAY/HIDE_QUOTE variables found" + +# Copy dashboards from source +if [ "$HIDE_QUOTE_DASHBOARD" = "true" ]; then + echo "HIDE_QUOTE_DASHBOARD=true - hiding Quote API Latency Benchmark dashboard" + cp /dashboards-source/head_lag.json /var/lib/grafana/dashboards/ +else + echo "Copying all dashboards" + cp /dashboards-source/*.json /var/lib/grafana/dashboards/ +fi + +echo "Dashboards copied:" +ls -la /var/lib/grafana/dashboards/ + +echo "=== GRAFANA ENTRYPOINT SCRIPT COMPLETE ===" + +# Start Grafana with default entrypoint +exec /run.sh diff --git a/harnesses/aggregator-head-lag/grafana/Dockerfile b/harnesses/aggregator-head-lag/grafana/Dockerfile new file mode 100644 index 00000000..eb500db9 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/Dockerfile @@ -0,0 +1,34 @@ +FROM grafana/grafana:latest + +USER root + +# Cache buster - update this to force rebuild: v7 +ARG CACHE_BUST=7 + +# Copy provisioning configs and dashboards source (we're in grafana folder) +COPY provisioning /etc/grafana/provisioning +COPY dashboards /dashboards-source + +# Copy entrypoint script +COPY grafana-entrypoint.sh /grafana-entrypoint.sh +RUN chmod +x /grafana-entrypoint.sh + +# Set permissions +RUN chown -R grafana:root /etc/grafana/provisioning /dashboards-source + +USER grafana + +# Environment variables +ENV GF_AUTH_ANONYMOUS_ENABLED=true +ENV GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer +ENV GF_SECURITY_ADMIN_USER=admin +ENV GF_SECURITY_ADMIN_PASSWORD=admin +ENV GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/head_lag.json + +# Prometheus datasource URL (overridden by staging deployment) +ENV PROMETHEUS_URL=http://prometheus.railway.internal:9090 + +EXPOSE 3000 + +# Use custom entrypoint +ENTRYPOINT ["/bin/sh", "/grafana-entrypoint.sh"] diff --git a/harnesses/aggregator-head-lag/grafana/dashboards/disabled/mobula_pulse_vs_fasttrade.json b/harnesses/aggregator-head-lag/grafana/dashboards/disabled/mobula_pulse_vs_fasttrade.json new file mode 100644 index 00000000..373e2f06 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/dashboards/disabled/mobula_pulse_vs_fasttrade.json @@ -0,0 +1,595 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Latency (ms)", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 2000 + }, + { + "color": "orange", + "value": 5000 + }, + { + "color": "red", + "value": 10000 + } + ] + }, + "unit": "ms" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*Pulse.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*Fast-Trade.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 1, + "options": { + "legend": { + "calcs": ["mean", "last", "max", "min"], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Mean", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "pool_discovery_latency_milliseconds{aggregator=\"mobula\"}", + "legendFormat": "[{{region}}] Pulse V2 - {{chain}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "fast_trade_latency_milliseconds{aggregator=\"mobula\"}", + "legendFormat": "[{{region}}] Fast-Trade - {{chain}}", + "range": true, + "refId": "B" + } + ], + "title": "Mobula: Pulse V2 (Discovery) vs Fast-Trade (Swap Indexation)", + "description": "Pulse V2 measures pool discovery latency (on-chain creation → Mobula indexation). Fast-Trade measures swap indexation latency (on-chain swap → WebSocket receipt).", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 2000 + }, + { + "color": "orange", + "value": 5000 + }, + { + "color": "red", + "value": 10000 + } + ] + }, + "unit": "ms" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 10 + }, + "id": 3, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["last"], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "fast_trade_latency_milliseconds{aggregator=\"mobula\"}", + "legendFormat": "[{{region}}] {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Fast-Trade - Current Swap Indexation Latency", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Latency (ms)", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto" + }, + "unit": "ms", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 2000 + }, + { + "color": "red", + "value": 5000 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 10 + }, + "id": 4, + "options": { + "legend": { + "calcs": ["last"], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "pool_discovery_latency_milliseconds{aggregator=\"mobula\",chain=\"solana\"}", + "legendFormat": "[{{region}}] Pulse V2", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "fast_trade_latency_milliseconds{aggregator=\"mobula\",chain=\"solana\"}", + "legendFormat": "[{{region}}] Fast-Trade", + "refId": "B" + } + ], + "title": "Solana - Pulse V2 vs Fast-Trade", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Latency (ms)", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto" + }, + "unit": "ms", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 2000 + }, + { + "color": "red", + "value": 5000 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 10 + }, + "id": 5, + "options": { + "legend": { + "calcs": ["last"], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "pool_discovery_latency_milliseconds{aggregator=\"mobula\",chain=\"base\"}", + "legendFormat": "[{{region}}] Pulse V2", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "fast_trade_latency_milliseconds{aggregator=\"mobula\",chain=\"base\"}", + "legendFormat": "[{{region}}] Fast-Trade", + "refId": "B" + } + ], + "title": "Base - Pulse V2 vs Fast-Trade", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Latency (ms)", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto" + }, + "unit": "ms", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 2000 + }, + { + "color": "red", + "value": 5000 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 10 + }, + "id": 6, + "options": { + "legend": { + "calcs": ["last"], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "pool_discovery_latency_milliseconds{aggregator=\"mobula\",chain=\"ethereum\"}", + "legendFormat": "[{{region}}] Pulse V2", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "fast_trade_latency_milliseconds{aggregator=\"mobula\",chain=\"ethereum\"}", + "legendFormat": "[{{region}}] Fast-Trade", + "refId": "B" + } + ], + "title": "Ethereum - Pulse V2 vs Fast-Trade", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Latency (ms)", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto" + }, + "unit": "ms", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 2000 + }, + { + "color": "red", + "value": 5000 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 18 + }, + "id": 7, + "options": { + "legend": { + "calcs": ["last"], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "pool_discovery_latency_milliseconds{aggregator=\"mobula\",chain=\"bnb\"}", + "legendFormat": "[{{region}}] Pulse V2", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "fast_trade_latency_milliseconds{aggregator=\"mobula\",chain=\"bnb\"}", + "legendFormat": "[{{region}}] Fast-Trade", + "refId": "B" + } + ], + "title": "BNB - Pulse V2 vs Fast-Trade", + "type": "timeseries" + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": ["mobula", "pulse", "fast-trade", "comparison"], + "templating": { + "list": [] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Mobula: Pulse V2 vs Fast-Trade Comparison", + "uid": "mobula_pulse_vs_fasttrade", + "version": 0, + "weekStart": "" +} diff --git a/harnesses/aggregator-head-lag/grafana/dashboards/disabled/pulse_vs_codex.json b/harnesses/aggregator-head-lag/grafana/dashboards/disabled/pulse_vs_codex.json new file mode 100644 index 00000000..8ced6186 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/dashboards/disabled/pulse_vs_codex.json @@ -0,0 +1,661 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Direct comparison of Pulse vs Codex head lag on monitored pools", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Latency (seconds)", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 5 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*pulse.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 1, + "options": { + "legend": { + "calcs": ["lastNotNull", "mean", "max"], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Mean", + "sortDesc": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "head_lag_seconds{aggregator=~\"pulse|codex\",chain=~\"$chain\"}", + "legendFormat": "{{chain}} | {{aggregator}}", + "range": true, + "refId": "A" + } + ], + "title": "Pulse vs Codex - Head Lag Comparison", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Shows which provider is faster (negative = Pulse faster, positive = Codex faster)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": true, + "axisColorMode": "text", + "axisLabel": "Delta (seconds)", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 30, + "gradientMode": "hue", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "area" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": -1 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 10 + }, + "id": 2, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull"], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "(head_lag_seconds{aggregator=\"pulse\",chain=~\"$chain\"} - ignoring(aggregator) head_lag_seconds{aggregator=\"codex\",chain=~\"$chain\"})", + "legendFormat": "{{chain}} - Pulse vs Codex delta", + "range": true, + "refId": "A" + } + ], + "title": "Latency Delta: Pulse - Codex (negative = Pulse faster)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "color-text" + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "orange", + "value": 2 + }, + { + "color": "red", + "value": 5 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Chain" + }, + "properties": [ + { + "id": "custom.width", + "value": 120 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Provider" + }, + "properties": [ + { + "id": "custom.width", + "value": 100 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 19 + }, + "id": 3, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": ["sum"], + "show": false + }, + "showHeader": true, + "sortBy": [ + { + "desc": false, + "displayName": "Mean (5m)" + } + ] + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "exemplar": false, + "expr": "head_lag_seconds{aggregator=~\"pulse|codex\",chain=~\"$chain\"}", + "format": "table", + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "Current" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "exemplar": false, + "expr": "avg_over_time(head_lag_seconds{aggregator=~\"pulse|codex\",chain=~\"$chain\"}[5m])", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "Mean5m" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "exemplar": false, + "expr": "max_over_time(head_lag_seconds{aggregator=~\"pulse|codex\",chain=~\"$chain\"}[5m])", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "__auto", + "range": false, + "refId": "Max5m" + } + ], + "title": "Current Stats - Pulse vs Codex", + "transformations": [ + { + "id": "merge", + "options": {} + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "__name__": true, + "deployment": true, + "instance": true, + "job": true, + "region": true + }, + "includeByName": {}, + "indexByName": {}, + "renameByName": { + "aggregator": "Provider", + "chain": "Chain" + } + } + }, + { + "id": "groupBy", + "options": { + "fields": { + "Chain": { + "aggregations": [], + "operation": "groupby" + }, + "Provider": { + "aggregations": [], + "operation": "groupby" + }, + "Value #Current": { + "aggregations": ["lastNotNull"], + "operation": "aggregate" + }, + "Value #Max5m": { + "aggregations": ["lastNotNull"], + "operation": "aggregate" + }, + "Value #Mean5m": { + "aggregations": ["lastNotNull"], + "operation": "aggregate" + } + } + } + }, + { + "id": "organize", + "options": { + "excludeByName": {}, + "includeByName": {}, + "indexByName": { + "Chain": 0, + "Provider": 1, + "Value #Current (lastNotNull)": 2, + "Value #Max5m (lastNotNull)": 4, + "Value #Mean5m (lastNotNull)": 3 + }, + "renameByName": { + "Value #Current (lastNotNull)": "Current", + "Value #Max5m (lastNotNull)": "Max (5m)", + "Value #Mean5m (lastNotNull)": "Mean (5m)" + } + } + } + ], + "type": "table" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 3 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 27 + }, + "id": 4, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "head_lag_seconds{aggregator=\"pulse\",chain=~\"$chain\"}", + "instant": true, + "legendFormat": "{{chain}}", + "refId": "A" + } + ], + "title": "Pulse - Current Head Lag", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + }, + { + "color": "red", + "value": 3 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 27 + }, + "id": 5, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "head_lag_seconds{aggregator=\"codex\",chain=~\"$chain\"}", + "instant": true, + "legendFormat": "{{chain}}", + "refId": "A" + } + ], + "title": "Codex - Current Head Lag", + "type": "gauge" + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": ["pulse", "codex", "comparison", "head-lag"], + "templating": { + "list": [ + { + "current": { + "selected": true, + "text": ["All"], + "value": ["$__all"] + }, + "hide": 0, + "includeAll": true, + "label": "Chain", + "multi": true, + "name": "chain", + "options": [ + { + "selected": true, + "text": "All", + "value": "$__all" + }, + { + "selected": false, + "text": "solana", + "value": "solana" + }, + { + "selected": false, + "text": "base", + "value": "base" + }, + { + "selected": false, + "text": "bnb", + "value": "bnb" + } + ], + "query": "solana,base,bnb", + "queryValue": "", + "skipUrlSync": false, + "type": "custom" + } + ] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Pulse vs Codex - Head Lag Comparison", + "uid": "pulse_vs_codex", + "version": 1, + "weekStart": "" +} diff --git a/harnesses/aggregator-head-lag/grafana/dashboards/disabled/quote_api_latency.json b/harnesses/aggregator-head-lag/grafana/dashboards/disabled/quote_api_latency.json new file mode 100644 index 00000000..d0e12537 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/dashboards/disabled/quote_api_latency.json @@ -0,0 +1,916 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "opacity", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 3, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 500 + }, + { + "color": "red", + "value": 2000 + } + ] + }, + "unit": "ms" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*jupiter.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*kyberswap.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "purple", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*paraswap.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*openocean.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 1, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max", "min"], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Mean", + "sortDesc": false + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{chain!=\"arbitrum\"}[1m])) by (le, provider, chain))", + "legendFormat": "{{provider}} - {{chain}} (P50)", + "range": true, + "refId": "A" + } + ], + "title": "Quote API Latency Comparison - All Providers (P50)", + "description": "Median Quote API response time - Mobula vs competitors (30s polling interval)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 300 + }, + { + "color": "orange", + "value": 700 + }, + { + "color": "red", + "value": 1500 + } + ] + }, + "unit": "ms" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 24, + "x": 0, + "y": 10 + }, + "id": 2, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["lastNotNull"], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket[5m])) by (le, provider))", + "legendFormat": "{{provider}}", + "range": true, + "refId": "A" + } + ], + "title": "Latest Quote API Latency (P50) by Provider", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "opacity", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 3, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1000 + } + ] + }, + "unit": "ms" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*jupiter.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 3, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max"], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{chain=\"solana\"}[1m])) by (le, provider))", + "legendFormat": "{{provider}} (P50)", + "range": true, + "refId": "A" + } + ], + "title": "🌞 Solana Quote APIs - Mobula vs Jupiter", + "description": "Solana swap quote latency comparison", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "opacity", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 3, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1000 + } + ] + }, + "unit": "ms" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 4, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max"], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{chain=\"base\"}[1m])) by (le, provider))", + "legendFormat": "{{provider}} (P50)", + "range": true, + "refId": "A" + } + ], + "title": "🔵 Base Quote APIs - Mobula vs Competitors", + "description": "Base chain swap quote latency comparison", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "opacity", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 3, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1000 + } + ] + }, + "unit": "ms" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 24 + }, + "id": 6, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max"], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{chain=\"ethereum\"}[1m])) by (le, provider))", + "legendFormat": "{{provider}} (P50)", + "range": true, + "refId": "A" + } + ], + "title": "⟠ Ethereum Quote APIs - Competitors Only", + "description": "Ethereum chain swap quote latency (Mobula not deployed)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "bars", + "fillOpacity": 80, + "gradientMode": "hue", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 0, + "y": 32 + }, + "id": 7, + "options": { + "legend": { + "calcs": ["sum"], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "sum(increase(quote_api_errors_total{chain!=\"arbitrum\"}[5m])) by (provider, chain)", + "legendFormat": "{{provider}} - {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Quote API Errors (Last 5min)", + "description": "Number of errors per provider and chain", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 95 + }, + { + "color": "red", + "value": 99 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 12, + "y": 32 + }, + "id": 8, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["lastNotNull"], + "fields": "" + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "100 * sum(rate(quote_api_status_codes_total{status_code=\"200\"}[5m])) by (provider) / sum(rate(quote_api_status_codes_total[5m])) by (provider)", + "legendFormat": "{{provider}}", + "range": true, + "refId": "A" + } + ], + "title": "Quote API Success Rate by Provider", + "description": "Percentage of successful (200) responses", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ms" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 38 + }, + "id": 9, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max"], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Mean", + "sortDesc": false + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{provider=\"mobula\",chain!=\"arbitrum\"}[1m])) by (le, chain))", + "legendFormat": "Mobula - {{chain}} (P50)", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum(rate(quote_api_latency_milliseconds_bucket{provider=\"mobula\",chain!=\"arbitrum\"}[1m])) by (le, chain))", + "legendFormat": "Mobula - {{chain}} (P95)", + "range": true, + "refId": "B" + } + ], + "title": "Mobula Quote API Latency by Chain (P50 & P95)", + "description": "Mobula swap quote latency across supported chains", + "type": "timeseries" + } + ], + "refresh": "5s", + "schemaVersion": 39, + "tags": ["quote-api", "swap", "latency", "mobula", "jupiter", "benchmark"], + "templating": { + "list": [] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Quote API Latency Benchmark", + "uid": "quote_api_latency", + "version": 0, + "weekStart": "" +} diff --git a/harnesses/aggregator-head-lag/grafana/dashboards/head_lag.json b/harnesses/aggregator-head-lag/grafana/dashboards/head_lag.json new file mode 100644 index 00000000..c325ba7c --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/dashboards/head_lag.json @@ -0,0 +1,945 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds Behind", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 30 + }, + { + "color": "orange", + "value": 60 + }, + { + "color": "red", + "value": 300 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#1F78B4", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#1F78B4", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D03F", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#F4D03F", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#27AE60", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#27AE60", + "value": null + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "legend": { + "calcs": [ + "mean", + "last", + "max", + "min" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Mean", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "avg_over_time(head_lag_seconds[1m])", + "legendFormat": "[{{region}}] {{aggregator}} - {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Head Lag (Estimated Seconds Behind)", + "description": "Estimated time in seconds the aggregator is behind the blockchain head. Calculated using average block time per chain.", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "#1F78B4" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "orange", + "value": 10 + }, + { + "color": "red", + "value": 30 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 18 + }, + "id": 3, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "avg_over_time(head_lag_seconds{aggregator=\"mobula\"}[1m])", + "legendFormat": "[{{region}}] {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Mobula - Current Head Lag (Seconds)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "#F4D03F" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "orange", + "value": 10 + }, + { + "color": "red", + "value": 30 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 18 + }, + "id": 4, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "avg_over_time(head_lag_seconds{aggregator=\"codex\"}[1m])", + "legendFormat": "[{{region}}] {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Codex - Current Head Lag (Seconds)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "#27AE60" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "orange", + "value": 10 + }, + { + "color": "red", + "value": 30 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 18 + }, + "id": 5, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "avg_over_time(head_lag_seconds{aggregator=\"geckoterminal\"}[1m])", + "legendFormat": "[{{region}}] {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "GeckoTerminal - Current Head Lag (Seconds)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 0, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto", + "spanNulls": true + }, + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "red", + "value": 20 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#1F78B4", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#1F78B4", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D03F", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#F4D03F", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#27AE60", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#27AE60", + "value": null + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 10 + }, + "id": 6, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "avg_over_time(head_lag_seconds{region=\"eu-west\"}[1m])", + "legendFormat": "[{{region}}] {{aggregator}} - {{chain}}", + "refId": "A" + } + ], + "title": "EU West - Head Lag", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 0, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto", + "spanNulls": true + }, + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "red", + "value": 20 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#1F78B4", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#1F78B4", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D03F", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#F4D03F", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#27AE60", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#27AE60", + "value": null + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 10 + }, + "id": 7, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "avg_over_time(head_lag_seconds{region=\"us-east\"}[1m])", + "legendFormat": "[{{region}}] {{aggregator}} - {{chain}}", + "refId": "A" + } + ], + "title": "US East - Head Lag", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 0, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto", + "spanNulls": true + }, + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "red", + "value": 20 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#1F78B4", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#1F78B4", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#F4D03F", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#F4D03F", + "value": null + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "#27AE60", + "mode": "fixed" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "#27AE60", + "value": null + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 10 + }, + "id": 8, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "avg_over_time(head_lag_seconds{region=\"sgp\"}[1m])", + "legendFormat": "[{{region}}] {{aggregator}} - {{chain}}", + "refId": "A" + } + ], + "title": "Singapore - Head Lag", + "type": "timeseries" + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": [ + "head-lag", + "indexation", + "blockchain", + "sync" + ], + "templating": { + "list": [] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Head Lag Monitor - Blockchain vs Aggregator Sync", + "uid": "head_lag_monitor", + "version": 0, + "weekStart": "" +} \ No newline at end of file diff --git a/harnesses/aggregator-head-lag/grafana/grafana-entrypoint.sh b/harnesses/aggregator-head-lag/grafana/grafana-entrypoint.sh new file mode 100755 index 00000000..ccf63ca7 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/grafana-entrypoint.sh @@ -0,0 +1,29 @@ +#!/bin/sh + +echo "=== GRAFANA ENTRYPOINT SCRIPT STARTING ===" +echo "Working directory: $(pwd)" +echo "User: $(whoami)" + +# Create dashboards directory +mkdir -p /var/lib/grafana/dashboards + +# Debug: print environment variables +echo "Checking environment..." +env | grep -E '(RAILWAY|HIDE_QUOTE)' || echo "No RAILWAY/HIDE_QUOTE variables found" + +# Copy dashboards from source +if [ "$HIDE_QUOTE_DASHBOARD" = "true" ]; then + echo "HIDE_QUOTE_DASHBOARD=true - hiding Quote API Latency Benchmark dashboard" + cp /dashboards-source/head_lag.json /var/lib/grafana/dashboards/ +else + echo "Copying all dashboards" + cp /dashboards-source/*.json /var/lib/grafana/dashboards/ +fi + +echo "Dashboards copied:" +ls -la /var/lib/grafana/dashboards/ + +echo "=== GRAFANA ENTRYPOINT SCRIPT COMPLETE ===" + +# Start Grafana with default entrypoint +exec /run.sh diff --git a/harnesses/aggregator-head-lag/grafana/provisioning/dashboards/dashboard.yml b/harnesses/aggregator-head-lag/grafana/provisioning/dashboards/dashboard.yml new file mode 100644 index 00000000..332c8f34 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/provisioning/dashboards/dashboard.yml @@ -0,0 +1,13 @@ +apiVersion: 1 + +providers: + - name: 'Aggregator Latency Dashboards' + orgId: 1 + folder: '' + type: file + disableDeletion: false + updateIntervalSeconds: 10 + allowUiUpdates: false + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: true diff --git a/harnesses/aggregator-head-lag/grafana/provisioning/datasources/prometheus.yml b/harnesses/aggregator-head-lag/grafana/provisioning/datasources/prometheus.yml new file mode 100644 index 00000000..30d5cc21 --- /dev/null +++ b/harnesses/aggregator-head-lag/grafana/provisioning/datasources/prometheus.yml @@ -0,0 +1,10 @@ +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: ${PROMETHEUS_URL} + uid: prometheus + isDefault: true + editable: false diff --git a/harnesses/aggregator-head-lag/monitoring/alert_rules.yml b/harnesses/aggregator-head-lag/monitoring/alert_rules.yml new file mode 100644 index 00000000..e08b8a73 --- /dev/null +++ b/harnesses/aggregator-head-lag/monitoring/alert_rules.yml @@ -0,0 +1,127 @@ +groups: + - name: aggregator_latency_alerts + interval: 30s + rules: + # Missing metrics alerts + - alert: MissingMobulaMetrics + expr: absent(rest_api_latency_milliseconds_count{aggregator="mobula"}) + for: 5m + labels: + severity: critical + aggregator: mobula + alert_type: missing_metrics + annotations: + summary: "Mobula metrics are missing" + description: "No REST API metrics received from Mobula for 5 minutes. Check API key and monitor status." + + # Stale metrics alerts (>20 minutes without update) + - alert: MobulaEthereumStaleMetrics + expr: (time() - timestamp(rest_api_latency_milliseconds_count{aggregator="mobula",chain="ethereum"})) > 1200 + for: 2m + labels: + severity: warning + aggregator: mobula + chain: ethereum + alert_type: stale_metrics + annotations: + summary: "Mobula Ethereum metrics are stale" + description: "Mobula Ethereum REST API hasn't responded in over 20 minutes. Check API connectivity." + + - alert: MobulaBaseStaleMetrics + expr: (time() - timestamp(rest_api_latency_milliseconds_count{aggregator="mobula",chain="base"})) > 1200 + for: 2m + labels: + severity: warning + aggregator: mobula + chain: base + alert_type: stale_metrics + annotations: + summary: "Mobula Base metrics are stale" + description: "Mobula Base REST API hasn't responded in over 20 minutes. Check API connectivity." + + # Latency spike alerts + - alert: MobulaEthereumLatencySpike + expr: | + ( + rate(rest_api_latency_milliseconds_sum{aggregator="mobula",chain="ethereum"}[5m]) / + rate(rest_api_latency_milliseconds_count{aggregator="mobula",chain="ethereum"}[5m]) + ) > 500 + for: 3m + labels: + severity: warning + aggregator: mobula + chain: ethereum + alert_type: latency_spike + annotations: + summary: "Mobula Ethereum latency spike" + description: "Mobula Ethereum average latency is {{ $value | humanize }}ms over the last 5 minutes (threshold: 500ms)" + + - alert: MobulaBaseLatencySpike + expr: | + ( + rate(rest_api_latency_milliseconds_sum{aggregator="mobula",chain="base"}[5m]) / + rate(rest_api_latency_milliseconds_count{aggregator="mobula",chain="base"}[5m]) + ) > 500 + for: 3m + labels: + severity: warning + aggregator: mobula + chain: base + alert_type: latency_spike + annotations: + summary: "Mobula Base latency spike" + description: "Mobula Base average latency is {{ $value | humanize }}ms over the last 5 minutes (threshold: 500ms)" + + # Instant spike detection for Mobula (any single request >10s) + - alert: MobulaInstantLatencySpike + expr: | + rest_api_latency_milliseconds{aggregator="mobula"} > 10000 + labels: + severity: warning + aggregator: mobula + alert_type: instant_spike + annotations: + summary: "Mobula instant latency spike on {{ $labels.chain }}" + description: "Mobula {{ $labels.chain }} single request took {{ $value | humanize }}ms (>10s). Brief spike detected." + + # Extreme latency spikes (>5x normal) + - alert: MobulaExtremeLatencySpike + expr: | + ( + rate(rest_api_latency_milliseconds_sum{aggregator="mobula"}[5m]) / + rate(rest_api_latency_milliseconds_count{aggregator="mobula"}[5m]) + ) > 1000 + for: 1m + labels: + severity: critical + aggregator: mobula + alert_type: extreme_latency + annotations: + summary: "Mobula EXTREME latency spike on {{ $labels.chain }}" + description: "Mobula {{ $labels.chain }} latency is {{ $value | humanize }}ms (>1000ms). Possible service degradation." + + # Error rate alerts + - alert: HighRESTErrorRate + expr: | + ( + rate(rest_api_errors_total[5m]) / + (rate(rest_api_errors_total[5m]) + rate(rest_api_latency_milliseconds_count[5m])) + ) > 0.1 + for: 5m + labels: + severity: warning + alert_type: high_error_rate + annotations: + summary: "High REST API error rate for {{ $labels.aggregator }} {{ $labels.chain }}" + description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes (threshold: 10%)" + + # Service availability + - alert: CodexServiceDown + expr: up{job="latency_monitor"} == 0 + for: 2m + labels: + severity: critical + alert_type: service_down + annotations: + summary: "Latency monitor service is down" + description: "The aggregator latency monitor has been down for 2 minutes. No metrics are being collected." diff --git a/harnesses/aggregator-head-lag/monitoring/alertmanager.yml b/harnesses/aggregator-head-lag/monitoring/alertmanager.yml new file mode 100644 index 00000000..77b9e263 --- /dev/null +++ b/harnesses/aggregator-head-lag/monitoring/alertmanager.yml @@ -0,0 +1,36 @@ +global: + resolve_timeout: 5m + +route: + group_by: ['alertname', 'chain', 'aggregator'] + group_wait: 10s + group_interval: 30s + repeat_interval: 4h + receiver: 'slack-webhook' + +receivers: + - name: 'slack-webhook' + webhook_configs: + - url: 'https://agent-slack-production.up.railway.app/webhook/grafana' + send_resolved: true + +inhibit_rules: + # Inhibit warning alerts if critical alert is firing + - source_match: + severity: 'critical' + target_match: + severity: 'warning' + equal: ['alertname', 'chain', 'aggregator'] + + # Inhibit stale metrics alerts if service is down + - source_match: + alert_type: 'service_down' + target_match: + alert_type: 'stale_metrics' + + # Inhibit stale metrics if missing metrics alert is firing + - source_match: + alert_type: 'missing_metrics' + target_match: + alert_type: 'stale_metrics' + equal: ['aggregator'] diff --git a/harnesses/aggregator-head-lag/monitoring/grafana/dashboards/disabled/quote_api_latency.json b/harnesses/aggregator-head-lag/monitoring/grafana/dashboards/disabled/quote_api_latency.json new file mode 100644 index 00000000..16392e30 --- /dev/null +++ b/harnesses/aggregator-head-lag/monitoring/grafana/dashboards/disabled/quote_api_latency.json @@ -0,0 +1,819 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "opacity", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 3, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 500 + }, + { + "color": "red", + "value": 2000 + } + ] + }, + "unit": "ms" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*jupiter.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*kyberswap.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "blue", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "purple", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*paraswap.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "yellow", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*openocean.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "red", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 1, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max", "min"], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Mean", + "sortDesc": false + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{chain!=\"arbitrum\"}[1m])) by (le, provider, chain))", + "legendFormat": "{{provider}} - {{chain}} (P50)", + "range": true, + "refId": "A" + } + ], + "title": "Quote API Latency Comparison - All Providers (P50)", + "description": "Median Quote API response time - Mobula vs competitors (30s polling interval)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 300 + }, + { + "color": "orange", + "value": 700 + }, + { + "color": "red", + "value": 1500 + } + ] + }, + "unit": "ms" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 24, + "x": 0, + "y": 10 + }, + "id": 2, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["lastNotNull"], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket[5m])) by (le, provider))", + "legendFormat": "{{provider}}", + "range": true, + "refId": "A" + } + ], + "title": "Latest Quote API Latency (P50) by Provider", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "opacity", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 3, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1000 + } + ] + }, + "unit": "ms" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*jupiter.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 3, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max"], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{chain=\"solana\"}[1m])) by (le, provider))", + "legendFormat": "{{provider}} (P50)", + "range": true, + "refId": "A" + } + ], + "title": "\ud83c\udf1e Solana Quote APIs - Mobula vs Jupiter", + "description": "Solana swap quote latency comparison", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 20, + "gradientMode": "opacity", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 3, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1000 + } + ] + }, + "unit": "ms" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 4, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max"], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{chain=\"base\"}[1m])) by (le, provider))", + "legendFormat": "{{provider}} (P50)", + "range": true, + "refId": "A" + } + ], + "title": "\ud83d\udd35 Base Quote APIs - Mobula vs Competitors", + "description": "Base chain swap quote latency comparison", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "bars", + "fillOpacity": 80, + "gradientMode": "hue", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "normal" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 0, + "y": 32 + }, + "id": 7, + "options": { + "legend": { + "calcs": ["sum"], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "sum(increase(quote_api_errors_total{chain!=\"arbitrum\"}[5m])) by (provider, chain)", + "legendFormat": "{{provider}} - {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Quote API Errors (Last 5min)", + "description": "Number of errors per provider and chain", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 95 + }, + { + "color": "red", + "value": 99 + } + ] + }, + "unit": "percent" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 12, + "y": 32 + }, + "id": 8, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["lastNotNull"], + "fields": "" + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "100 * sum(rate(quote_api_status_codes_total{status_code=\"200\"}[5m])) by (provider) / sum(rate(quote_api_status_codes_total[5m])) by (provider)", + "legendFormat": "{{provider}}", + "range": true, + "refId": "A" + } + ], + "title": "Quote API Success Rate by Provider", + "description": "Percentage of successful (200) responses", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ms" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 38 + }, + "id": 9, + "options": { + "legend": { + "calcs": ["mean", "lastNotNull", "max"], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Mean", + "sortDesc": false + }, + "tooltip": { + "mode": "multi", + "sort": "asc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum(rate(quote_api_latency_milliseconds_bucket{provider=\"mobula\",chain!=\"arbitrum\"}[1m])) by (le, chain))", + "legendFormat": "Mobula - {{chain}} (P50)", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum(rate(quote_api_latency_milliseconds_bucket{provider=\"mobula\",chain!=\"arbitrum\"}[1m])) by (le, chain))", + "legendFormat": "Mobula - {{chain}} (P95)", + "range": true, + "refId": "B" + } + ], + "title": "Mobula Quote API Latency by Chain (P50 & P95)", + "description": "Mobula swap quote latency across supported chains", + "type": "timeseries" + } + ], + "refresh": "5s", + "schemaVersion": 39, + "tags": ["quote-api", "swap", "latency", "mobula", "jupiter", "benchmark"], + "templating": { + "list": [] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Quote API Latency Benchmark", + "uid": "quote_api_latency", + "version": 0, + "weekStart": "" +} diff --git a/harnesses/aggregator-head-lag/monitoring/grafana/dashboards/head_lag.json b/harnesses/aggregator-head-lag/monitoring/grafana/dashboards/head_lag.json new file mode 100644 index 00000000..02606a47 --- /dev/null +++ b/harnesses/aggregator-head-lag/monitoring/grafana/dashboards/head_lag.json @@ -0,0 +1,773 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds Behind", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "tooltip": false, + "viz": false, + "legend": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 30 + }, + { + "color": "orange", + "value": 60 + }, + { + "color": "red", + "value": 300 + } + ] + }, + "unit": "s" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "purple", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "legend": { + "calcs": ["median", "lastNotNull", "max", "min"], + "displayMode": "table", + "placement": "right", + "showLegend": true, + "sortBy": "Median", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "head_lag_seconds{chain!=\"ethereum\"}", + "legendFormat": "{{aggregator}} - {{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Head Lag (Estimated Seconds Behind)", + "description": "Estimated time in seconds the aggregator is behind the blockchain head. Calculated using average block time per chain.", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "orange", + "value": 10 + }, + { + "color": "red", + "value": 30 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 18 + }, + "id": 3, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["lastNotNull"], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "head_lag_seconds{aggregator=\"mobula\",chain!=\"ethereum\"}", + "legendFormat": "{{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Mobula - Current Head Lag (Seconds)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "orange", + "value": 10 + }, + { + "color": "red", + "value": 30 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 18 + }, + "id": 4, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["lastNotNull"], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "head_lag_seconds{aggregator=\"codex\",chain!=\"ethereum\"}", + "legendFormat": "{{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "Codex - Current Head Lag (Seconds)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "orange", + "value": 10 + }, + { + "color": "red", + "value": 30 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 18 + }, + "id": 5, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": ["lastNotNull"], + "fields": "" + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto" + }, + "pluginVersion": "11.0.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "head_lag_seconds{aggregator=\"geckoterminal\",chain!=\"ethereum\"}", + "legendFormat": "{{chain}}", + "range": true, + "refId": "A" + } + ], + "title": "GeckoTerminal - Current Head Lag (Seconds)", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto" + }, + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "red", + "value": 20 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "purple", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 10 + }, + "id": 6, + "options": { + "legend": { + "calcs": ["lastNotNull"], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "head_lag_seconds{region=\"eu-west\",chain!=\"ethereum\"}", + "legendFormat": "{{aggregator}} - {{chain}}", + "refId": "A" + } + ], + "title": "EU West - Head Lag", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto" + }, + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "red", + "value": 20 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "purple", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 10 + }, + "id": 7, + "options": { + "legend": { + "calcs": ["lastNotNull"], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "head_lag_seconds{region=\"us-west\",chain!=\"ethereum\"}", + "legendFormat": "{{aggregator}} - {{chain}}", + "refId": "A" + } + ], + "title": "US West - Head Lag", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Seconds", + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineInterpolation": "linear", + "lineWidth": 2, + "showPoints": "auto" + }, + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 5 + }, + { + "color": "red", + "value": 20 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": ".*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "orange", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*codex.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "green", + "mode": "fixed" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": ".*geckoterminal.*" + }, + "properties": [ + { + "id": "color", + "value": { + "fixedColor": "purple", + "mode": "fixed" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 10 + }, + "id": 8, + "options": { + "legend": { + "calcs": ["lastNotNull"], + "displayMode": "list", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "head_lag_seconds{region=\"singapore\",chain!=\"ethereum\"}", + "legendFormat": "{{aggregator}} - {{chain}}", + "refId": "A" + } + ], + "title": "Singapore - Head Lag", + "type": "timeseries" + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": ["head-lag", "indexation", "blockchain", "sync"], + "templating": { + "list": [] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Head Lag Monitor - Blockchain vs Aggregator Sync", + "uid": "head_lag_monitor", + "version": 0, + "weekStart": "" +} diff --git a/harnesses/aggregator-head-lag/monitoring/grafana/provisioning/dashboards/dashboard.yml b/harnesses/aggregator-head-lag/monitoring/grafana/provisioning/dashboards/dashboard.yml new file mode 100644 index 00000000..49b448da --- /dev/null +++ b/harnesses/aggregator-head-lag/monitoring/grafana/provisioning/dashboards/dashboard.yml @@ -0,0 +1,13 @@ +apiVersion: 1 + +providers: + - name: 'Aggregator Latency Dashboards' + orgId: 1 + folder: '' + type: file + disableDeletion: false + updateIntervalSeconds: 10 + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: true diff --git a/harnesses/aggregator-head-lag/monitoring/grafana/provisioning/datasources/prometheus.yml b/harnesses/aggregator-head-lag/monitoring/grafana/provisioning/datasources/prometheus.yml new file mode 100644 index 00000000..40a18e69 --- /dev/null +++ b/harnesses/aggregator-head-lag/monitoring/grafana/provisioning/datasources/prometheus.yml @@ -0,0 +1,10 @@ +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: ${PROMETHEUS_URL:-http://prometheus:9090} + uid: prometheus + isDefault: true + editable: false diff --git a/harnesses/aggregator-head-lag/monitoring/prometheus.yml b/harnesses/aggregator-head-lag/monitoring/prometheus.yml new file mode 100644 index 00000000..bd7cf041 --- /dev/null +++ b/harnesses/aggregator-head-lag/monitoring/prometheus.yml @@ -0,0 +1,21 @@ +global: + scrape_interval: 5s + evaluation_interval: 5s + +# Load alert rules +rule_files: + - '/etc/prometheus/alert_rules.yml' + +# Alertmanager configuration +alerting: + alertmanagers: + - static_configs: + - targets: ['alertmanager:9093'] + +scrape_configs: + - job_name: 'latency_monitor' + static_configs: + - targets: ['monitor:2112'] + labels: + app: 'aggregator_latency_monitor' + environment: 'production' diff --git a/harnesses/aggregator-head-lag/prometheus/Dockerfile b/harnesses/aggregator-head-lag/prometheus/Dockerfile new file mode 100644 index 00000000..a422384f --- /dev/null +++ b/harnesses/aggregator-head-lag/prometheus/Dockerfile @@ -0,0 +1,14 @@ +FROM prom/prometheus:v2.49.1 + +USER root + +# Pick which prometheus.yml to bake in (prometheus.yml for prod, prometheus.staging.yml for staging). +ARG PROMETHEUS_CONFIG=prometheus.yml + +COPY ${PROMETHEUS_CONFIG} /etc/prometheus/prometheus.yml +COPY alert_rules.yml /etc/prometheus/alert_rules.yml + +EXPOSE 9090 + +# --enable-feature=expand-external-labels: lets global.external_labels read ${ENVIRONMENT} from env +CMD ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.path=/prometheus", "--web.enable-lifecycle", "--web.enable-admin-api", "--enable-feature=expand-external-labels"] diff --git a/harnesses/aggregator-head-lag/prometheus/alert_rules.yml b/harnesses/aggregator-head-lag/prometheus/alert_rules.yml new file mode 100644 index 00000000..dd89ab5e --- /dev/null +++ b/harnesses/aggregator-head-lag/prometheus/alert_rules.yml @@ -0,0 +1,223 @@ +groups: + - name: aggregator_latency_alerts + interval: 30s + rules: + # Missing metrics alerts + - alert: MissingMobulaMetrics + expr: absent(rest_api_latency_milliseconds_count{aggregator="mobula"}) + for: 5m + labels: + severity: critical + aggregator: mobula + alert_type: missing_metrics + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula metrics are missing' + description: "No REST API metrics received from Mobula for 5 minutes. Check API key and monitor status." + + # Stale metrics alerts (>20 minutes without update) + - alert: MobulaBaseStaleMetrics + expr: (time() - timestamp(rest_api_latency_milliseconds_count{aggregator="mobula",chain="base"})) > 1200 + for: 2m + labels: + severity: warning + aggregator: mobula + chain: base + alert_type: stale_metrics + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula Base metrics are stale' + description: "Mobula Base REST API hasn't responded in over 20 minutes. Check API connectivity." + + # Latency spike alerts + - alert: MobulaSolanaLatencySpike + expr: | + ( + rate(rest_api_latency_milliseconds_sum{aggregator="mobula",chain="solana"}[5m]) / + rate(rest_api_latency_milliseconds_count{aggregator="mobula",chain="solana"}[5m]) + ) > 2000 + for: 3m + labels: + severity: warning + aggregator: mobula + chain: solana + alert_type: latency_spike + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula Solana latency spike' + description: "Mobula Solana average latency is {{ $value | humanize }}ms over the last 5 minutes (threshold: 2000ms)" + + - alert: MobulaBaseLatencySpike + expr: | + ( + rate(rest_api_latency_milliseconds_sum{aggregator="mobula",chain="base"}[5m]) / + rate(rest_api_latency_milliseconds_count{aggregator="mobula",chain="base"}[5m]) + ) > 2500 + for: 3m + labels: + severity: warning + aggregator: mobula + chain: base + alert_type: latency_spike + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula Base latency spike' + description: "Mobula Base average latency is {{ $value | humanize }}ms over the last 5 minutes (threshold: 2500ms)" + + - alert: MobulaBNBLatencySpike + expr: | + ( + rate(rest_api_latency_milliseconds_sum{aggregator="mobula",chain="bnb"}[5m]) / + rate(rest_api_latency_milliseconds_count{aggregator="mobula",chain="bnb"}[5m]) + ) > 2500 + for: 3m + labels: + severity: warning + aggregator: mobula + chain: bnb + alert_type: latency_spike + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula BNB latency spike' + description: "Mobula BNB average latency is {{ $value | humanize }}ms over the last 5 minutes (threshold: 2500ms)" + + # Instant spike detection for Mobula (any single request >10s) + - alert: MobulaInstantLatencySpike + expr: | + rest_api_latency_milliseconds{aggregator="mobula"} > 10000 + labels: + severity: warning + aggregator: mobula + alert_type: instant_spike + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula instant latency spike on {{ $labels.chain }}' + description: "Mobula {{ $labels.chain }} single request took {{ $value | humanize }}ms (>10s). Brief spike detected." + + # Error rate alerts + - alert: HighRESTErrorRate + expr: | + ( + rate(rest_api_errors_total[5m]) / + (rate(rest_api_errors_total[5m]) + rate(rest_api_latency_milliseconds_count[5m])) + ) > 0.1 + for: 5m + labels: + severity: warning + alert_type: high_error_rate + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}High REST API error rate for {{ $labels.aggregator }} {{ $labels.chain }}' + description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes (threshold: 10%)" + + # Head lag (WebSocket latency) alerts - per-chain thresholds + # NOTE: alerts use fixed-cardinality gauges now (no tx_hash label) to avoid stale-series spam. + # Sustained 30s window avoids single-spike firings (reconnect bursts, transient network jitter). + - alert: MobulaHeadLagSpikeBase + expr: mobula_head_lag_detailed_seconds{chain="base"} > 2.5 + for: 30s + labels: + severity: warning + aggregator: mobula + alert_type: head_lag_spike + app: aggregator_latency_monitor + chain: base + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula head lag spike on Base' + description: | + **Mobula WebSocket Latency Spike - Base** + + **Latency:** {{ $value }}s (threshold: 2.5s, sustained ≥30s) + • Mobula Processing: {{ with query (printf "mobula_processing_lag_seconds{chain=\"base\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}{{ . | first | value }}s{{ end }} + • Network: {{ with query (printf "mobula_network_lag_seconds{chain=\"base\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}{{ . | first | value }}s{{ end }} + + • Pool: {{ $labels.pool_address }} + • Region: {{ $labels.region }} + • Latest tx: {{ with query (printf "mobula_last_tx_hash{chain=\"base\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}https://basescan.org/tx/{{ . | first | label "tx_hash" }}{{ end }} + + - alert: MobulaHeadLagSpikeSolana + expr: mobula_head_lag_detailed_seconds{chain="solana"} > 2 + for: 30s + labels: + severity: warning + aggregator: mobula + alert_type: head_lag_spike + app: aggregator_latency_monitor + chain: solana + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula head lag spike on Solana' + description: | + **Mobula WebSocket Latency Spike - Solana** + + **Latency:** {{ $value }}s (threshold: 2s, sustained ≥30s) + • Mobula Processing: {{ with query (printf "mobula_processing_lag_seconds{chain=\"solana\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}{{ . | first | value }}s{{ end }} + • Network: {{ with query (printf "mobula_network_lag_seconds{chain=\"solana\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}{{ . | first | value }}s{{ end }} + + • Pool: {{ $labels.pool_address }} + • Region: {{ $labels.region }} + • Latest tx: {{ with query (printf "mobula_last_tx_hash{chain=\"solana\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}https://solscan.io/tx/{{ . | first | label "tx_hash" }}{{ end }} + + - alert: MobulaHeadLagSpikeBNB + expr: mobula_head_lag_detailed_seconds{chain="bnb"} > 2.5 + for: 30s + labels: + severity: warning + aggregator: mobula + alert_type: head_lag_spike + app: aggregator_latency_monitor + chain: bnb + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Mobula head lag spike on BNB Chain' + description: | + **Mobula WebSocket Latency Spike - BNB Chain** + + **Latency:** {{ $value }}s (threshold: 2.5s, sustained ≥30s) + • Mobula Processing: {{ with query (printf "mobula_processing_lag_seconds{chain=\"bnb\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}{{ . | first | value }}s{{ end }} + • Network: {{ with query (printf "mobula_network_lag_seconds{chain=\"bnb\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}{{ . | first | value }}s{{ end }} + + • Pool: {{ $labels.pool_address }} + • Region: {{ $labels.region }} + • Latest tx: {{ with query (printf "mobula_last_tx_hash{chain=\"bnb\",region=\"%s\",pool_address=\"%s\"}" $labels.region $labels.pool_address) }}https://bscscan.com/tx/{{ . | first | label "tx_hash" }}{{ end }} + + # Missing head lag metrics (no data for 5 minutes = monitor down) + - alert: MissingHeadLagMetrics + expr: absent(head_lag_seconds) + for: 5m + labels: + severity: critical + alert_type: missing_head_lag + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Head lag metrics missing' + description: "No head_lag_seconds metrics received for 5 minutes. Check if monitors are running." + + # Per-aggregator staleness: fires when one provider stops pushing data + # (e.g. Codex WS disconnect) while others keep running — global absent() + # would NOT catch this. + - alert: AggregatorHeadLagStale + expr: (time() - timestamp(head_lag_seconds)) > 300 + for: 1m + labels: + severity: warning + alert_type: head_lag_stale + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}{{ $labels.aggregator }} head_lag stale on {{ $labels.chain }} ({{ $labels.region }})' + description: | + **{{ $labels.aggregator }}** hasn't pushed a head_lag sample for **{{ $labels.chain }} / {{ $labels.region }}** in over 5 minutes. + + Likely cause: WebSocket disconnected, JWT expired, proxy IP banned, or auth cookie rotated. + + Check the monitor logs for `[HEAD-LAG][{{ $labels.aggregator | toUpper }}]` errors. + + # Service availability + - alert: CodexServiceDown + expr: up{job="latency_monitor"} == 0 + for: 2m + labels: + severity: critical + alert_type: service_down + app: aggregator_latency_monitor + annotations: + summary: '{{ if eq $externalLabels.environment "staging" }}[STAGING] {{ end }}Latency monitor service is down' + description: "The aggregator latency monitor has been down for 2 minutes. No metrics are being collected." diff --git a/harnesses/aggregator-head-lag/prometheus/alertmanager.yml b/harnesses/aggregator-head-lag/prometheus/alertmanager.yml new file mode 100644 index 00000000..77b9e263 --- /dev/null +++ b/harnesses/aggregator-head-lag/prometheus/alertmanager.yml @@ -0,0 +1,36 @@ +global: + resolve_timeout: 5m + +route: + group_by: ['alertname', 'chain', 'aggregator'] + group_wait: 10s + group_interval: 30s + repeat_interval: 4h + receiver: 'slack-webhook' + +receivers: + - name: 'slack-webhook' + webhook_configs: + - url: 'https://agent-slack-production.up.railway.app/webhook/grafana' + send_resolved: true + +inhibit_rules: + # Inhibit warning alerts if critical alert is firing + - source_match: + severity: 'critical' + target_match: + severity: 'warning' + equal: ['alertname', 'chain', 'aggregator'] + + # Inhibit stale metrics alerts if service is down + - source_match: + alert_type: 'service_down' + target_match: + alert_type: 'stale_metrics' + + # Inhibit stale metrics if missing metrics alert is firing + - source_match: + alert_type: 'missing_metrics' + target_match: + alert_type: 'stale_metrics' + equal: ['aggregator'] diff --git a/harnesses/aggregator-head-lag/prometheus/prometheus.staging.yml b/harnesses/aggregator-head-lag/prometheus/prometheus.staging.yml new file mode 100644 index 00000000..a5c329de --- /dev/null +++ b/harnesses/aggregator-head-lag/prometheus/prometheus.staging.yml @@ -0,0 +1,26 @@ +global: + scrape_interval: 15s + evaluation_interval: 15s + # Staging external label triggers [STAGING] prefix in alert summaries via templating. + external_labels: + environment: 'staging' + +# Load alert rules (shared with production; [STAGING] prefix is conditional via $externalLabels) +rule_files: + - '/etc/prometheus/alert_rules.yml' + +# Staging Alertmanager (separate from production) +alerting: + alertmanagers: + - static_configs: + - targets: ['alertmanager-staging.railway.internal:9093'] + +# Staging scrape targets — 3 regions, same pattern as production +scrape_configs: + - job_name: 'monitor-staging' + static_configs: + - targets: + - 'agg-staging-eu.railway.internal:2112' + - 'agg-staging-us.railway.internal:2112' + - 'agg-staging-sgp.railway.internal:2112' + metrics_path: /metrics diff --git a/harnesses/aggregator-head-lag/prometheus/prometheus.yml b/harnesses/aggregator-head-lag/prometheus/prometheus.yml new file mode 100644 index 00000000..f9a4dfce --- /dev/null +++ b/harnesses/aggregator-head-lag/prometheus/prometheus.yml @@ -0,0 +1,27 @@ +global: + scrape_interval: 15s + evaluation_interval: 15s + # Expanded at startup via --enable-feature=expand-external-labels. + # Unset => empty value => alert rule templates treat it as "not staging" and don't prefix. + external_labels: + environment: '${ENVIRONMENT}' + +# Load alert rules +rule_files: + - '/etc/prometheus/alert_rules.yml' + +# Alertmanager configuration +alerting: + alertmanagers: + - static_configs: + - targets: ['alertmanager.railway.internal:9093'] + +scrape_configs: + - job_name: 'monitor' + static_configs: + - targets: + - 'agg-eu-west.railway.internal:2112' + - 'alert-reflection.railway.internal:2112' + - 'aggregator-latency-benchmark.railway.internal:2112' + metrics_path: /metrics + diff --git a/harnesses/bridge-fee/README.md b/harnesses/bridge-fee/README.md deleted file mode 100644 index 75253234..00000000 --- a/harnesses/bridge-fee/README.md +++ /dev/null @@ -1,50 +0,0 @@ -# Harness · bridge-fee - -> Extracts the effective cost (fees + slippage + destination gas) from each bridge quote and records it. - -**Bench**: [№ 003 · Bridge Fee](../../benchmarks/bridge-fee.yml) - -## What it measures - -- **Effective fee** — what a user actually pays, as a percentage of trade size. Includes protocol fees, relayer fees, slippage and destination gas. -- **Sample count** — number of quotes captured per bridge per route per amount. - -This harness shares the same runner as `bridge-quote` — it emits cost -metrics from the same quote responses. The split into two benchmarks is -purely editorial: one report on speed, one on cost. - -## Bridges - -`mobula` · `relay` · `lifi` · `debridge` - -## Inputs - -- Routes: USDC pairs spanning Solana, Base and Arbitrum. -- Reference notional: $300 USDC. The harness also captures $5 and $50 but the bench filters to $300 to avoid fixed-fee blowups skewing the percentile aggregates. -- Cadence: full sweep every 5 minutes. -- Region: currently `eu-west` only. - -## Metrics emitted - -``` -bridge_fees_percent{bridge, ...} gauge — explicit fee field, in percent -bridge_cost_percent{bridge, ...} gauge — fees + slippage + gas, in percent (used by bench) -bridge_fees_usd{bridge, ...} gauge -bridge_cost_usd{bridge, ...} gauge -bridge_slippage_usd{bridge, ...} gauge -bridge_gas_usd{bridge, ...} gauge -bridge_fix_fee_usd{bridge, ...} gauge — Debridge native-token fees -bridge_output_usd{bridge, ...} gauge -``` - -## Why `bridge_cost_percent` and not `bridge_fees_percent` - -Some providers (Mobula in particular) bake their take into the spread -rather than reporting an explicit fee field. Comparing only -`bridge_fees_percent` would render those providers at 0% — misleading -readers about what they actually pay. `bridge_cost_percent` measures the -bottom-line cost regardless of how each provider structures pricing. - -## Running locally - -> Implementation TBD — same runner as `bridge-quote/`. diff --git a/harnesses/bridge-monitor/.gitignore b/harnesses/bridge-monitor/.gitignore new file mode 100644 index 00000000..13b984a0 --- /dev/null +++ b/harnesses/bridge-monitor/.gitignore @@ -0,0 +1,7 @@ +.env +bin/ +*.log +prometheus_data/ +grafana_data/ +.DS_Store +monitor diff --git a/harnesses/bridge-monitor/Dockerfile b/harnesses/bridge-monitor/Dockerfile new file mode 100644 index 00000000..75a62cad --- /dev/null +++ b/harnesses/bridge-monitor/Dockerfile @@ -0,0 +1,26 @@ +FROM golang:1.24-alpine AS builder + +WORKDIR /build + +# Copy go mod files +COPY go.mod go.sum ./ +RUN go mod download + +# Copy source code +COPY . . + +# Build the binary +RUN CGO_ENABLED=0 GOOS=linux go build -o /bridge-monitor ./cmd/monitor + +# Final stage +FROM alpine:latest + +RUN apk --no-cache add ca-certificates + +WORKDIR /app + +COPY --from=builder /bridge-monitor . + +EXPOSE 9090 + +CMD ["./bridge-monitor"] diff --git a/harnesses/bridge-monitor/README.md b/harnesses/bridge-monitor/README.md new file mode 100644 index 00000000..14691d38 --- /dev/null +++ b/harnesses/bridge-monitor/README.md @@ -0,0 +1,266 @@ +# Harness · bridge-monitor + +> Real-world benchmark of cross-chain bridges. Produces every metric consumed by both [`benchmarks/bridge-quote-latency.yml`](../../benchmarks/bridge-quote-latency.yml) and [`benchmarks/bridge-fee.yml`](../../benchmarks/bridge-fee.yml) — quote latency, cost percent, fees, success rate, end-to-end execution latency. + +**Benches**: + +- [№ 002 · Bridge — Quote Latency](../../benchmarks/bridge-quote-latency.yml) +- [№ 003 · Bridge — Cost Percent](../../benchmarks/bridge-fee.yml) + +**Tracked bridges**: Mobula · Relay · Li.Fi (executed) · Debridge (quote-only — flat fees too high for small tickets) + +## How it works + +Two loops run simultaneously inside one Go binary: + +1. **Quote loop** (free, every 5 minutes) — fetches a quote from every bridge for every route × every notional ($5 / $50 / $300). Records quote latency, fees, slippage, estimated time. All four bridges benchmarked here. +2. **Execution loop** (paid, fixed UTC times) — actually broadcasts real transactions through Mobula + Relay + Li.Fi. Measures quote → broadcast → settlement end-to-end latency and revert rates. Debridge is excluded from execution. + +On top of those: + +- **Wallet balance metrics** refreshed every 5 minutes +- **Alertmanager → Slack** with balance / gas / consecutive-failure / stale-data alerts +- **Daily P&L cron** at 09:30 UTC (per-chain wallet diff vs 24 h ago) with tier viability health check +- **Pre-flight tier simulation** — a tier whose full cycle can't complete is skipped cleanly with a single Slack message + +## Test routes + +### USDC triangle + +``` + Solana USDC + ↗ ↘ + R3 R1 + ↗ ↘ +Arbitrum USDT ←──R2─── Base USDC +``` + +Capital circulates through the triangle — only bridge fees + gas are consumed. Each tier executes one bridge at a time round-trip, then the next bridge starts a fresh triangle with the replenished wallet (peak capital = 1× tier per leg, not 3×). + +| Route | From | To | Schedule | +| --- | --- | --- | --- | +| R1 | USDC (Solana) | USDC (Base) | Tiers $5/$50/$300 | +| R2 | USDC (Base) | USDT (Arbitrum) | Tiers $5/$50/$300 | +| R3 | USDT (Arbitrum) | USDC (Solana) | Tiers $5/$50/$300 | +| R4 | TRUMP (Solana) | BRETT (Base) | Weekly $5 only (one-way) | +| R5 | USDC (Arbitrum) | USDC (HyperCore) | Quote-only $5/$50/$300 | + +## Metrics produced + +### Bridge performance + +| Metric | Description | +| --- | --- | +| `bridge_quote_latency_ms` | Time to receive quote | +| `bridge_execution_latency_ms` | Broadcast → funds received | +| `bridge_e2e_latency_ms` | Quote start → funds received | +| `bridge_fees_usd` / `bridge_fees_percent` | Bridge fees only | +| `bridge_cost_usd` / `bridge_cost_percent` | Total cost = fees + slippage + destination gas | +| `bridge_slippage_usd` / `bridge_gas_usd` / `bridge_fix_fee_usd` | Cost breakdown | +| `bridge_estimated_time_ms` | Bridge's own promise from the quote | +| `bridge_output_usd` | USD received on destination | +| `bridge_quote_success` | 1 on success, 0 on error | +| `bridge_success_total` / `bridge_reverts_total` / `bridge_errors_total` | Outcome counters | +| `bridge_consecutive_failures` | Streak length (resets on success) — drives paging alert | + +### Wallet state + +| Metric | Description | +| --- | --- | +| `wallet_balance_usd{chain,token}` | Current USD per leg, refreshed every 5 min | +| `wallet_balance_last_update_timestamp_seconds` | Last refresh timestamp (drives staleness alert) | + +The site queries these via the YAML specs — `bridge-quote-latency.yml` reads `bridge_quote_latency_ms_*`, `bridge-fee.yml` reads `bridge_cost_percent`. + +## Alerts + +### Bridge performance / reliability + +- `HighBridgeQuoteLatency` — p95 > 2 s for 5 m (warn) +- `HighBridgeExecutionLatency` — p95 > 2 min for 5 m (warn) +- `HighBridgeFees` — fees > 5 % for 15 m (warn) +- `HighBridgeRevertRate` — > 10 % for 5 m (critical) +- `BridgeConsecutiveFailures` — ≥3 fails in a row (critical) — bridge-side or integration broken +- `BridgeMonitorDown` — monitor unreachable 2 m (critical) + +### Wallet / capital + +- `LowTriangleBalance` — any USDC/USDT leg < $50 (warn) +- `CriticalTriangleBalance` — leg < $10 (critical) +- `LowGasBalance` — ETH/SOL < $3 (warn) +- `WalletBalanceStale` — no refresh for 20 min (warn) + +## Run locally + +```bash +cp .env.example .env +# Fill in API keys, wallet keys (or leave dry-run defaults) +docker-compose -f deploy/docker-compose.yml up -d +``` + +Access: + +- Prometheus: +- Grafana: (anonymous viewer) +- Monitor metrics: +- Alertmanager: + +If `EXECUTION_MODE=dry-run` (default) the monitor only fetches quotes — no transactions, no fees, no wallets needed. Quote-loop metrics still flow into Prometheus normally. + +## Execution modes + +| Mode | What it does | +| --- | --- | +| `dry-run` | Quotes only. Default. No TX, no fees. | +| `single-test` | One real TX per route at `TEST_AMOUNT_USD` (default $1) then exits. | +| `production` | Full loop + scheduler. Real TX at 10:00 UTC. | + +```bash +go run ./cmd/monitor/ # dry-run +EXECUTION_MODE=single-test TEST_AMOUNT_USD=0.50 go run ./cmd/monitor/ +EXECUTION_MODE=production go run ./cmd/monitor/ +``` + +## Deploy on Railway + +Four services from this repo, all rooted at `harnesses/bridge-monitor/`: + +| Service | Root | Dockerfile | Public | Env vars | +| --- | --- | --- | --- | --- | +| **bridge-monitor** | `harnesses/bridge-monitor` | `./Dockerfile` | no | wallet keys, API keys, `EXECUTION_MODE`, `SLACK_WEBHOOK_URL` | +| **prometheus** | `harnesses/bridge-monitor/deploy/prometheus` | `./Dockerfile` | yes (read-only) | none | +| **alertmanager** | `harnesses/bridge-monitor/deploy/alertmanager` | `./Dockerfile` | no | `SLACK_WEBHOOK_URL` | +| **grafana** | `harnesses/bridge-monitor/deploy/grafana` | `./Dockerfile` | yes | none | + +Internal networking: `bridge-monitor.railway.internal:9090`, `prometheus.railway.internal:9090`, `alertmanager.railway.internal:9093`. Detailed instructions in [`deploy/DEPLOY.md`](./deploy/DEPLOY.md). + +## Project layout + +``` +cmd/monitor/ Main binary — quote loop, execution, scheduler + ├── main.go + ├── config.go Env loading + ├── metrics.go Prometheus metric definitions + ├── scheduler.go Fixed UTC time scheduling + ├── executor.go Per-route balance check, exec orchestration + ├── cycle_sim.go Pre-flight viability simulation + ├── tx_executor.go TX signing + broadcast (overridable RPCs) + ├── wallet.go Wallet manager + ├── balance.go Balance fetch + Prometheus export + ├── onchain_balance.go Post-fill balance reader (EVM + Solana + HL) + ├── wallet_snapshot.go Daily 09:30 P&L cron + tier health grid + ├── slack.go Per-exec, skip, P&L, startup notifs + ├── pricer.go USD pricing + ├── mobula_bridge.go Mobula integration + ├── relay_bridge.go Relay integration + ├── lifi_bridge.go Li.Fi integration + └── debridge_bridge.go Debridge integration (quote-only) + +cmd/rebalance/ Manual one-shot rebalance tool (unused in prod) + +deploy/ + ├── docker-compose.yml Local full stack + ├── prometheus/ Scrape config + alert_rules.yml + Dockerfile + ├── alertmanager/ envsubst entrypoint + Slack template + Dockerfile + └── grafana/ Dashboards + datasource provisioning + Dockerfile + +Dockerfile Monitor binary (multi-stage Go build) +``` + +## Bridge integration details + +### Mobula + +- Quote: `GET /api/2/bridge/quote?...&apiKey=...` +- TX data: `deposit.solana.serializedTx` (Solana) or `deposit.evm` (EVM) +- 2-step EVM (approve + deposit) handled +- Status: `GET /api/2/bridge/status/{txHash}`, terminal `"settled"` + +### Relay + +- Quote: `POST /quote` JSON body +- TX data: `steps[].items[].data.instructions[]` (Solana) or `steps[].items[].data` (EVM) +- 2-step EVM (approve + deposit) handled — iterates `steps[]` to find approve before deposit +- Request ID: `steps[].requestId` (deposit step) +- Status: `GET /intents/status/v3?requestId=...` + +### Li.Fi + +- Quote: `GET /v1/quote?...` with `x-lifi-api-key` header +- TX data: `transactionRequest.data` (Solana base64) or `transactionRequest` (EVM) +- Approval address: `estimate.approvalAddress` (EVM) +- Status: `GET /v1/status?txHash=...&fromChain=...&toChain=...` + +### Debridge + +- Quote only — execution disabled because flat native-token fees make $5 tickets unprofitable. + +## Wallet funding (production mode only) + +| Chain | Token | Minimum | Purpose | +| --- | --- | --- | --- | +| Solana | SOL | $5 | Gas (~$0.01/TX) | +| Solana | USDC | $T for tier $T | R1 source | +| Solana | TRUMP | $20+ | R4 (weekly $5) | +| Base | USDC | $0+ (R1 auto-fills before R2 needs it) | R2 source | +| Base | ETH | $5 | Gas (~$0.10/TX) | +| Arbitrum | USDT | $0+ (R2 auto-fills before R3 needs it) | R3 source | +| Arbitrum | ETH | $5 | Gas (~$0.10/TX) | + +Each bridge runs the triangle sequentially and settles before the next bridge starts, so the **peak capital need on any leg = 1× tier** (not 3×). Pre-flight `SimulateTriangleCycle` checks this threshold before every scheduled tier — non-viable tiers skip cleanly with a single Slack message. + +## Environment variables + +| Var | Required | Notes | +| --- | --- | --- | +| `MOBULA_API_KEY` | yes | Quotes + balance API | +| `LIFI_API_KEY` | recommended | Better rate limits | +| `RELAY_API_KEY` | optional | Public quotes work without it | +| `WALLET_EVM_PRIVATE_KEY` | for prod | Hex `0x...` | +| `WALLET_EVM_ADDRESS` | for prod | | +| `WALLET_SOL_PRIVATE_KEY` | for prod | Base58 | +| `WALLET_SOL_ADDRESS` | for prod | | +| `EXECUTION_MODE` | yes | `dry-run` / `single-test` / `production` | +| `TEST_AMOUNT_USD` | for `single-test` | Default $1 | +| `MAX_DAILY_SPEND_USD` | no | Safety cap, default $10 | +| `MONITOR_REGION` | yes | Metric label | +| `SLACK_WEBHOOK_URL` | yes if alerts | Used by both monitor and alertmanager | +| `SOLANA_RPC` / `BASE_RPC` / `ARB_RPC` | no | Override public RPCs | + +## Safety features + +1. **Per-tier pre-flight** — full cycle simulated before execution, skip cleanly if not viable. +2. **Per-route balance check** — secondary guard during `RunReal` (1.05× buffer). +3. **Daily spending cap** — `MAX_DAILY_SPEND_USD` hard stops runaway. +4. **Consecutive-failure streak** — 3 fails in a row on the same bridge → critical alert. +5. **Dry-run / single-test default** — no accidental production runs. +6. **EVM revert detection** — TX receipt inspected. +7. **Gas price +50 % bump** on Arbitrum — handles rapid base-fee variance. + +## Estimated monthly cost (production mode) + +Based on $0.50 single-test validation: + +| Tier | Fees/cycle | Cycles/mo | Monthly | +| --- | --- | --- | --- | +| Triangle $5 daily (9 TX) | ~$0.48 | 30 | $14.40 | +| Triangle $50 Mon+Thu | ~$1.62 | 8.5 | $13.80 | +| Triangle $300 weekly Mon | ~$4.66 | 4.3 | $20.04 | +| R4 TRUMP $5 weekly | ~$0.60 | 4.3 | $2.60 | +| **On-chain subtotal** | | | **~$51/mo** | +| Railway (4 services) | | | **~$25/mo** | +| **Total** | | | **~$76/mo** | + +Variance up to ~$90/mo during gas spikes. + +## Adding a bridge + +1. Create `cmd/monitor/_bridge.go` mirroring an existing integration (e.g. `relay_bridge.go`). +2. Implement quote, execute, status methods conforming to the `Bridge` interface. +3. Add API key field to `Config` and `.env.example`. +4. Register the bridge in `main.go`'s bridge list. +5. Update Grafana dashboards if you want it visualised internally. + +## License + +MIT — same as the rest of OpenChainBench. diff --git a/harnesses/bridge-monitor/cmd/rebalance/main.go b/harnesses/bridge-monitor/cmd/rebalance/main.go new file mode 100644 index 00000000..de5f8be2 --- /dev/null +++ b/harnesses/bridge-monitor/cmd/rebalance/main.go @@ -0,0 +1,193 @@ +package main + +import ( + "bufio" + "fmt" + "log" + "os" + "strings" + "time" +) + +// Test 2 — manual Mobula HyperCore deposit benchmark. +// Bridges $5 of Base USDC into the EVM wallet's Hyperliquid perp account. +// Used once to validate the on-chain fill measurement path before wiring the +// HL clearinghouse balance reader into the main monitor. + +func main() { + log.Println("🧪 Test 2 — Mobula Base USDC → HyperCore deposit") + log.Println("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━") + + config := loadEnvSimple() + if config.mobulaAPIKey == "" || config.evmPrivateKey == "" { + log.Fatal("❌ Missing required env vars (MOBULA_API_KEY, WALLET_EVM_PRIVATE_KEY)") + } + + amountUSD := 5.0 + if v := os.Getenv("TEST_AMOUNT_USD"); v != "" { + fmt.Sscanf(v, "%f", &amountUSD) + } + + log.Printf("EVM Address (sender + HC recipient): %s", config.evmAddress) + log.Printf("Plan: Base USDC $%.2f → HyperCore (hl:mainnet) USDC via Mobula", amountUSD) + log.Println("\n⚠️ Real funds, real TX. Costs ~$0.10 gas, $5 capital moves to HL.") + log.Print("Continue? (yes/no): ") + + reader := bufio.NewReader(os.Stdin) + input, _ := reader.ReadString('\n') + if strings.TrimSpace(input) != "yes" { + log.Println("Aborted.") + return + } + + mobula := NewMobulaBridgeSimple(config.mobulaAPIKey) + txExec, err := NewTxExecutorSimple(config.solPrivateKey, config.evmPrivateKey, config.mobulaAPIKey) + if err != nil { + log.Fatalf("❌ Failed to init TX executor: %v", err) + } + + if err := executeBridge(mobula, txExec, BridgeParams{ + FromChain: "evm:8453", // Base + FromToken: "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913", // Base USDC + ToChain: "hl:mainnet", + ToToken: "USDC", + Amount: amountUSD, + SenderAddr: config.evmAddress, + ReceiverAddr: config.evmAddress, + }); err != nil { + log.Fatalf("❌ Test 2 failed: %v", err) + } + + log.Println("\n⏳ Waiting 15s for HL indexer to credit the perp account...") + time.Sleep(15 * time.Second) + log.Println("\n✅ Test 2 deposit complete. Run the post-snapshot curl now to read withdrawable on HL.") +} + +type simpleConfig struct { + mobulaAPIKey string + evmAddress string + solAddress string + evmPrivateKey string + solPrivateKey string +} + +func loadEnvSimple() *simpleConfig { + config := &simpleConfig{} + config.mobulaAPIKey = os.Getenv("MOBULA_API_KEY") + config.evmAddress = os.Getenv("WALLET_EVM_ADDRESS") + config.solAddress = os.Getenv("WALLET_SOL_ADDRESS") + config.evmPrivateKey = os.Getenv("WALLET_EVM_PRIVATE_KEY") + config.solPrivateKey = os.Getenv("WALLET_SOL_PRIVATE_KEY") + + file, err := os.Open(".env") + if err == nil { + defer file.Close() + scanner := bufio.NewScanner(file) + for scanner.Scan() { + line := strings.TrimSpace(scanner.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + parts := strings.SplitN(line, "=", 2) + if len(parts) != 2 { + continue + } + key, value := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1]) + value = strings.Trim(value, `"'`) + switch key { + case "MOBULA_API_KEY": + if config.mobulaAPIKey == "" { + config.mobulaAPIKey = value + } + case "WALLET_EVM_ADDRESS": + if config.evmAddress == "" { + config.evmAddress = value + } + case "WALLET_SOL_ADDRESS": + if config.solAddress == "" { + config.solAddress = value + } + case "WALLET_EVM_PRIVATE_KEY": + if config.evmPrivateKey == "" { + config.evmPrivateKey = value + } + case "WALLET_SOL_PRIVATE_KEY": + if config.solPrivateKey == "" { + config.solPrivateKey = value + } + } + } + } + return config +} + +type BridgeParams struct { + FromChain string + FromToken string + ToChain string + ToToken string + Amount float64 + SenderAddr string + ReceiverAddr string + IsSolanaSource bool +} + +func executeBridge(mobula *MobulaBridgeSimple, txExec *TxExecutorSimple, params BridgeParams) error { + log.Printf("\n📝 Getting quote for $%.2f...", params.Amount) + quote, err := mobula.GetQuote(params) + if err != nil { + return fmt.Errorf("quote failed: %w", err) + } + log.Printf("✅ Quote received: fee=$%s, output=$%s, time=%dms", + quote.Data.Fees.TotalFeeUsd, quote.Data.EstimatedAmountOutUsd, 0) + + approveIdx := -1 + for i, step := range quote.Data.Steps { + if step.Type == "approve" { + approveIdx = i + break + } + } + + if approveIdx >= 0 { + step := quote.Data.Steps[approveIdx] + log.Printf("📝 Sending approval TX → %s", step.Tx.To) + approvalHash, err := txExec.ExecuteEVMTx(params.FromChain, step.Tx.To, step.Tx.Data, step.Tx.Value) + if err != nil { + return fmt.Errorf("approval failed: %w", err) + } + log.Printf("✅ Approval TX: %s", approvalHash) + log.Printf("⏳ Waiting 15s for confirmation...") + time.Sleep(15 * time.Second) + } + + var txHash string + if quote.Data.Deposit.EVM.To != "" { + log.Printf("📤 Broadcasting EVM bridge TX → %s", quote.Data.Deposit.EVM.To) + txHash, err = txExec.ExecuteEVMTx(params.FromChain, + quote.Data.Deposit.EVM.To, quote.Data.Deposit.EVM.Data, quote.Data.Deposit.EVM.Value) + } else { + for _, step := range quote.Data.Steps { + if step.Type != "approve" { + log.Printf("📤 Broadcasting EVM bridge TX (from steps) → %s", step.Tx.To) + txHash, err = txExec.ExecuteEVMTx(params.FromChain, step.Tx.To, step.Tx.Data, step.Tx.Value) + break + } + } + } + if err != nil { + return fmt.Errorf("broadcast failed: %w", err) + } + log.Printf("✅ Bridge TX broadcast: %s", txHash) + + log.Printf("⏳ Polling Mobula status (timeout 5min)...") + status, err := txExec.PollMobulaStatus(txHash, 5*time.Minute) + if err != nil { + return fmt.Errorf("status poll failed: %w", err) + } + if status.Data.Status == "filled" || status.Data.Status == "settled" { + log.Printf("🎉 Bridge complete! Status: %s | TX: https://basescan.org/tx/%s", status.Data.Status, txHash) + return nil + } + return fmt.Errorf("unexpected final status: %s", status.Data.Status) +} diff --git a/harnesses/bridge-monitor/cmd/rebalance/mobula.go b/harnesses/bridge-monitor/cmd/rebalance/mobula.go new file mode 100644 index 00000000..61f7f7dc --- /dev/null +++ b/harnesses/bridge-monitor/cmd/rebalance/mobula.go @@ -0,0 +1,105 @@ +package main + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "strconv" + "time" +) + +type MobulaBridgeSimple struct { + apiKey string + client *http.Client +} + +type MobulaQuoteResp struct { + Data struct { + EstimatedAmountOut string `json:"estimatedAmountOut"` + EstimatedAmountOutUsd string `json:"estimatedAmountOutUsd"` + Fees struct { + TotalFeeUsd string `json:"totalFeeUsd"` + } `json:"fees"` + Deposit struct { + Solana struct { + SerializedTx string `json:"serializedTx"` + } `json:"solana"` + EVM struct { + To string `json:"to"` + Data string `json:"data"` + Value string `json:"value"` + } `json:"evm"` + } `json:"deposit"` + Steps []struct { + Type string `json:"type"` + Tx struct { + To string `json:"to"` + Data string `json:"data"` + Value string `json:"value"` + } `json:"tx"` + } `json:"steps"` + } `json:"data"` +} + +type MobulaStatusResp struct { + Data struct { + Status string `json:"status"` + } `json:"data"` +} + +func NewMobulaBridgeSimple(apiKey string) *MobulaBridgeSimple { + return &MobulaBridgeSimple{ + apiKey: apiKey, + client: &http.Client{Timeout: 45 * time.Second}, + } +} + +func (m *MobulaBridgeSimple) GetQuote(params BridgeParams) (*MobulaQuoteResp, error) { + url := fmt.Sprintf( + "https://api.mobula.io/api/2/bridge/quote?originChainId=%s&originToken=%s&destinationChainId=%s&destinationToken=%s&amount=%s&walletAddress=%s&senderAddress=%s&apiKey=%s", + params.FromChain, params.FromToken, + params.ToChain, params.ToToken, + strconv.FormatFloat(params.Amount, 'f', -1, 64), + params.ReceiverAddr, params.SenderAddr, + m.apiKey, + ) + + resp, err := m.client.Get(url) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + body, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("API error %d: %s", resp.StatusCode, string(body)) + } + + var quote MobulaQuoteResp + if err := json.Unmarshal(body, "e); err != nil { + return nil, err + } + + return "e, nil +} + +func (m *MobulaBridgeSimple) GetStatus(txHash string) (*MobulaStatusResp, error) { + url := fmt.Sprintf("https://api.mobula.io/api/2/bridge/status/%s", txHash) + + req, _ := http.NewRequest("GET", url, nil) + req.Header.Set("Authorization", m.apiKey) + + resp, err := m.client.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + var status MobulaStatusResp + if err := json.NewDecoder(resp.Body).Decode(&status); err != nil { + return nil, err + } + + return &status, nil +} diff --git a/harnesses/bridge-monitor/cmd/rebalance/relay_simple.go b/harnesses/bridge-monitor/cmd/rebalance/relay_simple.go new file mode 100644 index 00000000..653f0a92 --- /dev/null +++ b/harnesses/bridge-monitor/cmd/rebalance/relay_simple.go @@ -0,0 +1,94 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "time" +) + +type RelayQuoteReq struct { + User string `json:"user"` + OriginChainID int64 `json:"originChainId"` + DestinationChainID int64 `json:"destinationChainId"` + OriginCurrency string `json:"originCurrency"` + DestinationCurrency string `json:"destinationCurrency"` + Amount string `json:"amount"` + TradeType string `json:"tradeType"` + Recipient string `json:"recipient"` +} + +type RelayQuoteResp struct { + Details struct { + CurrencyIn struct{ AmountUsd string `json:"amountUsd"` } `json:"currencyIn"` + CurrencyOut struct{ AmountUsd string `json:"amountUsd"` } `json:"currencyOut"` + } `json:"details"` + Steps []struct { + ID string `json:"id"` + RequestId string `json:"requestId"` + Items []struct { + Data struct { + To string `json:"to"` + Data string `json:"data"` + Value string `json:"value"` + } `json:"data"` + Check struct { + Endpoint string `json:"endpoint"` + } `json:"check"` + } `json:"items"` + } `json:"steps"` +} + +type RelayStatus struct { + Status string `json:"status"` + Details string `json:"details"` +} + +func relayQuote(req RelayQuoteReq) (*RelayQuoteResp, error) { + body, _ := json.Marshal(req) + httpReq, _ := http.NewRequest("POST", "https://api.relay.link/quote", bytes.NewReader(body)) + httpReq.Header.Set("Content-Type", "application/json") + client := &http.Client{Timeout: 45 * time.Second} + resp, err := client.Do(httpReq) + if err != nil { + return nil, err + } + defer resp.Body.Close() + raw, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("relay %d: %s", resp.StatusCode, string(raw)) + } + var out RelayQuoteResp + if err := json.Unmarshal(raw, &out); err != nil { + return nil, err + } + return &out, nil +} + +func relayPollStatus(checkURL string, timeout time.Duration) (*RelayStatus, error) { + deadline := time.Now().Add(timeout) + client := &http.Client{Timeout: 10 * time.Second} + for time.Now().Before(deadline) { + resp, err := client.Get(checkURL) + if err != nil { + time.Sleep(5 * time.Second) + continue + } + raw, _ := io.ReadAll(resp.Body) + resp.Body.Close() + var s RelayStatus + if err := json.Unmarshal(raw, &s); err == nil { + fmt.Printf(" Status: %s\n", s.Status) + if strings.EqualFold(s.Status, "success") || strings.EqualFold(s.Status, "filled") || + strings.EqualFold(s.Status, "settled") || strings.EqualFold(s.Status, "refund") || + strings.EqualFold(s.Status, "failure") { + return &s, nil + } + } + time.Sleep(5 * time.Second) + } + return nil, fmt.Errorf("timeout polling Relay status") +} diff --git a/harnesses/bridge-monitor/cmd/rebalance/tx_executor.go b/harnesses/bridge-monitor/cmd/rebalance/tx_executor.go new file mode 100644 index 00000000..8d7091a6 --- /dev/null +++ b/harnesses/bridge-monitor/cmd/rebalance/tx_executor.go @@ -0,0 +1,197 @@ +package main + +import ( + "context" + "encoding/base64" + "encoding/hex" + "fmt" + "log" + "math/big" + "strings" + "time" + + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + "github.com/ethereum/go-ethereum/crypto" + "github.com/ethereum/go-ethereum/ethclient" + "github.com/gagliardetto/solana-go" + "github.com/gagliardetto/solana-go/rpc" +) + +type TxExecutorSimple struct { + solanaKey solana.PrivateKey + evmKey []byte + mobulaAPIKey string + mobula *MobulaBridgeSimple +} + +var chainRPCs = map[string]string{ + "evm:8453": "https://mainnet.base.org", + "evm:42161": "https://arb1.arbitrum.io/rpc", +} + +func NewTxExecutorSimple(solPrivKey, evmPrivKey, mobulaAPIKey string) (*TxExecutorSimple, error) { + // Parse Solana key + solKey, err := solana.PrivateKeyFromBase58(solPrivKey) + if err != nil { + return nil, fmt.Errorf("invalid solana key: %w", err) + } + + // Parse EVM key + evmKeyHex := strings.TrimPrefix(evmPrivKey, "0x") + evmKey, err := hex.DecodeString(evmKeyHex) + if err != nil { + return nil, fmt.Errorf("invalid evm key: %w", err) + } + + return &TxExecutorSimple{ + solanaKey: solKey, + evmKey: evmKey, + mobulaAPIKey: mobulaAPIKey, + mobula: NewMobulaBridgeSimple(mobulaAPIKey), + }, nil +} + +func (tx *TxExecutorSimple) ExecuteSolanaTx(serializedTxBase64 string) (string, error) { + // Decode base64 TX + txBytes, err := base64.StdEncoding.DecodeString(serializedTxBase64) + if err != nil { + return "", fmt.Errorf("failed to decode tx: %w", err) + } + + // Deserialize transaction + transaction, err := solana.TransactionFromBytes(txBytes) + if err != nil { + return "", fmt.Errorf("failed to deserialize tx: %w", err) + } + + // Connect to Solana + client := rpc.New("https://api.mainnet-beta.solana.com") + + // Get fresh blockhash (the one in the TX may have expired) + recent, err := client.GetLatestBlockhash(context.Background(), rpc.CommitmentFinalized) + if err != nil { + return "", fmt.Errorf("failed to get blockhash: %w", err) + } + transaction.Message.RecentBlockhash = recent.Value.Blockhash + + // Clear old signatures and re-sign + transaction.Signatures = nil + _, err = transaction.Sign(func(key solana.PublicKey) *solana.PrivateKey { + if key.Equals(tx.solanaKey.PublicKey()) { + return &tx.solanaKey + } + return nil + }) + if err != nil { + return "", fmt.Errorf("failed to sign tx: %w", err) + } + + // Send transaction + sig, err := client.SendTransaction(context.Background(), transaction) + if err != nil { + return "", fmt.Errorf("failed to send tx: %w", err) + } + + return sig.String(), nil +} + +func (tx *TxExecutorSimple) ExecuteEVMTx(chain, to, data, value string) (string, error) { + rpcURL, ok := chainRPCs[chain] + if !ok { + return "", fmt.Errorf("unknown chain: %s", chain) + } + + client, err := ethclient.Dial(rpcURL) + if err != nil { + return "", fmt.Errorf("failed to connect: %w", err) + } + defer client.Close() + + // Parse private key + privateKey, err := crypto.ToECDSA(tx.evmKey) + if err != nil { + return "", fmt.Errorf("invalid key: %w", err) + } + + fromAddr := crypto.PubkeyToAddress(privateKey.PublicKey) + toAddr := common.HexToAddress(to) + + // Parse data + dataBytes, err := hex.DecodeString(strings.TrimPrefix(data, "0x")) + if err != nil { + return "", fmt.Errorf("invalid data: %w", err) + } + + // Parse value + valueBig := new(big.Int) + if value != "" && value != "0" { + value = strings.TrimPrefix(value, "0x") + valueBig.SetString(value, 16) + } + + // Get nonce + nonce, err := client.PendingNonceAt(context.Background(), fromAddr) + if err != nil { + return "", fmt.Errorf("failed to get nonce: %w", err) + } + + // Get gas price + 50% buffer to handle base-fee variance (esp. Arbitrum) + gasPrice, err := client.SuggestGasPrice(context.Background()) + if err != nil { + return "", fmt.Errorf("failed to get gas price: %w", err) + } + gasPrice = new(big.Int).Mul(gasPrice, big.NewInt(150)) + gasPrice = new(big.Int).Div(gasPrice, big.NewInt(100)) + + // Estimate gas + gasLimit := uint64(500000) // Safe default + + // Get chain ID + chainID, err := client.NetworkID(context.Background()) + if err != nil { + return "", fmt.Errorf("failed to get chain ID: %w", err) + } + + // Create transaction + txData := types.NewTransaction(nonce, toAddr, valueBig, gasLimit, gasPrice, dataBytes) + + // Sign transaction + signedTx, err := types.SignTx(txData, types.NewEIP155Signer(chainID), privateKey) + if err != nil { + return "", fmt.Errorf("failed to sign: %w", err) + } + + // Send transaction + err = client.SendTransaction(context.Background(), signedTx) + if err != nil { + return "", fmt.Errorf("failed to send: %w", err) + } + + return signedTx.Hash().Hex(), nil +} + +func (tx *TxExecutorSimple) PollMobulaStatus(txHash string, timeout time.Duration) (*MobulaStatusResp, error) { + deadline := time.Now().Add(timeout) + pollInterval := 5 * time.Second + + for time.Now().Before(deadline) { + status, err := tx.mobula.GetStatus(txHash) + if err != nil { + log.Printf(" Status poll error: %v", err) + time.Sleep(pollInterval) + continue + } + + log.Printf(" Status: %s", status.Data.Status) + + if status.Data.Status == "filled" || status.Data.Status == "settled" || + status.Data.Status == "refunded" || status.Data.Status == "failed" { + return status, nil + } + + time.Sleep(pollInterval) + } + + return nil, fmt.Errorf("timeout waiting for status") +} diff --git a/harnesses/bridge-monitor/deploy/DEPLOY.md b/harnesses/bridge-monitor/deploy/DEPLOY.md new file mode 100644 index 00000000..78e9a882 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/DEPLOY.md @@ -0,0 +1,158 @@ +# Railway Deployment Guide + +## Architecture + +4 Railway services from the GitHub repo: + +``` +miniapps/bridging-latency/ +├── Dockerfile → Service "bridge-monitor" +└── deploy/ + ├── prometheus/Dockerfile → Service "prometheus" + ├── grafana/Dockerfile → Service "grafana" + └── alertmanager/Dockerfile → Service "alertmanager" +``` + +--- + +## Deployment Steps + +### 1. Service: bridge-monitor + +**Railway Config:** +- **Root Directory**: `miniapps/bridging-latency` +- **Dockerfile Path**: `./Dockerfile` +- **Port**: 9090 + +**Environment Variables:** +```env +MOBULA_API_KEY=your_key +WALLET_EVM_ADDRESS=0x... +WALLET_SOL_ADDRESS=... +EXECUTION_MODE=dry-run +MONITOR_REGION=railway-prod +``` + +**Service Name:** `bridge-monitor` + +--- + +### 2. Service: prometheus + +**Railway Config:** +- **Root Directory**: `miniapps/bridging-latency/deploy/prometheus` +- **Dockerfile Path**: `./Dockerfile` +- **Port**: 9090 + +**Service Name:** `prometheus` + +**Networking:** Accessible via `prometheus.railway.internal:9090` + +--- + +### 3. Service: grafana (Public Dashboard) + +**Railway Config:** +- **Root Directory**: `miniapps/bridging-latency/deploy/grafana` +- **Dockerfile Path**: `./Dockerfile` +- **Port**: 3000 + +**Service Name:** `grafana` + +**Networking:** +- ✅ **Generate public domain** (publicly accessible dashboard) +- Internal: `grafana.railway.internal:3000` + +--- + +### 4. Service: alertmanager + +**Railway Config:** +- **Root Directory**: `miniapps/bridging-latency/deploy/alertmanager` +- **Dockerfile Path**: `./Dockerfile` +- **Port**: 9093 + +**Service Name:** `alertmanager` + +**Networking:** `alertmanager.railway.internal:9093` + +--- + +## Recommended Order + +1. **prometheus** (no dependencies) +2. **alertmanager** (no dependencies) +3. **bridge-monitor** (scrapes to prometheus) +4. **grafana** (datasource to prometheus) + +--- + +## Verification + +### 1. Bridge Monitor +```bash +curl https://.railway.app/metrics +``` + +### 2. Prometheus +- Go to **Status → Targets** +- Verify `bridge-monitor` is **UP** + +### 3. Grafana (Public Dashboard) +- Dashboard displays without login (anonymous auth) +- Graphs show data after 2-3 minutes + +--- + +## Local Development + +Run the full stack locally: +```bash +cd deploy +docker-compose up -d +``` + +Access: +- **Prometheus**: http://localhost:9091 +- **Grafana**: http://localhost:3000 +- **Monitor Metrics**: http://localhost:9090/metrics +- **Alertmanager**: http://localhost:9093 + +--- + +## Key Points + +✅ **Service Names** must be exactly: `bridge-monitor`, `prometheus`, `grafana`, `alertmanager` +✅ Railway internal URLs: `.railway.internal` +✅ Grafana has **anonymous auth** = public access without login +✅ Prometheus scrapes bridge-monitor every 15s +✅ Quote tests run every **5 minutes** +✅ Execution tests run at fixed UTC times (10:00 UTC) + +--- + +## Estimated Cost + +- **Railway Hobby:** $5/service × 4 = **$20/month** +- **Execution costs:** ~$100/month (bridge fees for real transactions) + +--- + +## Troubleshooting + +### Prometheus not scraping bridge-monitor +```bash +railway logs prometheus +``` +Verify `bridge-monitor.railway.internal:9090` is accessible. + +### Grafana shows "No Data" +- Wait 2-3 minutes for data collection +- Check datasource: Settings → Data Sources → Prometheus +- URL should be: `http://prometheus.railway.internal:9090` +- Test query: `up` + +### Service crashes on startup +- Check logs: `railway logs ` +- Verify port matches (9090, 3000, 9093) +- Verify Dockerfile path is correct diff --git a/harnesses/bridge-monitor/deploy/alertmanager/Dockerfile b/harnesses/bridge-monitor/deploy/alertmanager/Dockerfile new file mode 100644 index 00000000..b35cdf77 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/alertmanager/Dockerfile @@ -0,0 +1,15 @@ +FROM prom/alertmanager:latest + +USER root + +# Newer prom/alertmanager images are busybox-based (no apk). We use plain sed in +# entrypoint.sh to substitute SLACK_WEBHOOK_URL — no envsubst dependency needed. + +COPY alertmanager.yml.tmpl /etc/alertmanager/alertmanager.yml.tmpl +COPY templates.tmpl /etc/alertmanager/templates.tmpl +COPY entrypoint.sh /entrypoint.sh +RUN chmod +x /entrypoint.sh + +EXPOSE 9093 + +ENTRYPOINT ["/entrypoint.sh"] diff --git a/harnesses/bridge-monitor/deploy/alertmanager/alertmanager.yml.tmpl b/harnesses/bridge-monitor/deploy/alertmanager/alertmanager.yml.tmpl new file mode 100644 index 00000000..f7d4eeea --- /dev/null +++ b/harnesses/bridge-monitor/deploy/alertmanager/alertmanager.yml.tmpl @@ -0,0 +1,41 @@ +global: + resolve_timeout: 5m + slack_api_url: '${SLACK_WEBHOOK_URL}' + +templates: + - '/etc/alertmanager/templates.tmpl' + +route: + receiver: 'slack-warnings' + group_by: ['alertname', 'chain'] + group_wait: 30s + group_interval: 5m + repeat_interval: 6h + routes: + - matchers: + - severity = "critical" + receiver: 'slack-criticals' + group_wait: 10s + repeat_interval: 1h + +receivers: + - name: 'slack-warnings' + slack_configs: + - channel: '#bridge-bench' + send_resolved: true + title: '{{ template "slack.title" . }}' + text: '{{ template "slack.text" . }}' + + - name: 'slack-criticals' + slack_configs: + - channel: '#bridge-bench' + send_resolved: true + title: ':rotating_light: CRITICAL — {{ template "slack.title" . }}' + text: '{{ template "slack.text" . }}' + +inhibit_rules: + - source_matchers: + - severity = "critical" + target_matchers: + - severity = "warning" + equal: ['alertname', 'chain'] diff --git a/harnesses/bridge-monitor/deploy/alertmanager/entrypoint.sh b/harnesses/bridge-monitor/deploy/alertmanager/entrypoint.sh new file mode 100644 index 00000000..73ca47ff --- /dev/null +++ b/harnesses/bridge-monitor/deploy/alertmanager/entrypoint.sh @@ -0,0 +1,19 @@ +#!/bin/sh +set -e + +# Alertmanager does not expand env vars in its YAML by itself. Substitute the +# SLACK_WEBHOOK_URL placeholder at container start using sed (busybox-safe — no +# envsubst dependency). The Slack webhook URL contains only /, alphanumerics and +# colons so | as the sed delimiter is safe. +if [ -z "$SLACK_WEBHOOK_URL" ]; then + echo "⚠️ SLACK_WEBHOOK_URL not set — alerts will be dropped by Slack receiver." +fi + +sed "s|\${SLACK_WEBHOOK_URL}|${SLACK_WEBHOOK_URL}|g" \ + /etc/alertmanager/alertmanager.yml.tmpl \ + > /etc/alertmanager/alertmanager.yml + +exec /bin/alertmanager \ + --config.file=/etc/alertmanager/alertmanager.yml \ + --storage.path=/alertmanager \ + --log.level=info diff --git a/harnesses/bridge-monitor/deploy/alertmanager/templates.tmpl b/harnesses/bridge-monitor/deploy/alertmanager/templates.tmpl new file mode 100644 index 00000000..c4a344ed --- /dev/null +++ b/harnesses/bridge-monitor/deploy/alertmanager/templates.tmpl @@ -0,0 +1,19 @@ +{{ define "slack.title" }} +{{- if eq .Status "firing" -}} + [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} +{{- else -}} + [RESOLVED] {{ .CommonLabels.alertname }} +{{- end -}} +{{ end }} + +{{ define "slack.text" }} +{{ range .Alerts -}} +*{{ .Annotations.summary }}* +{{ .Annotations.description }} +{{ if .Labels.bridge }}• bridge: `{{ .Labels.bridge }}`{{ end }} +{{ if .Labels.from_chain }}• route: `{{ .Labels.from_chain }} → {{ .Labels.to_chain }}`{{ end }} +{{ if .Labels.chain }}• chain: `{{ .Labels.chain }}`{{ end }} +• severity: `{{ .Labels.severity }}` +• started: {{ .StartsAt.Format "2006-01-02 15:04:05 UTC" }} +{{ end }} +{{ end }} diff --git a/harnesses/bridge-monitor/deploy/docker-compose.yml b/harnesses/bridge-monitor/deploy/docker-compose.yml new file mode 100644 index 00000000..d9e8fb95 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/docker-compose.yml @@ -0,0 +1,80 @@ +version: '3.8' + +services: + monitor: + build: + context: .. + dockerfile: Dockerfile + container_name: bridge-monitor + restart: unless-stopped + env_file: + - ../.env + ports: + - "9090:9090" + networks: + - monitoring + + prometheus: + image: prom/prometheus:latest + container_name: bridge-prometheus + restart: unless-stopped + command: + - '--config.file=/etc/prometheus/prometheus.yml' + - '--storage.tsdb.path=/prometheus' + - '--storage.tsdb.retention.time=365d' + - '--web.enable-lifecycle' + - '--web.enable-admin-api' + volumes: + - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml + - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml + - prometheus_data:/prometheus + ports: + - "9091:9090" + networks: + - monitoring + + alertmanager: + build: + context: ./alertmanager + dockerfile: Dockerfile + container_name: bridge-alertmanager + restart: unless-stopped + environment: + - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL} + volumes: + - alertmanager_data:/alertmanager + ports: + - "9093:9093" + networks: + - monitoring + + grafana: + image: grafana/grafana:latest + container_name: bridge-grafana + restart: unless-stopped + environment: + - GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD:-admin} + - GF_USERS_ALLOW_SIGN_UP=false + - GF_AUTH_ANONYMOUS_ENABLED=true + - GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer + - GF_AUTH_BASIC_ENABLED=false + - GF_AUTH_DISABLE_LOGIN_FORM=true + volumes: + - ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards + - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources + - grafana_data:/var/lib/grafana + ports: + - "3000:3000" + depends_on: + - prometheus + networks: + - monitoring + +networks: + monitoring: + driver: bridge + +volumes: + prometheus_data: + alertmanager_data: + grafana_data: diff --git a/harnesses/bridge-monitor/deploy/grafana/Dockerfile b/harnesses/bridge-monitor/deploy/grafana/Dockerfile new file mode 100644 index 00000000..81553e8e --- /dev/null +++ b/harnesses/bridge-monitor/deploy/grafana/Dockerfile @@ -0,0 +1,19 @@ +FROM grafana/grafana:latest + +USER root + +# Copy provisioning configs (we're in grafana folder) +COPY provisioning /etc/grafana/provisioning + +# Set permissions +RUN chown -R grafana:root /etc/grafana/provisioning + +USER grafana + +# Environment variables +ENV GF_AUTH_ANONYMOUS_ENABLED=true +ENV GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer +ENV GF_SECURITY_ADMIN_USER=admin +ENV GF_SECURITY_ADMIN_PASSWORD=admin + +EXPOSE 3000 diff --git a/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/bridge-latency.json b/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/bridge-latency.json new file mode 100644 index 00000000..ea8ad0a2 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/bridge-latency.json @@ -0,0 +1,2044 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 2000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_quote_latency_ms_sum[24h]) / max_over_time(bridge_quote_latency_ms_count[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} ({{amount_usd}}$) avg", + "refId": "A" + } + ], + "title": "Quote Latency (p50/p95)", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "currencyUSD" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 0 + }, + "id": 3, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "bridge_fees_usd", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} ({{amount_usd}}$)", + "refId": "A" + } + ], + "title": "Bridge Fees (USD)", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 2 + }, + { + "color": "red", + "value": 5 + } + ] + }, + "unit": "percent" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 8 + }, + "id": 4, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "bridge_fees_percent", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} ({{amount_usd}}$)", + "refId": "A" + } + ], + "title": "Bridge Fees (%)", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 6, + "options": { + "legend": { + "calcs": [ + "sum" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "rate(bridge_success_total[5m])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} success", + "refId": "A" + }, + { + "expr": "rate(bridge_reverts_total[5m])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} reverts", + "refId": "B" + }, + { + "expr": "rate(bridge_errors_total[5m])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} errors", + "refId": "C" + } + ], + "title": "Success/Revert/Error Rate", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 120000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 7, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_execution_latency_ms_sum[24h]) / max_over_time(bridge_execution_latency_ms_count[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} ({{amount_usd}}$) avg", + "refId": "A" + } + ], + "title": "Execution Latency (All)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 24 + }, + "id": 20, + "panels": [], + "title": "📊 Execution by Amount Tier", + "type": "row" + }, + { + "datasource": "Prometheus", + "description": "Daily at 10:00 UTC", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "points", + "fillOpacity": 0, + "lineWidth": 2, + "pointSize": 8, + "showPoints": "always", + "spanNulls": false, + "scaleDistribution": { + "type": "log", + "log": 2 + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 120000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 25 + }, + "id": 21, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_execution_latency_ms_sum{amount_usd=\"5\"}[24h]) / max_over_time(bridge_execution_latency_ms_count{amount_usd=\"5\"}[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} avg", + "refId": "A" + } + ], + "title": "⚡ $5 Execution (Daily)", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "description": "Monday & Thursday at 10:00 UTC", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "points", + "fillOpacity": 0, + "lineWidth": 2, + "pointSize": 8, + "showPoints": "always", + "spanNulls": false, + "scaleDistribution": { + "type": "log", + "log": 2 + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 120000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 25 + }, + "id": 22, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_execution_latency_ms_sum{amount_usd=\"50\"}[24h]) / max_over_time(bridge_execution_latency_ms_count{amount_usd=\"50\"}[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} avg", + "refId": "A" + } + ], + "title": "💰 $50 Execution (Mon/Thu)", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "description": "Monday at 10:00 UTC (weekly)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "points", + "fillOpacity": 0, + "lineWidth": 2, + "pointSize": 8, + "showPoints": "always", + "spanNulls": false, + "scaleDistribution": { + "type": "log", + "log": 2 + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 120000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 25 + }, + "id": 23, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_execution_latency_ms_sum{amount_usd=\"300\"}[24h]) / max_over_time(bridge_execution_latency_ms_count{amount_usd=\"300\"}[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} avg", + "refId": "A" + } + ], + "title": "💎 $300 Execution (weekly Mon)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 33 + }, + "id": 30, + "panels": [], + "title": "🎯 End-to-End Latency by Tier", + "type": "row" + }, + { + "datasource": "Prometheus", + "description": "Quote start → funds received", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "points", + "fillOpacity": 0, + "lineWidth": 2, + "pointSize": 8, + "showPoints": "always", + "spanNulls": false, + "scaleDistribution": { + "type": "log", + "log": 2 + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 180000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 34 + }, + "id": 31, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_e2e_latency_ms_sum{amount_usd=\"5\"}[24h]) / max_over_time(bridge_e2e_latency_ms_count{amount_usd=\"5\"}[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} avg", + "refId": "A" + } + ], + "title": "⚡ $5 E2E Latency", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "description": "Quote start → funds received", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "points", + "fillOpacity": 0, + "lineWidth": 2, + "pointSize": 8, + "showPoints": "always", + "spanNulls": false, + "scaleDistribution": { + "type": "log", + "log": 2 + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 180000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 34 + }, + "id": 32, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_e2e_latency_ms_sum{amount_usd=\"50\"}[24h]) / max_over_time(bridge_e2e_latency_ms_count{amount_usd=\"50\"}[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} avg", + "refId": "A" + } + ], + "title": "💰 $50 E2E Latency", + "type": "timeseries" + }, + { + "datasource": "Prometheus", + "description": "Quote start → funds received", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "points", + "fillOpacity": 0, + "lineWidth": 2, + "pointSize": 8, + "showPoints": "always", + "spanNulls": false, + "scaleDistribution": { + "type": "log", + "log": 2 + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 180000 + } + ] + }, + "unit": "suffix: ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 34 + }, + "id": 33, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "single" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_e2e_latency_ms_sum{amount_usd=\"300\"}[24h]) / max_over_time(bridge_e2e_latency_ms_count{amount_usd=\"300\"}[24h])", + "legendFormat": "{{bridge}} {{from_chain}}→{{to_chain}} avg", + "refId": "A" + } + ], + "title": "💎 $300 E2E Latency", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 42 + }, + "id": 40, + "panels": [], + "title": "📈 Execution Counts by Tier", + "type": "row" + }, + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 6, + "w": 8, + "x": 0, + "y": 43 + }, + "id": 41, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ] + } + }, + "targets": [ + { + "expr": "sum(max_over_time(bridge_success_total{amount_usd=\"5\"}[24h]))", + "legendFormat": "Success", + "refId": "A" + } + ], + "title": "⚡ $5 Executions", + "type": "stat" + }, + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 6, + "w": 8, + "x": 8, + "y": 43 + }, + "id": 42, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ] + } + }, + "targets": [ + { + "expr": "sum(max_over_time(bridge_success_total{amount_usd=\"50\"}[24h]))", + "legendFormat": "Success", + "refId": "A" + } + ], + "title": "💰 $50 Executions", + "type": "stat" + }, + { + "datasource": "Prometheus", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "purple", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 6, + "w": 8, + "x": 16, + "y": 43 + }, + "id": 43, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ] + } + }, + "targets": [ + { + "expr": "sum(max_over_time(bridge_success_total{amount_usd=\"300\"}[24h]))", + "legendFormat": "Success", + "refId": "A" + } + ], + "title": "💎 $300 Executions", + "type": "stat" + }, + { + "id": 50, + "type": "row", + "title": "🪙 HyperCore Deposit Benchmark (Arb USDC → HL Perps)", + "collapsed": false, + "panels": [], + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 50 + } + }, + { + "id": 51, + "type": "timeseries", + "datasource": "Prometheus", + "title": "HyperCore Cost % (Mobula vs Relay vs LiFi)", + "description": "Total cost as % of trade size on Arb USDC → HyperCore deposit. Lower is better. Mobula's $0 service fee shines on small tickets where Relay/LiFi take a flat $1.20 (24% on $5).", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "pointSize": 5, + "showPoints": "auto", + "spanNulls": true + }, + "mappings": [], + "unit": "percent", + "decimals": 3 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 51 + }, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "right" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "expr": "bridge_cost_percent{to_chain=\"HyperCore\"}", + "legendFormat": "{{bridge}} ${{amount_usd}}", + "refId": "A" + } + ] + }, + { + "id": 52, + "type": "timeseries", + "datasource": "Prometheus", + "title": "HyperCore Quote Latency", + "description": "Quote API latency for HyperCore deposit. Measures only the quote step (broadcast not included for Phase 1 quote-only).", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 5, + "lineWidth": 2, + "pointSize": 5, + "showPoints": "auto", + "spanNulls": true, + "scaleDistribution": { + "type": "log", + "log": 2 + } + }, + "mappings": [], + "unit": "ms", + "decimals": 0 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 51 + }, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "right" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "expr": "max_over_time(bridge_quote_latency_ms_sum{to_chain=\"HyperCore\"}[24h]) / max_over_time(bridge_quote_latency_ms_count{to_chain=\"HyperCore\"}[24h])", + "legendFormat": "{{bridge}} ${{amount_usd}} avg", + "refId": "A" + } + ] + }, + { + "id": 53, + "type": "timeseries", + "datasource": "Prometheus", + "title": "HyperCore Fees USD (absolute)", + "description": "Absolute fee paid in USD per provider per tier. Mobula stays flat at gas-only (~$0.10), Relay charges $1.20 flat regardless of amount, LiFi has no route below $50.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 5, + "lineWidth": 2, + "pointSize": 6, + "showPoints": "always", + "spanNulls": true + }, + "mappings": [], + "unit": "currencyUSD", + "decimals": 2 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 59 + }, + "options": { + "legend": { + "calcs": [ + "mean", + "lastNotNull" + ], + "displayMode": "table", + "placement": "right" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "expr": "bridge_cost_usd{to_chain=\"HyperCore\"}", + "legendFormat": "{{bridge}} ${{amount_usd}}", + "refId": "A" + } + ] + }, + { + "id": 54, + "type": "stat", + "datasource": "Prometheus", + "title": "HyperCore Quote Success Rate (24h)", + "description": "Fraction of HyperCore quotes returning successfully over 24h. LiFi will show partial coverage (no route <$50). Debridge skipped (chain not supported).", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 0.5 + }, + { + "color": "green", + "value": 0.95 + } + ] + }, + "unit": "percentunit", + "decimals": 2 + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 59 + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "vertical", + "reduceOptions": { + "values": false, + "calcs": [ + "lastNotNull" + ] + }, + "textMode": "auto" + }, + "targets": [ + { + "expr": "avg by (bridge, amount_usd) (avg_over_time(bridge_quote_success{to_chain=\"HyperCore\"}[24h]))", + "legendFormat": "{{bridge}} ${{amount_usd}}", + "refId": "A" + } + ] + } + ], + "refresh": "30s", + "schemaVersion": 27, + "style": "dark", + "tags": [ + "bridge", + "latency", + "monitoring" + ], + "templating": { + "list": [] + }, + "time": { + "from": "now-7d", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "Bridge Latency Benchmark", + "uid": "bridge-latency", + "version": 0 +} \ No newline at end of file diff --git a/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/bridge-pricing.json b/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/bridge-pricing.json new file mode 100644 index 00000000..5453d0a8 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/bridge-pricing.json @@ -0,0 +1,1619 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": "-- Grafana --", + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "gnetId": null, + "graphTooltip": 0, + "id": null, + "links": [ + { + "asDropdown": false, + "icon": "external link", + "includeVars": false, + "keepTime": false, + "tags": [], + "targetBlank": true, + "title": "Latency Dashboard", + "tooltip": "View execution latency metrics", + "type": "link", + "url": "/d/bridge-latency" + } + ], + "panels": [ + { + "type": "row", + "title": "\ud83c\udfc6 Provider Ranking \u2014 $5 / $50 / $300 (change Route above)", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 10, + "collapsed": false + }, + { + "datasource": "Prometheus", + "type": "table", + "title": "Main Ranking Table (one row per provider)", + "description": "One row per provider. Columns show cost in USD & % for each amount. Select a route via the 'Route' variable above. Sorted by $5 cost ascending.", + "gridPos": { + "h": 12, + "w": 24, + "x": 0, + "y": 1 + }, + "id": 11, + "targets": [ + { + "expr": "bridge_cost_usd{from_token=~\"$route\",amount_usd=\"5\"}", + "refId": "A", + "format": "table", + "instant": true, + "legendFormat": "" + }, + { + "expr": "bridge_cost_percent{from_token=~\"$route\",amount_usd=\"5\"}", + "refId": "B", + "format": "table", + "instant": true, + "legendFormat": "" + }, + { + "expr": "bridge_cost_usd{from_token=~\"$route\",amount_usd=\"50\"}", + "refId": "C", + "format": "table", + "instant": true, + "legendFormat": "" + }, + { + "expr": "bridge_cost_percent{from_token=~\"$route\",amount_usd=\"50\"}", + "refId": "D", + "format": "table", + "instant": true, + "legendFormat": "" + }, + { + "expr": "bridge_cost_usd{from_token=~\"$route\",amount_usd=\"300\"}", + "refId": "E", + "format": "table", + "instant": true, + "legendFormat": "" + }, + { + "expr": "bridge_cost_percent{from_token=~\"$route\",amount_usd=\"300\"}", + "refId": "F", + "format": "table", + "instant": true, + "legendFormat": "" + }, + { + "expr": "bridge_estimated_time_ms{from_token=~\"$route\",amount_usd=\"300\"}", + "refId": "G", + "format": "table", + "instant": true, + "legendFormat": "" + } + ], + "transformations": [ + { + "id": "joinByField", + "options": { + "byField": "bridge", + "mode": "outer" + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time 1": true, + "Time 2": true, + "Time 3": true, + "Time 4": true, + "Time 5": true, + "Time 6": true, + "Time 7": true, + "__name__ 1": true, + "__name__ 2": true, + "__name__ 3": true, + "__name__ 4": true, + "__name__ 5": true, + "__name__ 6": true, + "__name__ 7": true, + "job 1": true, + "job 2": true, + "job 3": true, + "job 4": true, + "job 5": true, + "job 6": true, + "job 7": true, + "instance 1": true, + "instance 2": true, + "instance 3": true, + "instance 4": true, + "instance 5": true, + "instance 6": true, + "instance 7": true, + "region 1": true, + "region 2": true, + "region 3": true, + "region 4": true, + "region 5": true, + "region 6": true, + "region 7": true, + "from_chain 1": true, + "from_chain 2": true, + "from_chain 3": true, + "from_chain 4": true, + "from_chain 5": true, + "from_chain 6": true, + "from_chain 7": true, + "to_chain 1": true, + "to_chain 2": true, + "to_chain 3": true, + "to_chain 4": true, + "to_chain 5": true, + "to_chain 6": true, + "to_chain 7": true, + "from_token 1": true, + "from_token 2": true, + "from_token 3": true, + "from_token 4": true, + "from_token 5": true, + "from_token 6": true, + "from_token 7": true, + "to_token 1": true, + "to_token 2": true, + "to_token 3": true, + "to_token 4": true, + "to_token 5": true, + "to_token 6": true, + "to_token 7": true, + "amount_usd 1": true, + "amount_usd 2": true, + "amount_usd 3": true, + "amount_usd 4": true, + "amount_usd 5": true, + "amount_usd 6": true, + "amount_usd 7": true, + "service 1": true, + "service 2": true, + "service 3": true, + "service 4": true, + "service 5": true, + "service 6": true, + "service 7": true, + "service 8": true, + "service 9": true, + "service": true + }, + "indexByName": { + "bridge": 0, + "Value #A": 1, + "Value #B": 2, + "Value #C": 3, + "Value #D": 4, + "Value #E": 5, + "Value #F": 6, + "Value #G": 7 + }, + "renameByName": { + "bridge": "Provider", + "Value #A": "$5 Cost USD", + "Value #B": "$5 Cost %", + "Value #C": "$50 Cost USD", + "Value #D": "$50 Cost %", + "Value #E": "$300 Cost USD", + "Value #F": "$300 Cost %", + "Value #G": "Est Time ms" + } + } + }, + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "$5 Cost USD", + "desc": false + } + ] + } + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "align": "auto", + "displayMode": "auto" + }, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "Cost USD" + }, + "properties": [ + { + "id": "unit", + "value": "currencyUSD" + }, + { + "id": "decimals", + "value": 3 + }, + { + "id": "custom.displayMode", + "value": "color-background" + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 0.5 + }, + { + "color": "red", + "value": 2 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "Cost %" + }, + "properties": [ + { + "id": "unit", + "value": "percent" + }, + { + "id": "decimals", + "value": 2 + }, + { + "id": "custom.displayMode", + "value": "color-background" + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 0.5 + }, + { + "color": "red", + "value": 5 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Est Time ms" + }, + "properties": [ + { + "id": "unit", + "value": "ms" + }, + { + "id": "decimals", + "value": 0 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Provider" + }, + "properties": [ + { + "id": "custom.width", + "value": 120 + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "type": "row", + "title": "\ud83c\udf69 Cost Distribution \u2014 $5", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 13 + }, + "id": 100, + "collapsed": false + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$5 \u2014 USDC (Sol) \u2192 USDC (Base)", + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 14 + }, + "id": 101, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"5\",from_chain=\"Solana\",to_chain=\"Base\",from_token=\"EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$5 \u2014 USDC (Base) \u2192 USDT (Arb)", + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 14 + }, + "id": 102, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"5\",from_chain=\"Base\",to_chain=\"Arbitrum\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$5 \u2014 TRUMP (Sol) \u2192 BRETT (Base)", + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 14 + }, + "id": 103, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"5\",from_token=\"6p6xgHyF7AeE6TZkSmFsko444wqoP15icUSqi2jfGiPN\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "type": "row", + "title": "\ud83c\udf69 Cost Distribution \u2014 $50", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 23 + }, + "id": 200, + "collapsed": false + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$50 \u2014 USDC (Sol) \u2192 USDC (Base)", + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 24 + }, + "id": 201, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"50\",from_chain=\"Solana\",to_chain=\"Base\",from_token=\"EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$50 \u2014 USDC (Base) \u2192 USDT (Arb)", + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 24 + }, + "id": 202, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"50\",from_chain=\"Base\",to_chain=\"Arbitrum\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$50 \u2014 TRUMP (Sol) \u2192 BRETT (Base)", + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 24 + }, + "id": 203, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"50\",from_token=\"6p6xgHyF7AeE6TZkSmFsko444wqoP15icUSqi2jfGiPN\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "type": "row", + "title": "\ud83c\udf69 Cost Distribution \u2014 $300", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 33 + }, + "id": 300, + "collapsed": false + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$300 \u2014 USDC (Sol) \u2192 USDC (Base)", + "gridPos": { + "h": 9, + "w": 8, + "x": 0, + "y": 34 + }, + "id": 301, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"300\",from_chain=\"Solana\",to_chain=\"Base\",from_token=\"EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$300 \u2014 USDC (Base) \u2192 USDT (Arb)", + "gridPos": { + "h": 9, + "w": 8, + "x": 8, + "y": 34 + }, + "id": 302, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"300\",from_chain=\"Base\",to_chain=\"Arbitrum\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "datasource": "Prometheus", + "type": "piechart", + "title": "$300 \u2014 TRUMP (Sol) \u2192 BRETT (Base)", + "gridPos": { + "h": 9, + "w": 8, + "x": 16, + "y": 34 + }, + "id": 303, + "targets": [ + { + "expr": "bridge_cost_usd{amount_usd=\"300\",from_token=\"6p6xgHyF7AeE6TZkSmFsko444wqoP15icUSqi2jfGiPN\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "options": { + "pieType": "donut", + "displayLabels": [ + "name", + "percent" + ], + "legend": { + "displayMode": "table", + "placement": "right", + "values": [ + "value", + "percent" + ] + }, + "tooltip": { + "mode": "single" + }, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "decimals": 4, + "color": { + "mode": "palette-classic" + } + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*mobula.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#228be6" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*relay.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#fd7e14" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*lifi.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#40c057" + } + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "(?i).*debridge.*" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#7950f2" + } + } + ] + } + ] + } + }, + { + "type": "row", + "title": "\ud83d\udcc8 Cost Evolution Over Time (selected route + amount)", + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 43 + }, + "id": 400, + "collapsed": true, + "panels": [ + { + "datasource": "Prometheus", + "type": "timeseries", + "title": "Cost USD Over Time", + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 44 + }, + "id": 401, + "targets": [ + { + "expr": "bridge_cost_usd{from_token=~\"$route\",amount_usd=\"$amount\"}", + "legendFormat": "{{bridge}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "currencyUSD", + "custom": { + "drawStyle": "line", + "lineWidth": 2, + "fillOpacity": 10 + } + } + } + } + ] + } + ], + "refresh": "30s", + "schemaVersion": 27, + "style": "dark", + "tags": [ + "bridge", + "pricing", + "ranking" + ], + "templating": { + "list": [ + { + "current": { + "selected": true, + "text": "USDC Sol \u2192 Base", + "value": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v" + }, + "hide": 0, + "includeAll": false, + "label": "Route (for ranking table)", + "multi": false, + "name": "route", + "options": [ + { + "selected": true, + "text": "USDC Sol \u2192 Base", + "value": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v" + }, + { + "selected": false, + "text": "USDC Base \u2192 USDT Arb", + "value": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913" + }, + { + "selected": false, + "text": "TRUMP Sol \u2192 BRETT Base", + "value": "6p6xgHyF7AeE6TZkSmFsko444wqoP15icUSqi2jfGiPN" + } + ], + "query": "USDC Sol \u2192 Base : EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v, USDC Base \u2192 USDT Arb : 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913, TRUMP Sol \u2192 BRETT Base : 6p6xgHyF7AeE6TZkSmFsko444wqoP15icUSqi2jfGiPN", + "skipUrlSync": false, + "type": "custom" + }, + { + "current": { + "selected": true, + "text": "5", + "value": "5" + }, + "hide": 0, + "includeAll": false, + "label": "Amount USD (for time series)", + "multi": false, + "name": "amount", + "options": [ + { + "selected": true, + "text": "5", + "value": "5" + }, + { + "selected": false, + "text": "50", + "value": "50" + }, + { + "selected": false, + "text": "300", + "value": "300" + } + ], + "query": "5, 50, 300", + "skipUrlSync": false, + "type": "custom" + } + ] + }, + "time": { + "from": "now-1h", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "\ud83c\udf09 Bridge Pricing & Ranking", + "uid": "bridge-pricing", + "version": 0 +} \ No newline at end of file diff --git a/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/dashboard.yml b/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/dashboard.yml new file mode 100644 index 00000000..b6fa1088 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/grafana/provisioning/dashboards/dashboard.yml @@ -0,0 +1,12 @@ +apiVersion: 1 + +providers: + - name: 'Bridge Latency Dashboards' + orgId: 1 + folder: '' + type: file + disableDeletion: false + updateIntervalSeconds: 10 + allowUiUpdates: false + options: + path: /etc/grafana/provisioning/dashboards diff --git a/harnesses/bridge-monitor/deploy/grafana/provisioning/datasources/prometheus.yml b/harnesses/bridge-monitor/deploy/grafana/provisioning/datasources/prometheus.yml new file mode 100644 index 00000000..0672b9c6 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/grafana/provisioning/datasources/prometheus.yml @@ -0,0 +1,9 @@ +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: https://prometheus-production-9ffe.up.railway.app + isDefault: true + editable: false diff --git a/harnesses/bridge-monitor/deploy/prometheus/Dockerfile b/harnesses/bridge-monitor/deploy/prometheus/Dockerfile new file mode 100644 index 00000000..21ebb852 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/prometheus/Dockerfile @@ -0,0 +1,12 @@ +FROM prom/prometheus:v2.49.1 + +USER root + +# Copy config and alert rules (we're already in prometheus folder as root directory) +COPY prometheus.yml /etc/prometheus/prometheus.yml +COPY alert_rules.yml /etc/prometheus/alert_rules.yml + +EXPOSE 9090 + +# Run as root to avoid permission issues with Railway volumes +CMD ["--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.path=/prometheus", "--storage.tsdb.retention.time=365d", "--web.enable-lifecycle", "--web.enable-admin-api"] diff --git a/harnesses/bridge-monitor/deploy/prometheus/alert_rules.yml b/harnesses/bridge-monitor/deploy/prometheus/alert_rules.yml new file mode 100644 index 00000000..b1afbf9e --- /dev/null +++ b/harnesses/bridge-monitor/deploy/prometheus/alert_rules.yml @@ -0,0 +1,130 @@ +groups: + - name: bridge_latency + interval: 30s + rules: + - alert: HighBridgeQuoteLatency + expr: histogram_quantile(0.95, rate(bridge_quote_latency_ms_bucket[5m])) > 2000 + for: 5m + labels: + severity: warning + annotations: + summary: "High bridge quote latency" + description: "{{ $labels.bridge }} quote p95 ({{ $labels.from_chain }} → {{ $labels.to_chain }}) is {{ $value | humanize }}ms" + + - alert: HighBridgeExecutionLatency + expr: histogram_quantile(0.95, rate(bridge_execution_latency_ms_bucket[5m])) > 120000 + for: 5m + labels: + severity: warning + annotations: + summary: "High bridge execution latency" + description: "{{ $labels.bridge }} exec p95 ({{ $labels.from_chain }} → {{ $labels.to_chain }}) is {{ $value | humanize }}ms" + + - alert: HighBridgeFees + expr: bridge_fees_percent > 5 + for: 15m + labels: + severity: warning + annotations: + summary: "High bridge fees" + description: "{{ $labels.bridge }} fees ({{ $labels.from_chain }} → {{ $labels.to_chain }}) = {{ $value }}%" + + - name: bridge_reliability + interval: 30s + rules: + # Computed revert rate over a 1h window from counters (the bridge_revert_rate + # gauge was never populated — see PR fix). clamp_min prevents div-by-zero. + - alert: HighBridgeRevertRate + expr: | + sum by (bridge, from_chain, to_chain) (rate(bridge_reverts_total[1h])) + / + clamp_min(sum by (bridge, from_chain, to_chain) (rate(bridge_reverts_total[1h]) + rate(bridge_success_total[1h])), 0.0001) + > 0.1 + for: 5m + labels: + severity: critical + annotations: + summary: "High bridge revert rate" + description: "{{ $labels.bridge }} revert rate ({{ $labels.from_chain }} → {{ $labels.to_chain }}) = {{ $value | humanizePercentage }}" + + # Three real-execution failures in a row on the same bridge — almost certainly a + # bridge-side outage or our integration broke. Page immediately. + - alert: BridgeConsecutiveFailures + expr: bridge_consecutive_failures >= 3 + for: 1m + labels: + severity: critical + annotations: + summary: "{{ $labels.bridge }}: {{ $value }} consecutive execution failures" + description: "{{ $labels.bridge }} has failed {{ $value }} executions in a row (region: {{ $labels.region }})" + + - alert: BridgeMonitorDown + expr: up{job="bridge-monitor"} == 0 + for: 2m + labels: + severity: critical + annotations: + summary: "Bridge monitor is down" + description: "bridge-monitor unreachable for 2m — scheduler not running, no executions." + + - name: wallet_balance + interval: 30s + rules: + # Any triangle leg below $50 means the next $50 scheduled run on that chain will fail. + - alert: LowTriangleBalance + expr: wallet_balance_usd{token=~"USDC|USDT|USDT0"} < 50 + for: 5m + labels: + severity: warning + annotations: + summary: "Triangle leg low: {{ $labels.chain }}/{{ $labels.token }} = ${{ $value | printf \"%.2f\" }}" + description: "Next $50 test from {{ $labels.chain }} will fail insufficient funds. Rebalance needed." + + - alert: CriticalTriangleBalance + expr: wallet_balance_usd{token=~"USDC|USDT|USDT0"} < 10 + for: 1m + labels: + severity: critical + annotations: + summary: "Triangle leg critically low: {{ $labels.chain }}/{{ $labels.token }} = ${{ $value | printf \"%.2f\" }}" + description: "Even the daily $5 test will fail on this chain." + + # Gas: we keep ~$10 native on each chain. Alert when dropping under $3. + - alert: LowGasBalance + expr: wallet_balance_usd{token=~"ETH|SOL"} < 3 + for: 5m + labels: + severity: warning + annotations: + summary: "Low gas on {{ $labels.chain }}: {{ $labels.token }} = ${{ $value | printf \"%.2f\" }}" + description: "Top up native-token balance — TXs will start failing." + + # R4 meme route is one-way (TRUMP Sol → BRETT Base), no inflow refill. + # Weekly $5 × 3 bridges = $15 per run. Alert at $30 (~2 weekly cycles left). + - alert: LowMemeSourceBalance + expr: wallet_balance_usd{chain="Solana", token="TRUMP"} < 30 + for: 5m + labels: + severity: warning + annotations: + summary: "TRUMP balance low on Solana: ${{ $value | printf \"%.2f\" }}" + description: "R4 TRUMP→BRETT will run out in ~{{ $value | humanize }} / 15 weekly cycles. Manual refill required (one-way route)." + + - alert: CriticalMemeSourceBalance + expr: wallet_balance_usd{chain="Solana", token="TRUMP"} < 15 + for: 1m + labels: + severity: critical + annotations: + summary: "TRUMP critically low: ${{ $value | printf \"%.2f\" }}" + description: "Next weekly R4 run will fail. Bridge TRUMP from another wallet to resume." + + # Balance refresh loop died: last update > 20 min ago (updater runs every 5 min). + - alert: WalletBalanceStale + expr: time() - wallet_balance_last_update_timestamp_seconds > 1200 + for: 1m + labels: + severity: warning + annotations: + summary: "Wallet balance metrics stale" + description: "Balance refresh hasn't run in {{ $value | humanizeDuration }}. Check monitor logs." diff --git a/harnesses/bridge-monitor/deploy/prometheus/prometheus.yml b/harnesses/bridge-monitor/deploy/prometheus/prometheus.yml new file mode 100644 index 00000000..61b5e4f7 --- /dev/null +++ b/harnesses/bridge-monitor/deploy/prometheus/prometheus.yml @@ -0,0 +1,19 @@ +global: + scrape_interval: 15s + evaluation_interval: 15s + external_labels: + cluster: 'bridge-latency' + +alerting: + alertmanagers: + - static_configs: + - targets: + - alertmanager.railway.internal:9093 + +rule_files: + - 'alert_rules.yml' + +scrape_configs: + - job_name: 'bridge-monitor' + static_configs: + - targets: ['bridge-monitor.railway.internal:9090'] diff --git a/harnesses/bridge-monitor/go.mod b/harnesses/bridge-monitor/go.mod new file mode 100644 index 00000000..043718fd --- /dev/null +++ b/harnesses/bridge-monitor/go.mod @@ -0,0 +1,71 @@ +module bridging-latency + +go 1.24.0 + +toolchain go1.24.4 + +require ( + github.com/ethereum/go-ethereum v1.17.2 + github.com/gagliardetto/solana-go v1.18.0 + github.com/prometheus/client_golang v1.20.5 +) + +require ( + filippo.io/edwards25519 v1.2.0 // indirect + github.com/Microsoft/go-winio v0.6.2 // indirect + github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6 // indirect + github.com/StackExchange/wmi v1.2.1 // indirect + github.com/benbjohnson/clock v1.3.5 // indirect + github.com/beorn7/perks v1.0.1 // indirect + github.com/bits-and-blooms/bitset v1.20.0 // indirect + github.com/blendle/zapdriver v1.3.1 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/consensys/gnark-crypto v0.18.1 // indirect + github.com/crate-crypto/go-eth-kzg v1.5.0 // indirect + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/deckarep/golang-set/v2 v2.6.0 // indirect + github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect + github.com/ethereum/c-kzg-4844/v2 v2.1.6 // indirect + github.com/fatih/color v1.18.0 // indirect + github.com/gagliardetto/binary v0.8.0 // indirect + github.com/gagliardetto/treeout v0.1.4 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/go-ole/go-ole v1.3.0 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/gorilla/websocket v1.5.3 // indirect + github.com/holiman/uint256 v1.3.2 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/klauspost/compress v1.18.0 // indirect + github.com/logrusorgru/aurora v2.0.3+incompatible // indirect + github.com/mattn/go-colorable v0.1.14 // indirect + github.com/mattn/go-isatty v0.0.20 // indirect + github.com/mitchellh/go-testing-interface v1.14.1 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/mostynb/zstdpool-freelist v0.0.0-20201229113212-927304c0c3b1 // indirect + github.com/mr-tron/base58 v1.2.0 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/prometheus/client_model v0.6.1 // indirect + github.com/prometheus/common v0.55.0 // indirect + github.com/prometheus/procfs v0.15.1 // indirect + github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible // indirect + github.com/streamingfast/logging v0.0.0-20250404134358-92b15d2fbd2e // indirect + github.com/supranational/blst v0.3.16 // indirect + github.com/tklauser/go-sysconf v0.3.12 // indirect + github.com/tklauser/numcpus v0.6.1 // indirect + go.mongodb.org/mongo-driver/v2 v2.5.0 // indirect + go.opentelemetry.io/auto/sdk v1.2.1 // indirect + go.opentelemetry.io/otel v1.40.0 // indirect + go.opentelemetry.io/otel/metric v1.40.0 // indirect + go.opentelemetry.io/otel/trace v1.40.0 // indirect + go.uber.org/multierr v1.11.0 // indirect + go.uber.org/ratelimit v0.3.1 // indirect + go.uber.org/zap v1.27.0 // indirect + golang.org/x/crypto v0.47.0 // indirect + golang.org/x/sync v0.18.0 // indirect + golang.org/x/sys v0.40.0 // indirect + golang.org/x/term v0.39.0 // indirect + golang.org/x/time v0.11.0 // indirect + google.golang.org/protobuf v1.36.11 // indirect +) diff --git a/harnesses/bridge-monitor/go.sum b/harnesses/bridge-monitor/go.sum new file mode 100644 index 00000000..b4e068d8 --- /dev/null +++ b/harnesses/bridge-monitor/go.sum @@ -0,0 +1,309 @@ +filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo= +filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc= +github.com/AlekSi/pointer v1.2.0 h1:glcy/gc4h8HnG2Z3ZECSzZ1IX1x2JxRVuDzaJwQE0+w= +github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tSNSBle0= +github.com/DataDog/zstd v1.4.5 h1:EndNeuB0l9syBZhut0wns3gV1hL8zX8LIu6ZiVHWLIQ= +github.com/DataDog/zstd v1.4.5/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo= +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= +github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6 h1:1zYrtlhrZ6/b6SAjLSfKzWtdgqK0U+HtH/VcBWh1BaU= +github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6/go.mod h1:ioLG6R+5bUSO1oeGSDxOV3FADARuMoytZCSX6MEMQkI= +github.com/StackExchange/wmi v1.2.1 h1:VIkavFPXSjcnS+O8yTq7NI32k0R5Aj+v39y29VYDOSA= +github.com/StackExchange/wmi v1.2.1/go.mod h1:rcmrprowKIVzvc+NUiLncP2uuArMWLCbu9SBzvHz7e8= +github.com/VictoriaMetrics/fastcache v1.13.0 h1:AW4mheMR5Vd9FkAPUv+NH6Nhw+fmbTMGMsNAoA/+4G0= +github.com/VictoriaMetrics/fastcache v1.13.0/go.mod h1:hHXhl4DA2fTL2HTZDJFXWgW0LNjo6B+4aj2Wmng3TjU= +github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= +github.com/benbjohnson/clock v1.3.5 h1:VvXlSJBzZpA/zum6Sj74hxwYI2DIxRWuNIoXAzHZz5o= +github.com/benbjohnson/clock v1.3.5/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/bits-and-blooms/bitset v1.20.0 h1:2F+rfL86jE2d/bmw7OhqUg2Sj/1rURkBn3MdfoPyRVU= +github.com/bits-and-blooms/bitset v1.20.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8= +github.com/blendle/zapdriver v1.3.1 h1:C3dydBOWYRiOk+B8X9IVZ5IOe+7cl+tGOexN4QqHfpE= +github.com/blendle/zapdriver v1.3.1/go.mod h1:mdXfREi6u5MArG4j9fewC+FGnXaBR+T4Ox4J2u4eHCc= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/cockroachdb/errors v1.11.3 h1:5bA+k2Y6r+oz/6Z/RFlNeVCesGARKuC6YymtcDrbC/I= +github.com/cockroachdb/errors v1.11.3/go.mod h1:m4UIW4CDjx+R5cybPsNrRbreomiFqt8o1h1wUVazSd8= +github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce h1:giXvy4KSc/6g/esnpM7Geqxka4WSqI1SZc7sMJFd3y4= +github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce/go.mod h1:9/y3cnZ5GKakj/H4y9r9GTjCvAFta7KLgSHPJJYc52M= +github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b h1:r6VH0faHjZeQy818SGhaone5OnYfxFR/+AzdY3sf5aE= +github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b/go.mod h1:Vz9DsVWQQhf3vs21MhPMZpMGSht7O/2vFW2xusFUVOs= +github.com/cockroachdb/pebble v1.1.5 h1:5AAWCBWbat0uE0blr8qzufZP5tBjkRyy/jWe1QWLnvw= +github.com/cockroachdb/pebble v1.1.5/go.mod h1:17wO9el1YEigxkP/YtV8NtCivQDgoCyBg5c4VR/eOWo= +github.com/cockroachdb/redact v1.1.5 h1:u1PMllDkdFfPWaNGMyLD1+so+aq3uUItthCFqzwPJ30= +github.com/cockroachdb/redact v1.1.5/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg= +github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 h1:zuQyyAKVxetITBuuhv3BI9cMrmStnpT18zmgmTxunpo= +github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ= +github.com/consensys/gnark-crypto v0.18.1 h1:RyLV6UhPRoYYzaFnPQA4qK3DyuDgkTgskDdoGqFt3fI= +github.com/consensys/gnark-crypto v0.18.1/go.mod h1:L3mXGFTe1ZN+RSJ+CLjUt9x7PNdx8ubaYfDROyp2Z8c= +github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc= +github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/crate-crypto/go-eth-kzg v1.5.0 h1:FYRiJMJG2iv+2Dy3fi14SVGjcPteZ5HAAUe4YWlJygc= +github.com/crate-crypto/go-eth-kzg v1.5.0/go.mod h1:J9/u5sWfznSObptgfa92Jq8rTswn6ahQWEuiLHOjCUI= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dchest/siphash v1.2.3 h1:QXwFc8cFOR2dSa/gE6o/HokBMWtLUaNDVd+22aKHeEA= +github.com/dchest/siphash v1.2.3/go.mod h1:0NvQU092bT0ipiFN++/rXm69QG9tVxLAlQHIXMPAkHc= +github.com/deckarep/golang-set/v2 v2.6.0 h1:XfcQbWM1LlMB8BsJ8N9vW5ehnnPVIw0je80NsVHagjM= +github.com/deckarep/golang-set/v2 v2.6.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4= +github.com/decred/dcrd/crypto/blake256 v1.0.0 h1:/8DMNYp9SGi5f0w7uCm6d6M4OU2rGFK09Y2A4Xv7EE0= +github.com/decred/dcrd/crypto/blake256 v1.0.0/go.mod h1:sQl2p6Y26YV+ZOcSTP6thNdn47hh8kt6rqSlvmrXFAc= +github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 h1:YLtO71vCjJRCBcrPMtQ9nqBsqpA1m5sE92cU+pd5Mcc= +github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1/go.mod h1:hyedUtir6IdtD/7lIxGeCxkaw7y45JueMRL4DIyJDKs= +github.com/emicklei/dot v1.6.2 h1:08GN+DD79cy/tzN6uLCT84+2Wk9u+wvqP+Hkx/dIR8A= +github.com/emicklei/dot v1.6.2/go.mod h1:DeV7GvQtIw4h2u73RKBkkFdvVAz0D9fzeJrgPW6gy/s= +github.com/ethereum/c-kzg-4844/v2 v2.1.6 h1:xQymkKCT5E2Jiaoqf3v4wsNgjZLY0lRSkZn27fRjSls= +github.com/ethereum/c-kzg-4844/v2 v2.1.6/go.mod h1:8HMkUZ5JRv4hpw/XUrYWSQNAUzhHMg2UDb/U+5m+XNw= +github.com/ethereum/go-bigmodexpfix v0.0.0-20250911101455-f9e208c548ab h1:rvv6MJhy07IMfEKuARQ9TKojGqLVNxQajaXEp/BoqSk= +github.com/ethereum/go-bigmodexpfix v0.0.0-20250911101455-f9e208c548ab/go.mod h1:IuLm4IsPipXKF7CW5Lzf68PIbZ5yl7FFd74l/E0o9A8= +github.com/ethereum/go-ethereum v1.17.2 h1:ag6geu0kn8Hv5FLKTpH+Hm2DHD+iuFtuqKxEuwUsDOI= +github.com/ethereum/go-ethereum v1.17.2/go.mod h1:KHcRXfGOUfUmKg51IhQ0IowiqZ6PqZf08CMtk0g5K1o= +github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM= +github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU= +github.com/ferranbt/fastssz v0.1.4 h1:OCDB+dYDEQDvAgtAGnTSidK1Pe2tW3nFV40XyMkTeDY= +github.com/ferranbt/fastssz v0.1.4/go.mod h1:Ea3+oeoRGGLGm5shYAeDgu6PGUlcvQhE2fILyD9+tGg= +github.com/gagliardetto/binary v0.8.0 h1:U9ahc45v9HW0d15LoN++vIXSJyqR/pWw8DDlhd7zvxg= +github.com/gagliardetto/binary v0.8.0/go.mod h1:2tfj51g5o9dnvsc+fL3Jxr22MuWzYXwx9wEoN0XQ7/c= +github.com/gagliardetto/solana-go v1.18.0 h1:+D9KfL5DiuBdhsc1DhlnUU2jww0nSlIPjGBDWNCBxRU= +github.com/gagliardetto/solana-go v1.18.0/go.mod h1:lIYMhelbrx7OaiysurLQS9hC8bqpzAH4OZ51nZo2uaA= +github.com/gagliardetto/treeout v0.1.4 h1:ozeYerrLCmCubo1TcIjFiOWTTGteOOHND1twdFpgwaw= +github.com/gagliardetto/treeout v0.1.4/go.mod h1:loUefvXTrlRG5rYmJmExNryyBRh8f89VZhmMOyCyqok= +github.com/getsentry/sentry-go v0.27.0 h1:Pv98CIbtB3LkMWmXi4Joa5OOcwbmnX88sF5qbK3r3Ps= +github.com/getsentry/sentry-go v0.27.0/go.mod h1:lc76E2QywIyW8WuBnwl8Lc4bkmQH4+w1gwTf25trprY= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-ole/go-ole v1.2.5/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= +github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE= +github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78= +github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E= +github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI= +github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= +github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs= +github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= +github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/grafana/pyroscope-go v1.2.7 h1:VWBBlqxjyR0Cwk2W6UrE8CdcdD80GOFNutj0Kb1T8ac= +github.com/grafana/pyroscope-go v1.2.7/go.mod h1:o/bpSLiJYYP6HQtvcoVKiE9s5RiNgjYTj1DhiddP2Pc= +github.com/grafana/pyroscope-go/godeltaprof v0.1.9 h1:c1Us8i6eSmkW+Ez05d3co8kasnuOY813tbMN8i/a3Og= +github.com/grafana/pyroscope-go/godeltaprof v0.1.9/go.mod h1:2+l7K7twW49Ct4wFluZD3tZ6e0SjanjcUUBPVD/UuGU= +github.com/hashicorp/go-bexpr v0.1.10 h1:9kuI5PFotCboP3dkDYFr/wi0gg0QVbSNz5oFRpxn4uE= +github.com/hashicorp/go-bexpr v0.1.10/go.mod h1:oxlubA2vC/gFVfX1A6JGp7ls7uCDlfJn732ehYYg+g0= +github.com/holiman/billy v0.0.0-20250707135307-f2f9b9aae7db h1:IZUYC/xb3giYwBLMnr8d0TGTzPKFGNTCGgGLoyeX330= +github.com/holiman/billy v0.0.0-20250707135307-f2f9b9aae7db/go.mod h1:xTEYN9KCHxuYHs+NmrmzFcnvHMzLLNiGFafCb1n3Mfg= +github.com/holiman/bloomfilter/v2 v2.0.3 h1:73e0e/V0tCydx14a0SCYS/EWCxgwLZ18CZcZKVu0fao= +github.com/holiman/bloomfilter/v2 v2.0.3/go.mod h1:zpoh+gs7qcpqrHr3dB55AMiJwo0iURXE7ZOP9L9hSkA= +github.com/holiman/uint256 v1.3.2 h1:a9EgMPSC1AAaj1SZL5zIQD3WbwTuHrMGOerLjGmM/TA= +github.com/holiman/uint256 v1.3.2/go.mod h1:EOMSn4q6Nyt9P6efbI3bueV4e1b3dGlUCXeiRV4ng7E= +github.com/huin/goupnp v1.3.0 h1:UvLUlWDNpoUdYzb2TCn+MuTWtcjXKSza2n6CBdQ0xXc= +github.com/huin/goupnp v1.3.0/go.mod h1:gnGPsThkYa7bFi/KWmEysQRf48l2dvR5bxr2OFckNX8= +github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= +github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/klauspost/compress v1.11.4/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= +github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo= +github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ= +github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4= +github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= +github.com/leanovate/gopter v0.2.11 h1:vRjThO1EKPb/1NsDXuDrzldR28RLkBflWYcU9CvzWu4= +github.com/leanovate/gopter v0.2.11/go.mod h1:aK3tzZP/C+p1m3SPRE4SYZFGP7jjkuSI4f7Xvpt0S9c= +github.com/logrusorgru/aurora v2.0.3+incompatible h1:tOpm7WcpBTn4fjmVfgpQq0EfczGlG91VSDkswnjF5A8= +github.com/logrusorgru/aurora v2.0.3+incompatible/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4= +github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE= +github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= +github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= +github.com/mitchellh/go-testing-interface v1.14.1 h1:jrgshOhYAUVNMAJiKbEu7EqAwgJJ2JqpQmpLJOu07cU= +github.com/mitchellh/go-testing-interface v1.14.1/go.mod h1:gfgS7OtZj6MA4U1UrDRp04twqAjfvlZyCfX3sDjEym8= +github.com/mitchellh/mapstructure v1.4.1 h1:CpVNEelQCZBooIPDn+AR3NpivK/TIKU8bDxdASFVQag= +github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= +github.com/mitchellh/pointerstructure v1.2.0 h1:O+i9nHnXS3l/9Wu7r4NrEdwA2VFTicjUEN1uBnDo34A= +github.com/mitchellh/pointerstructure v1.2.0/go.mod h1:BRAsLI5zgXmw97Lf6s25bs8ohIXc3tViBH44KcwB2g4= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/mostynb/zstdpool-freelist v0.0.0-20201229113212-927304c0c3b1 h1:mPMvm6X6tf4w8y7j9YIt6V9jfWhL6QlbEc7CCmeQlWk= +github.com/mostynb/zstdpool-freelist v0.0.0-20201229113212-927304c0c3b1/go.mod h1:ye2e/VUEtE2BHE+G/QcKkcLQVAEJoYRFj5VUOQatCRE= +github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o= +github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/onsi/gomega v1.10.1 h1:o0+MgICZLuZ7xjH7Vx6zS/zcu93/BEp1VwkIW1mEXCE= +github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= +github.com/pion/dtls/v2 v2.2.7 h1:cSUBsETxepsCSFSxC3mc/aDo14qQLMSL+O6IjG28yV8= +github.com/pion/dtls/v2 v2.2.7/go.mod h1:8WiMkebSHFD0T+dIU+UeBaoV7kDhOW5oDCzZ7WZ/F9s= +github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY= +github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms= +github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= +github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= +github.com/pion/transport/v2 v2.2.1 h1:7qYnCBlpgSJNYMbLCKuSY9KbQdBFoETvPNETv0y4N7c= +github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= +github.com/pion/transport/v3 v3.0.1 h1:gDTlPJwROfSfz6QfSi0ZmeCSkFcnWWiiR9ES0ouANiM= +github.com/pion/transport/v3 v3.0.1/go.mod h1:UY7kiITrlMv7/IKgd5eTUcaahZx5oUN3l9SzK5f5xE0= +github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_golang v1.20.5 h1:cxppBPuYhUnsO6yo/aoRol4L7q7UFfdm+bR9r+8l63Y= +github.com/prometheus/client_golang v1.20.5/go.mod h1:PIEt8X02hGcP8JWbeHyeZ53Y/jReSnHgO035n//V5WE= +github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E= +github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY= +github.com/prometheus/common v0.55.0 h1:KEi6DK7lXW/m7Ig5i47x0vRzuBsHuvJdi5ee6Y3G1dc= +github.com/prometheus/common v0.55.0/go.mod h1:2SECS4xJG1kd8XF9IcM1gMX6510RAEL65zxzNImwdc8= +github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc= +github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk= +github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= +github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= +github.com/rs/cors v1.7.0 h1:+88SsELBHx5r+hZ8TCkggzSstaWNbDvThkVK8H6f9ik= +github.com/rs/cors v1.7.0/go.mod h1:gFx+x8UowdsKA9AchylcLynDq+nNFfI8FkUZdN/jGCU= +github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk= +github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible h1:Bn1aCHHRnjv4Bl16T8rcaFjYSrGrIZvpiGO6P3Q4GpU= +github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA= +github.com/shopspring/decimal v1.3.1 h1:2Usl1nmF/WZucqkFZhnfFYxxxu8LG21F6nPQBE5gKV8= +github.com/shopspring/decimal v1.3.1/go.mod h1:DKyhrW/HYNuLGql+MJL6WCR6knT2jwCFRcu2hWCYk4o= +github.com/streamingfast/logging v0.0.0-20230608130331-f22c91403091/go.mod h1:VlduQ80JcGJSargkRU4Sg9Xo63wZD/l8A5NC/Uo1/uU= +github.com/streamingfast/logging v0.0.0-20250404134358-92b15d2fbd2e h1:qGVGDR2/bXLyR498un1hvhDQPUJ/m14JBRTJz+c67Bc= +github.com/streamingfast/logging v0.0.0-20250404134358-92b15d2fbd2e/go.mod h1:VlduQ80JcGJSargkRU4Sg9Xo63wZD/l8A5NC/Uo1/uU= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +github.com/supranational/blst v0.3.16 h1:bTDadT+3fK497EvLdWRQEjiGnUtzJ7jjIUMF0jqwYhE= +github.com/supranational/blst v0.3.16/go.mod h1:jZJtfjgudtNl4en1tzwPIV3KjUnQUvG3/j+w+fVonLw= +github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY= +github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc= +github.com/test-go/testify v1.1.4 h1:Tf9lntrKUMHiXQ07qBScBTSA0dhYQlu83hswqelv1iE= +github.com/test-go/testify v1.1.4/go.mod h1:rH7cfJo/47vWGdi4GPj16x3/t1xGOj2YxzmNQzk2ghU= +github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU= +github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI= +github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk= +github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY= +github.com/urfave/cli/v2 v2.27.5 h1:WoHEJLdsXr6dDWoJgMq/CboDmyY/8HMMH1fTECbih+w= +github.com/urfave/cli/v2 v2.27.5/go.mod h1:3Sevf16NykTbInEnD0yKkjDAeZDS0A6bzhBH5hrMvTQ= +github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4= +github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM= +github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= +go.mongodb.org/mongo-driver/v2 v2.5.0 h1:yXUhImUjjAInNcpTcAlPHiT7bIXhshCTL3jVBkF3xaE= +go.mongodb.org/mongo-driver/v2 v2.5.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/otel v1.40.0 h1:oA5YeOcpRTXq6NN7frwmwFR0Cn3RhTVZvXsP4duvCms= +go.opentelemetry.io/otel v1.40.0/go.mod h1:IMb+uXZUKkMXdPddhwAHm6UfOwJyh4ct1ybIlV14J0g= +go.opentelemetry.io/otel/metric v1.40.0 h1:rcZe317KPftE2rstWIBitCdVp89A2HqjkxR3c11+p9g= +go.opentelemetry.io/otel/metric v1.40.0/go.mod h1:ib/crwQH7N3r5kfiBZQbwrTge743UDc7DTFVZrrXnqc= +go.opentelemetry.io/otel/sdk v1.40.0 h1:KHW/jUzgo6wsPh9At46+h4upjtccTmuZCFAc9OJ71f8= +go.opentelemetry.io/otel/sdk v1.40.0/go.mod h1:Ph7EFdYvxq72Y8Li9q8KebuYUr2KoeyHx0DRMKrYBUE= +go.opentelemetry.io/otel/trace v1.40.0 h1:WA4etStDttCSYuhwvEa8OP8I5EWu24lkOzp+ZYblVjw= +go.opentelemetry.io/otel/trace v1.40.0/go.mod h1:zeAhriXecNGP/s2SEG3+Y8X9ujcJOTqQ5RgdEJcawiA= +go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= +go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= +go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= +go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= +go.uber.org/goleak v1.1.11/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= +go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= +go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= +go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= +go.uber.org/ratelimit v0.3.1 h1:K4qVE+byfv/B3tC+4nYWP7v/6SimcO7HzHekoMNBma0= +go.uber.org/ratelimit v0.3.1/go.mod h1:6euWsTB6U/Nb3X++xEUXA8ciPJvr19Q/0h1+oDcJhRk= +go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= +go.uber.org/zap v1.21.0/go.mod h1:wjWOCqI0f2ZZrJF/UufIOkiC8ii6tm1iqIsLo76RfJw= +go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8= +go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20220214200702-86341886e292/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= +golang.org/x/crypto v0.47.0 h1:V6e3FRj+n4dbpw86FJ8Fv7XVOql7TEwpHapKoMJ/GO8= +golang.org/x/crypto v0.47.0/go.mod h1:ff3Y9VzzKbwSSEzWqJsJVBnWmRwRSHt/6Op5n9bQc4A= +golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df h1:UA2aFVmmsIlefxMk29Dp2juaUSth8Pyn3Tq5Y5mJGME= +golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc= +golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= +golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o= +golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I= +golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ= +golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= +golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.39.0 h1:RclSuaJf32jOqZz74CkPA9qFuVTX7vhLlpfj/IGWlqY= +golang.org/x/term v0.39.0/go.mod h1:yxzUCTP/U+FzoxfdKmLaA0RV1WgE0VY7hXBwKtY/4ww= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE= +golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8= +golang.org/x/time v0.11.0 h1:/bpjEDfN9tkoN/ryeYHnv5hcMlc8ncjMcM4XBk5NWV0= +golang.org/x/time v0.11.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= +gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/harnesses/bridge-quote/README.md b/harnesses/bridge-quote/README.md deleted file mode 100644 index 118b914c..00000000 --- a/harnesses/bridge-quote/README.md +++ /dev/null @@ -1,50 +0,0 @@ -# Harness · bridge-quote - -> Issues identical cross-chain quote requests to bridge APIs and records latency + success. - -**Bench**: [№ 002 · Bridge Quote Latency](../../benchmarks/bridge-quote-latency.yml) - -## What it measures - -- **Quote latency** — wall-clock time from request emission to a usable cross-chain quote response. -- **Success rate** — share of requests that returned a valid quote within the 5,000 ms timeout. - -## Bridges - -`mobula` · `relay` · `lifi` · `debridge` - -## Inputs - -- Routes: USDC pairs spanning Solana, Base and Arbitrum. -- Notional sizes: $5, $50, $300. -- Cadence: full sweep (4 routes × 3 amounts × N bridges) every 5 minutes. -- Region: currently `eu-west` only. - -## Metrics emitted - -``` -bridge_quote_latency_ms_{bucket,sum,count}{bridge, from_chain, to_chain, from_token, to_token, amount_usd, region} histogram -bridge_quote_success{bridge, from_chain, to_chain, from_token, to_token, amount_usd, region} gauge 0/1 -bridge_errors_total{bridge, from_chain, to_chain, from_token, to_token, amount_usd, region, error_type} counter -bridge_estimated_time_ms{bridge, ...} gauge -``` - -## Env vars - -| Var | Required | Notes | -| --------------------- | -------- | ---------------------------------- | -| `MOBULA_API_KEY` | yes | https://mobula.io/dashboard/api | -| `LIFI_API_KEY` | optional | public endpoints work without it | -| `RELAY_API_KEY` | optional | | -| `DEBRIDGE_API_KEY` | optional | | -| `PROMETHEUS_PUSH_URL` | yes | Prometheus push gateway | -| `MONITOR_REGION` | yes | one of `us-east`, `eu-west`, `sgp` | - -## Running locally - -> Implementation TBD. See the spec YAML for the queries the site issues -> against the metrics emitted by this harness. - -```bash -make run REGION=eu-west -```